Characterisation tests for faster PageObject development

By David Röthlisberger. Tweet your comments @drothlis.

Published 18 Jun 2019.

Problem: Automated UI tests are difficult to maintain.
Solution: Press a button and instantly see how your PageObjects are affected by the latest change in your application-under-test.

This is the second article in the series “Better Tooling for PageObjects” based on our experiences with Stb-tester, which tests TV GUIs by looking at the pixels. Start with part 1 if you haven’t read it yet. See the full series here.

A characterisation test records the output of a particular API, or of a function in your code. It doesn’t judge if that output is correct or not, it just saves it away somewhere. Then when you change the system-under-test, you can run the characterisation tests again to see any differences in the output. If the diff is expected, you need to (manually) accept it so that it becomes the new baseline; if the diff is unexpected, then you’ve caught a bug! This is also known as approval testing, golden-master testing, snapshot testing, and probably several other names.

When testing a GUI, you can run characterisation tests against screenshots of the app to notice if anything has changed visually. Tools to do this include applitools and retest.

But this article is not about testing the GUI of your application; it’s about testing your PageObject code.

Now hang on: Automated tests for your automated tests? Really? Have I gone test mad? Well, the only reason we’d do this is if it makes writing the tests easier. Less work! Happier testers!

Stb-tester’s test-development process

Here’s the process when creating a PageObject with Stb-tester:

  1. Save a snapshot of the relevant page to disk. In Stb-tester’s case this is a screenshot. (In a hypothetical Selenium implementation, this would be a snapshot of the DOM instead.)

  2. Write your PageObject and test it against the saved screenshot. You can test it in the Python REPL or using our characterisation-testing tool (see the next step). In the previous article we saw that the application’s current state (the screenshot) is injected into the PageObject’s constructor, so testing against a screenshot is just as easy as testing against the live application.

  3. Run the characterisation tests. Stb-tester’s tool for this is called “stbt auto-selftest”:

    The output is a plain text file that shows all the public properties of the PageObject for each relevant screenshot, in a format suitable for diffing.

  4. Commit the generated output to Git alongside your test scripts. This is the baseline that you will compare future changes against. Store the screenshot in Git too, optionally using Git LFS if you don’t want to bloat your Git repo.

Now —and this is where it really shines— here’s the maintenance process when you need to update a PageObject after the application-under-test has changed:

  1. Update the screenshot of the relevant page. Typically we’ve noticed the change because the UI tests failed in CI; so we just grab the screenshot that was saved by the failed UI test.

  2. Re-run the characterisation tests, and compare the differences with git diff:

    In this example the output shows that the SquareKeyboard PageObject doesn’t recognize the new screenshot after the app’s on-screen-keyboard appearance changed.

  3. Fix your PageObject and repeat step 2 until satisfied.

  4. If the remaining diffs are desired behaviour, commit the changes so that they become the baseline for future characterisation testing.

Benefits

In our industry (TV & streaming video) our customers typically have an app with the same look & feel but with multiple different implementations for different devices (Roku, PlayStation, XBox, Apple TV, etc). In this case we re-use the same PageObject code, and often the same tests, across all the devices. Stb-tester can do this because it’s a “black box” tool – we don’t care which technology the app is implemented with, we just look at the HDMI video coming out of the device-under-test. This multiplies the benefit of our characterisation tests: If we need to modify a PageObject to handle some quirk of one specific device, we can easily verify that we haven’t broken the behaviour for the other devices.

Limitations

Our characterisation tests only apply to the PageObject’s properties (read-only getter methods for reading information from the page). Any actions (methods that interact with the application-under-test by clicking, entering text, or pressing buttons on a remote control) still need to be tested the old fashioned way.

And when I said “instantly”… well… it does take some time to run the code in each PageObject against each screenshot. If the PageObjects are doing a lot of OCR it can be slow. In a large test-pack it can take from a few seconds to a few minutes, depending on how many screenshots have changed (we cache all the slow operations like OCR in between invocations of the tool, for screenshots that haven’t changed). If you’re debugging a single PageObject then you don’t need to test all the other ones, so while you’re iterating it’s fast enough.

Terminology

We call these automatically-generated characterisation tests “selftests” because they test the code of the UI tests, rather than testing the end-user application.

Implementation notes

The source code of the stbt auto-selftest tool is on GitHub here.

Each PageObject must implement a property called is_visible that returns True if the screenshot is showing the relevant page or widget. stbt auto-selftest tests each PageObject class against each screenshot, but it only shows output for screenshots that match according to is_visible.

Any function that takes a video-frame (screenshot) as input can be tested like this. Just add an attribute called AUTO_SELFTEST_EXPRESSIONS to the function like this:

The PageObject base-class (on GitHub here) defines AUTO_SELFTEST_EXPRESSIONS automatically.

All of Stb-tester’s image-processing operations are cached between invocations of stbt auto-selftest. We use xxHash (an extremely fast hash algorithm) to hash each raw screenshot, and LMDB (a fast memory-mapped key-value store) to cache the output of each operation. Our cache implementation (on GitHub here) is provided as a Python decorator so that you can add this same caching to any expensive functions that you have implemented in your own test-pack.

The stbt auto-selftest tool and Stb-tester’s PageObject base class were designed and implemented by William Manley.

This was the second article in the series Better Tooling for PageObjects. Next article: Live documentation: Show me the relevant PageObjects for the page I’m looking at (coming soon). See the full series here.