Stb-tester’s test-development process

Here’s the process when creating a PageObject with Stb-tester:

1. Save a snapshot of the relevant page to disk. In Stb-tester’s case this is a screenshot. (In a hypothetical Selenium implementation, this would be a snapshot of the DOM instead.)

2. Write your PageObject and test it against the saved screenshot. You can test it in the Python REPL or using our characterisation-testing tool (see the next step). In the previous article we saw that the application’s current state (the screenshot) is injected into the PageObject’s constructor, so testing against a screenshot is just as easy as testing against the live application.

3. Run the characterisation tests. Stb-tester’s tool for this is called “stbt auto-selftest”:

   The output is a plain text file that shows all the public properties of the PageObject for each relevant screenshot, in a format suitable for diffing.

4. Commit the generated output to Git alongside your test scripts. This is the baseline that you will compare future changes against. Store the screenshot in Git too, optionally using Git LFS if you don’t want to bloat your Git repo.

Now —and this is where it really shines— here’s the maintenance process when you need to update a PageObject after the application-under-test has changed:

1. Update the screenshot of the relevant page. Typically we’ve noticed the change because the UI tests failed in CI; so we just grab the screenshot that was saved by the failed UI test.

2. Re-run the characterisation tests, and compare the differences with git diff:

   

   In this example the output shows that the SquareKeyboard PageObject doesn’t recognize the new screenshot after the app’s on-screen-keyboard appearance changed.

3. Fix your PageObject and repeat step 2 until satisfied.

4. If the remaining diffs are desired behaviour, commit the changes so that they become the baseline for future characterisation testing.

Benefits

• When the application-under-test has changed: Quickly seeing the effect on all of your PageObjects.

• When modifying a PageObject: Testing your changes without having to run an entire UI test (which is slooow).

• When refactoring: Testing that there are no unexpected effects on your PageObjects’ behaviours.

In our industry (TV & streaming video) our customers typically have an app with the same look & feel but with multiple different implementations for different devices (Roku, PlayStation, XBox, Apple TV, etc). In this case we re-use the same PageObject code, and often the same tests, across all the devices. Stb-tester can do this because it’s a “black box” tool – we don’t care which technology the app is implemented with, we just look at the HDMI video coming out of the device-under-test. This multiplies the benefit of our characterisation tests: If we need to modify a PageObject to handle some quirk of one specific device, we can easily verify that we haven’t broken the behaviour for the other devices.

Limitations

Our characterisation tests only apply to the PageObject’s properties (read-only getter methods for reading information from the page). Any actions (methods that interact with the application-under-test by clicking, entering text, or pressing buttons on a remote control) still need to be tested the old fashioned way.

And when I said “instantly”… well… it does take some time to run the code in each PageObject against each screenshot. If the PageObjects are doing a lot of OCR it can be slow. In a large test-pack it can take from a few seconds to a few minutes, depending on how many screenshots have changed (we cache all the slow operations like OCR in between invocations of the tool, for screenshots that haven’t changed). If you’re debugging a single PageObject then you don’t need to test all the other ones, so while you’re iterating it’s fast enough.

Terminology

We call these automatically-generated characterisation tests “selftests” because they test the code of the UI tests, rather than testing the end-user application.

Implementation notes

The source code of the stbt auto-selftest tool is on GitHub here.

Each PageObject must implement a property called is_visible that returns True if the screenshot is showing the relevant page or widget. stbt auto-selftest tests each PageObject class against each screenshot, but it only shows output for screenshots that match according to is_visible.

Any function that takes a video-frame (screenshot) as input can be tested like this. Just add an attribute called AUTO_SELFTEST_EXPRESSIONS to the function like this:

The PageObject base-class (on GitHub here) defines AUTO_SELFTEST_EXPRESSIONS automatically.

All of Stb-tester’s image-processing operations are cached between invocations of stbt auto-selftest. We use xxHash (an extremely fast hash algorithm) to hash each raw screenshot, and LMDB (a fast memory-mapped key-value store) to cache the output of each operation. Our cache implementation (on GitHub here) is provided as a Python decorator so that you can add this same caching to any expensive functions that you have implemented in your own test-pack.

The stbt auto-selftest tool and Stb-tester’s PageObject base class were designed and implemented by William Manley.