Running tests on remote hardware from your IDE, with pytest

At Stb-tester we have test scripts, written in Python, that require specialised hardware to run. I imagine this isn’t a totally unique scenario, because Python is used as a “glue” language in all sorts of domains. What I want is a button in my IDE, which is running locally, that I can click to run one of these tests —remotely, on my specialised hardware— so that I don’t have to switch to a terminal or a different GUI to run the test, type in the test name, etc.

Visual Studio Code provides a “Run Test” button right in the IDE

Should be simple, right? Surely the IDE allows me to specify a custom script to run instead of the built-in test runner?

No! Neither PyCharm nor Visual Studio Code, two of the most popular IDEs for Python code, allow this. Instead they provide specific support for these Python test runners: nose, pytest, and unittest. Coming from Emacs, this seems so limiting… and writing a custom plugin for PyCharm or VS Code is so daunting!

So what we’ve done is create a pytest plugin that allows us to use the IDE’s existing support for pytest. This pytest plugin is ~100 lines of Python code (not counting the code specific to our own tests/hardware; just the infrastructural code to integrate with pytest). It allows us to run a single test, at the click of a button, from any IDE that supports pytest.

Note: On the remote hardware we don’t use pytest to run the tests (but you could if you wanted to). We are merely using pytest as the mechanism to integrate with the IDE. You could even use pytest to collect tests that aren’t written in Python!

The code is here. This is the script we run locally (for example from a terminal) to kick off our tests on the remote platform. You can ignore most of it; the relevant pytest bits are in the highlighted lines.

At a high level, we implement the following pytest hooks:

• pytest_sessionstart which disables pytest’s built-in python plugin — otherwise pytest would try to run the test locally using the local python interpreter.

• pytest_addoption to teach pytest about any custom command-line arguments we need (such as the address of the hardware we want to run the test on).

• pytest_collect_file which returns our custom collector class (see next point).

• A custom pytest collector that returns our own pytest.Item subclass instead of pytest’s default pytest.Function items.

• Our own pytest.Item subclass, where we override the runtest method. Here is where you’d put your custom code to sync your test scripts to the remote hardware, run the test, and retrieve the test’s output / logs.

The code is light on comments, but the commit messages (see git blame) have detailed explanations for why we needed particular lines of code.

To load your pytest plugin, tell your IDE to add -p my_module_name to the pytest arguments. For specific configuration examples for PyCharm and VS Code, see Stb-tester’s manual.

Not pretty! But better than writing a PyCharm plugin in Java using PyCharm’s under-documented API. IDEs: Why do you make this so hard?