Running tests on remote hardware from your IDE, with pytest

By David Röthlisberger. Tweet your comments @drothlis.

Published 28 Jan 2020.

At Stb-tester we have test scripts, written in Python, that require specialised hardware to run. I imagine this isn’t a totally unique scenario, because Python is used as a “glue” language in all sorts of domains. What I want is a button in my IDE, which is running locally, that I can click to run one of these tests —remotely, on my specialised hardware— so that I don’t have to switch to a terminal or a different GUI to run the test, type in the test name, etc.

Visual Studio Code provides a “Run Test” button right in the IDE

Should be simple, right? Surely the IDE allows me to specify a custom script to run instead of the built-in test runner?

No! Neither PyCharm nor Visual Studio Code, two of the most popular IDEs for Python code, allow this. Instead they provide specific support for these Python test runners: nose, pytest, and unittest. Coming from Emacs, this seems so limiting… and writing a custom plugin for PyCharm or VS Code is so daunting!

So what we’ve done is create a pytest plugin that allows us to use the IDE’s existing support for pytest. This pytest plugin is ~100 lines of Python code (not counting the code specific to our own tests/hardware; just the infrastructural code to integrate with pytest). It allows us to run a single test, at the click of a button, from any IDE that supports pytest.

Note: On the remote hardware we don’t use pytest to run the tests (but you could if you wanted to). We are merely using pytest as the mechanism to integrate with the IDE. You could even use pytest to collect tests that aren’t written in Python!

The code is here. This is the script we run locally (for example from a terminal) to kick off our tests on the remote platform. You can ignore most of it; the relevant pytest bits are in the highlighted lines.

At a high level, we implement the following pytest hooks:

The code is light on comments, but the commit messages (see git blame) have detailed explanations for why we needed particular lines of code.

To load your pytest plugin, tell your IDE to add -p my_module_name to the pytest arguments. For specific configuration examples for PyCharm and VS Code, see Stb-tester’s manual.

Not pretty! But better than writing a PyCharm plugin in Java using PyCharm’s under-documented API. IDEs: Why do you make this so hard?