Testing infrastructure#
Our testing infrastructure uses pytest
Run the tests with a single environment#
You can run the test suite with the default environment like so:
$ tox
You can specify a specific environment like so:
# Run the tests with Python 3.9, Sphinx 4
$ tox -e py39-sphinx4
List all test environments#
To list all of the test environments to may choose from, run:
$ tox -a
Each one corresponds to a test environment or a task to build the documentation, and has its own virtual environment.
See the tox.ini
file in the repository’s root for details of all available development environments.
Supply arguments to pytest#
To supply arguments to pytest, use --
and pass arguments after.
For example:
$ tox -- -x
Re-build an environment from scratch#
By default, tox
will only install the necessary environment once.
If you’d like to force a re-build, use the -r
parameter. For example:
$ tox -r -e py38-sphinx3
Test audits with lighthouse#
This theme uses the treosh/lighthouse-ci-action
to run some basic audits on our performance, accessibility, etc.
To preview the output of these tests:
Click on the
docs-audit
GitHub Action job from a Pull Request.Scroll to the “Audit with Lighthouse” section
If there are errors, you should see them along with the link to preview the report at
report: <URL TO PAGE>
.If there are not errors, you can list them by clicking the
uploading
section, which will reveal links to the report for each page
Test multiple Sphinx versions#
This theme is tested against the latest two major versions of Sphinx. We try to set up our regression tests such that there are no differences between these two Sphinx versions.
Unit tests#
Use the variable sphinx_build.software_versions
to conditionally run tests based on the version of Sphinx.
For example:
if sphinx_build.software_versions == ".sphinx3":
foo
elif sphinx_build.software_versions == ".sphinx4":
bar
Regression tests#
Regression tests are trickier, because updating them generally requires re-running the tests, not hand-editing code. This is cumbersome for maintenance because we have to run the test suite two times for each regression that is updated. For this reason, we have a more lightweight approach:
If a regression test differs between Sphinx versions, decide if the difference is substantial. Do we gain something meaningful by testing both major versions of Sphinx, or is the difference unrelated to our theme’s functionality?
If not substantial, then add a conditional and only run the regression test on the latest Sphinx version we support. Add a note to explain why you’re only testing against one version of Sphinx.
For example:
if sphinx_build.software_versions == ".sphinx4": # Only Sphinx4 because Sphinx3 adds an extra whitespace and isn't important file_regression.check(...)
If it is substantial, follow these steps:
To support multiple Sphinx versions with regression tests, use the
extension
key to create a different regression file for that version of Sphinx. For example:file_regression.check( html.prettify(), basename="foo", extension=f"{sphinx_build.software_versions}.html", encoding="utf8", )