Pytest plugin

cwltest can also be used as a Pytest 7.x or 8.x plugin. The CWL test filename must end with .cwltest.yml or .cwltest.yaml.

In this case, the simple command:

cwltest --test conformance_xxx.cwltest.yml --tool cwl-runner

becomes:

pytest conformance_xxx.cwltest.yml --cwl-runner cwl-runner

Command Line Options

pytest

usage: pytest [--cwl-runner CWL_RUNNER] [--cwl-runner-verbose]
              [--cwl-badgedir CWL_BADGEDIR] [--cwl-include CWL_INCLUDE]
              [--cwl-exclude CWL_EXCLUDE] [--cwl-tags CWL_TAGS]
              [--cwl-exclude-tags CWL_EXCLUDE_TAGS] [--cwl-args CWL_ARGS]
              [--cwl-test-arg CWL_TEST_ARG] [--cwl-basedir CWL_BASEDIR]

--cwl-runner <cwl_runner>

Name of the CWL runner to use.

--cwl-runner-verbose

If set, don’t pass –quiet to the CWL runner.

--cwl-badgedir <cwl_badgedir>

Create badge JSON files and store them in this directory.

--cwl-include <cwl_include>

Run specific CWL tests using their short names separated by comma

--cwl-exclude <cwl_exclude>

Exclude specific CWL tests using their short names separated by comma

--cwl-tags <cwl_tags>

Tags to be tested.

--cwl-exclude-tags <cwl_exclude_tags>

Tags not to be tested.

--cwl-args <cwl_args>

one or more arguments to pass first to tool runner (separated by spaces)

--cwl-test-arg <cwl_test_arg>

Additional argument given in test cases and required prefix for tool runner.

--cwl-basedir <cwl_basedir>

Basedir to use for tests

Converting cwltest options to pytest options

The table below details all the available command conversions between the two formats.

Feature	cwltest	pytest
YAML file describing test cases	--test conformance_xxx.cwltest.yml	conformance_xxx.cwltest.yml
CWL runner executable to use	--tool CWL_RUNNER	--cwl-runner CWL_RUNNER
Specifies the number of tests to run simultaneously	-j CORES	-n CORES [1]
Automatically scale the number of tests to run simultaneously	UNSUPPORTED	-n auto [1]
Only run one test at a time (good for debugging cwltest itself)	-j 1 (or leave out -j)	-n 0 -s [1]
Time of execution in seconds after which the test will be skipped	--timeout TIMEOUT	--timeout TIMEOUT [3]
List tests then exit	-l	--collect-only
Run specific tests using their short names	-s TEST_NAME[,TEST_NAME…]	--cwl-include TEST_NAME[,TEST_NAME…]
Exclude specific tests by short names	-S TEST_NAME[,TEST_NAME…]	--cwl-exclude TEST_NAME[,TEST_NAME…]
Tags to be tested	--tags TAG[,TAG…]	--cwl-tags TAG[,TAG…]
Tags not to be tested	--exclude-tags TAG[,TAG…]	--cwl-exclude-tags TAG[,TAG…]
Path to JUnit xml file	--junit-xml PATH	--junit-xml=PATH [4]
More verbose output during test run	--verbose	-v[vv]
Additional argument given in test cases and required prefix for tool runner	--test-arg ARG_NAME==ARG_PREFIX	--cwl-test-arg ARG_NAME==ARG_PREFIX
Arguments to pass first to tool runner	cwltest -- ARG [ARG …]	--cwl-args "ARG [ARG …]"
Only test CommandLineTools	--only-tools	UNSUPPORTED
Show all tags	--show-tags	UNSUPPORTED
Store more verbose output to JUnit xml file	--junit-verbose	--cwl-runner-verbose [4]
Specify classname for the Test Suite	--classname CLASS_NAME	UNSUPPORTED

[1] (1,2,3)

Requires pytest-xdist. See Running tests across multiple CPUs.

[2]

-s is a shortcut for --capture=no, also helps with debugging cwltest or the cwltest plugin to pytest.

[3]

Requires pytest-timeout. Note: even if pytest-timeout is installed, there is no default timeout. This is different than cwltest’s default timeout of 10 minutes.

Differences in the XML output

cwltest --junit-xml output

• top-level <testsuites> element has the elapsed time, and counts (errors, failures, skipped, and total)

• singular <testsuite> sub-element the same attributes as the top-level <testsuites> plus name which is the basename of the YAML test file

• each <testcase> element has the follow attributes

  • name: the doc string

  • class: the tags

  • file: the test ID

  • url: like “cwltest:conformance_tests#1” (contains the basename of the YAML test file)

  • time: the elapsed time

• <testcase> elements always contain the following sub-elements, regardless of outcome

  • <system-out>: the output object

  • <system-err>: stderr (docker pull, other warnings, and errors)

• <testcase> elements for failed test cases do not have a <failure> sub-element

pytest with cwltest plugin XML output

• top-level <testsuites> element has no attributes

• singular <testsuite> sub-element has the same attributes as the cwltest XML version along with these additional attributes

  • name: default is pytest (can be customized with the pytest INI option junit_suite_name)

  • timestamp="2023-01-08T11:39:07.425159"

  • hostname: the hostname of the machine where the tests ran

• inside the <testsuite> is a <properties>..</properties> element with two <property name="…" value="…" /> elements. But this does not work with pytest-xdist.

  • runner: the name of the CWL runner

  • runner_extra_args: the value of –cwl-args`

• each <testcase> element has the following attributes

  • classname: always the name of the YAML file (conformance_test_v1.2.cwltest.yaml)

  • name: the test ID

  • time: the elapsed time

• <testcase> elements for failed test cases do have a <failure> sub-element with a message attribute containing the cwltest.plugin.CWLItem.repr_failure() output. This text is repeated as the content of the <failure> element. The presensce of <system-out> and <system-err> sub-elements varies. [4]

[4] (1,2,3)

Depending on the value of the pytest INI option junit_logging, then <system-out> and <system-err> sub-elements will be generated. However the default value for junit_logging is no, so to get either of these pick one from the full list. You can set junit_logging in a configuration file or on the command line: pytest -o junit_logging=out-err.