Common Workflow Language testing framework
This is a testing tool for checking the output of Tools and Workflows described with the Common Workflow Language. Among other uses, it is used to run the CWL conformance tests.
This is written and tested for Python 3.8, 3.9, 3.10, 3.11, and 3.12.
Install
Installing the official package from PyPi
pip install cwltest
Or from bioconda
conda install -c bioconda cwltest
Or from source
git clone https://github.com/common-workflow-language/cwltest.git
cd cwltest && pip install .
Run on the command line
Simple command:
cwltest --test test-descriptions.yml --tool cwl-runner
Generate conformance badges using cwltest
To make badges that show the results of the conformance test, you can generate JSON files for https://badgen.net by using –badgedir option
To generate JSON files:
cwltest --test test-descriptions.yml --tool cwl-runner --badgedir badges
...
$ cat badges/command_line_tool.json | jq .
{
"subject": "command_line_tool",
"status": "100%",
"color": "green"
}
Once you upload JSON file to a server, you make a badge by using a link like https://badgen.net/https/path/to/generated/json or https://flat.badgen.net/https/path/to/generated/json (for flat badges).
Here is an example of markdown to add a badge:
![test result](https://flat.badgen.net/https/path/to/generated/json?icon=commonwl)
Command Line Options
cwltest
Common Workflow Language testing framework
usage: cwltest [-h] --test TEST [--basedir BASEDIR] [-l] [-n N] [-s S] [-N N]
[-S S] [--tool TOOL] [--only-tools] [--tags TAGS]
[--exclude-tags EXCLUDE_TAGS] [--show-tags]
[--junit-xml JUNIT_XML] [--junit-verbose]
[--test-arg cache==--cache-dir] [-j J] [--verbose]
[--classname CLASSNAME] [--timeout TIMEOUT]
[--badgedir BADGEDIR] [--version]
...
- args
arguments to pass first to tool runner
- -h, --help
show this help message and exit
- --test <test>
YAML file describing test cases
- --basedir <basedir>
Basedir to use for tests
- -l
List tests then exit
- -n <n>
Run specific tests, format is 1,3-6,9
- -s <s>
Run specific tests using their short names separated by comma
- -N <n>
Exclude specific tests by number, format is 1,3-6,9
- -S <s>
Exclude specific tests by short names separated by comma
- --tool <tool>
CWL runner executable to use (default ‘cwl-runner’
- --only-tools
Only test CommandLineTools
- --tags <tags>
Tags to be tested
- --exclude-tags <exclude_tags>
Tags not to be tested
- --show-tags
Show all Tags.
- --junit-xml <junit_xml>
Path to JUnit xml file
- --junit-verbose
Store more verbose output to JUnit XML file by not passing ‘–quiet’ to the CWL runner.
- --test-arg <cache==--cache-dir>
Additional argument given in test cases and required prefix for tool runner.
- -j <j>
Specifies the number of tests to run simultaneously (defaults to one).
- --verbose
More verbose output during test run.
- --classname <classname>
Specify classname for the Test Suite.
- --timeout <timeout>
Time of execution in seconds after which the test will be skipped. Defaults to 600 seconds (10.0 minutes).
- --badgedir <badgedir>
Create JSON badges and store them in this directory.
- --version
show program’s version number and exit