Common Workflow Language testing framework

Linux Build Status Code coverage

PyPI: PyPI Version PyPI Downloads Month Total PyPI Downloads

Conda: Conda Version Conda Installs

This is a testing tool for checking the output of Tools and Workflows described with the Common Workflow Language. Among other uses, it is used to run the CWL conformance tests.

This is written and tested for Python 3.10, 3.11, 3.12, 3.13, and 3.14

Install

Installing the official package from PyPi

pip install cwltest

Or from bioconda

conda install -c bioconda cwltest

Or from source

git clone https://github.com/common-workflow-language/cwltest.git
cd cwltest && pip install  .

Run on the command line

Simple command:

cwltest --test test-descriptions.yml --tool cwl-runner

Generate conformance badges using cwltest

To make badges that show the results of the conformance test, you can generate JSON files for https://badgen.net by using –badgedir option

To generate JSON files:

cwltest --test test-descriptions.yml --tool cwl-runner --badgedir badges
...
$ cat badges/command_line_tool.json | jq .
{
  "subject": "command_line_tool",
  "status": "100%",
  "color": "green"
}

Once you upload JSON file to a server, you make a badge by using a link like https://badgen.net/https/path/to/generated/json or https://flat.badgen.net/https/path/to/generated/json (for flat badges).

Here is an example of markdown to add a badge:

![test result](https://flat.badgen.net/https/path/to/generated/json?icon=commonwl)

Custom file access module

If your CWL implementation does not write output files to a local file system location but instead to some other remote storage system, you can provide an alternate implementation of the StdFsAccess object that is able to access your storage system.

Step 1:

Implement your own class with the same public interface of the StdFsAccess object in cwltest/stdfsaccess.py (as of this writing, the methods are open, size, isfile and isdir). These methods should expect to be called with URIs from the location field of the outputs of test cases.

Define a function that, when called, returns a new instance of your object.

Step 2:

Create a Python package containing your class (or add it to an existing one).

In the package metadata, add an entry point that declares the module (in this example, my_cwl_runner.fsaccess) containing the function (in this example, get_fsaccess) that cwltest will invoke to get an object implementing the StdFsAccess interface.

In setup.py this looks like:

setup(
  ...
  entry_points={"cwltest.fsaccess": ["fsaccess=my_cwl_runner.fsaccess:get_fsaccess"]}},
  ...
)

In pyproject.toml it looks like:

[project.entry-points.'cwltest.fsaccess']
fsaccess = 'my_cwl_runner.fsaccess:get_fsaccess'

Step 3:

Install your package in the same Python environemnt as the installation of cwltest. When invoked, cwltest will query Python package metadata for a package with the cwltest.fsaccess entry point and call it to get back a custom filesystem access object.

Command Line Options

cwltest

Common Workflow Language testing framework

usage: cwltest [-h] --test TEST [--basedir BASEDIR] [--baseuri BASEURI] [-l]
               [-n N] [-s S] [-N N] [-S S] [--tool TOOL] [--only-tools]
               [--tags TAGS] [--exclude-tags EXCLUDE_TAGS] [--show-tags]
               [--junit-xml JUNIT_XML] [--junit-verbose]
               [--test-arg cache==--cache-dir] [-j J] [--verbose]
               [--classname CLASSNAME] [--timeout TIMEOUT]
               [--badgedir BADGEDIR] [--version]
               ...
args

arguments to pass first to tool runner

-h, --help

show this help message and exit

--test <test>

YAML file describing test cases

--basedir <basedir>

Basedir to use for tests

--baseuri <baseuri>

Base URI to use links in the report

-l

List tests then exit

-n <n>

Run specific tests, format is 1,3-6,9

-s <s>

Run specific tests using their short names separated by comma

-N <n>

Exclude specific tests by number, format is 1,3-6,9

-S <s>

Exclude specific tests by short names separated by comma

--tool <tool>

CWL runner executable to use (default ‘cwl-runner’

--only-tools

Only test CommandLineTools

--tags <tags>

Tags to be tested

--exclude-tags <exclude_tags>

Tags not to be tested

--show-tags

Show all Tags.

--junit-xml <junit_xml>

Path to JUnit xml file

--junit-verbose

Store more verbose output to JUnit XML file by not passing ‘–quiet’ to the CWL runner.

--test-arg <cache==--cache-dir>

Additional argument given in test cases and required prefix for tool runner.

-j <j>

Specifies the number of tests to run simultaneously (defaults to one).

--verbose

More verbose output during test run.

--classname <classname>

Specify classname for the Test Suite.

--timeout <timeout>

Time of execution in seconds after which the test will be skipped. Defaults to 600 seconds (10.0 minutes).

--badgedir <badgedir>

Create JSON badges, one for each tag (plus a computed ‘all’ tag) and store them in this directory.

--version

show program’s version number and exit

Indices and tables