al2-test-runner – Alternative way of running rackunit test suites
1 Motivating Example
2 API Documentation
run-tests
skip-test
7.8

al2-test-runner – Alternative way of running rackunit test suites

Alex Harsányi

 (require al2-test-runner) package: al2-test-runner

This package provides an alternative way of running rackunit tests, providing better visibility and improved reporting of test results. The run-tests function is a drop-in replacement of the equivalent function from rackunit/textui, providing the following benefits:

The package does not define another testing framework, instead it runs rackunit tests which are organized in test-suites and test-cases.

1 Motivating Example

When working on an application or package, you might disable a broken test temporarily, but once you do, there is no longer an indication that the test has been disabled. The test suite passes, and if there is a large number of tests, it is easy to miss that some were disabled:

> (require rackunit rackunit/text-ui)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      ; (test-case "Second Test Case" (check-equal? 1 0))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests a-test-suite 'verbose)

2 success(es) 0 failure(s) 0 error(s) 2 test(s) run

0

The commented out test might be easy to spot in the previous example, but in a bigger application with many tests, it is easy to miss. The al2-test-runner package provides an alternative: the test can be disabled by adding it to the exclusion list in run-tests. The test is still known to the test system and it is now reported as skipped.

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 0))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    ; Second test case is broken, will fix it later
    #:exclude '(("A Test Suite" "Second Test Case"))
    a-test-suite)

*** Testsuite A Test Suite

First Test Case: ok (0.14 ms)

Second Test Case: skipped

Third Test Case: ok (0.1 ms)

*** Testsuite A Test Suite completed in 0.24 ms

*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1

*** Writing results to my-package-test-results.xml

In addition, this version of run-tests will print out all the tests that are run along with their pass/fail status and will also write this information to a file, in standard JUnit test format. This file can be imported in various test management systems.

2 API Documentation

procedure

(run-tests [#:package package    
  #:results-file results-file    
  #:only only    
  #:exclude exclude]    
  test-suite ...)  any/c
  package : string? = "unnamed package"
  results-file : (or/c path-string? #f) = #f
  only : (or/c #f (listof (listof string?))) = #f
  exclude : (or/c #f (listof (listof string?))) = #f
  test-suite : test-suite?
Run the tests in test-suites and report the results to the standard output. For each test case, the name followed by the strings "ok", "failed", "error" or "skipped" is printed, along with the time in milliseconds it took to execute the test.

package is a string that is used as a package name when results are written to file. JUnit tests are grouped into packages, and the XML file format expects that, but there is no higher construct than a test-suite in rackunit, so the package name needs to be provided as a parameter to run-tests.

When results-file is specified, the test results are written to the specified file in JUnit XML format. If a file name is not specified, the results are not written to file.

When only is present, it needs to be a list of test suite names followed by test case names, and only these tests will be run. This is intended to be used when debugging a single test in a larger test suite. Tests that are not run will be reported as skipped. If the list of test cases is empty, and. only the test suite name is specified, all the tests cases in that test suite are run.

Here is an example of how to run only one test case in the test suite:

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 1))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    #:only '(("A Test Suite" "First Test Case"))
    a-test-suite)

*** Testsuite A Test Suite

First Test Case: ok (0.14 ms)

Second Test Case: skipped

Third Test Case: skipped

*** Testsuite A Test Suite completed in 0.14 ms

*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 2

*** Writing results to my-package-test-results.xml

When exclude is preset, it needs to be a list of test case names followed by test suite names. These tests will not be executed, and instead will be reported as skipped both on the standard output and in the output file. Note that tests can also be skipped using skip-test. This is indented for disabling tests which do not pass and they cannot be fixed immediately. If the list of test cases is empty, and. only the test suite name is specified, all the tests cases in that test suite are excluded.

Here is an example, that will not run "A Test Case":

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 1))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    #:exclude '(("A Test Suite" "First Test Case"))
    a-test-suite)

*** Testsuite A Test Suite

First Test Case: skipped

Second Test Case: ok (0.15 ms)

Third Test Case: ok (0.09 ms)

*** Testsuite A Test Suite completed in 0.24 ms

*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1

*** Writing results to my-package-test-results.xml

procedure

(skip-test)  any/c

This function can be used inside test cases to skip a test because some required testing condition cannot be met. It can be used, for example, when tests require test data files and these data files are not available and it is a better indicator that a test suite did not actually fail but it didn’t pass either.

Here is an example where the second test case is skipped:

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (unless (= 1 0) (skip-test)))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    a-test-suite)

*** Testsuite A Test Suite

First Test Case: ok (0.12 ms)

Second Test Case: skipped (0.34 ms)

Third Test Case: ok (0.11 ms)

*** Testsuite A Test Suite completed in 0.58 ms

*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1

*** Writing results to my-package-test-results.xml