rackcheck:   property testing
1 What about quickcheck?
2 Reference
2.1 Generators
2.1.1 Debugging
sample
shrink
2.1.2 Core Combinators
generator/  c
gen?
make-gen
gen:  const
gen:  map
gen:  bind
gen:  filter
gen:  choice
gen:  no-shrink
gen:  let
gen:  delay
2.1.3 Basic Generators
gen:  natural
gen:  integer-in
gen:  real
gen:  one-of
gen:  boolean
gen:  char
gen:  char-letter
gen:  char-digit
gen:  char-alphanumeric
gen:  tuple
gen:  list
gen:  vector
gen:  bytes
gen:  string
gen:  symbol
gen:  hash
gen:  hasheq
gen:  hasheqv
gen:  frequency
2.1.4 Unicode Generators
gen:  unicode
gen:  unicode-letter
gen:  unicode-mark
gen:  unicode-number
gen:  unicode-punctuation
gen:  unicode-symbol
gen:  unicode-separator
2.2 Properties
property?
property
define-property
check-property
label!
config?
make-config
7.8

rackcheck: property testing

Bogdan Popa <bogdan@defn.io>

Rackcheck is a property-based testing library for Racket with support for shrinking.

I am still in the process of experimenting with the implementation of this library so things may change without notice at this point.

1 What about quickcheck?

I initially started by forking the quickcheck library to add support for shrinking, but found that I would have to make many breaking changes to get shrinking to work the way I wanted so I decided to start from scratch instead.

2 Reference

 (require rackcheck) package: rackcheck

2.1 Generators

Generators produce arbitrary values based upon certain constraints. The generators provided by this library can be mixed and matched in order to produce complex values suitable for any domain.

By convention, all generators and combinators are prefixed with gen:.

2.1.1 Debugging

The following functions come in handy when debugging generators. Don’t use them to produce values for your tests.

procedure

(sample g n [rng])  (listof any/c)

  g : gen?
  n : exact-positive-integer?
  rng : pseudo-random-generator?
   = (current-pseudo-random-generator)
Samples n values from g.

procedure

(shrink g size [rng])  
any/c (listof any/c)
  g : gen?
  size : exact-nonnegative-integer?
  rng : pseudo-random-generator?
   = (current-pseudo-random-generator)
Produces a value and all of its shrinks from g.

2.1.2 Core Combinators

The contract for generator functions. Generator functions produce a stream of values where the first value is the generated value and the rest of the values are the possible shrinks of that value from the largest to the smallest.

In general, you won’t have to write generator functions yourself. Instead, you’ll use the generators and combinators provided by this library to generate values for your domain. That said, there may be cases where you want to tightly control how values are generated or shrunk and that’s when you might reach for a custom generator.

procedure

(gen? v)  boolean?

  v : any/c

procedure

(make-gen f)  gen?

  f : generator/c
gen? returns #t when v is a generator value and make-gen creates new generator values from generator functions.

procedure

(gen:const v)  gen?

  v : any/c
Creates a generator that always returns v.

procedure

(gen:map g f)  gen?

  g : gen?
  f : (-> any/c any/c)
Creates a generator that transforms the values generated by g by applying them to f before returning them.

procedure

(gen:bind g f)  gen?

  g : gen?
  f : (-> any/c gen?)
Creates a generator that depends on the values produced by g.

When shrinking, a new shrink sequence is produced for every application of f to all of g’s shrinks.

> (define gen:list-of-trues
    (gen:bind
     gen:natural
     (lambda (len)
       (make-gen
        (lambda (rng size)
          (stream (make-list len #t)))))))
> (sample gen:list-of-trues 5)

'(() () (#t #t #t #t) (#t #t #t #t #t) (#t #t #t #t #t #t #t #t))

> (shrink gen:list-of-trues 5)

'(#t)

'(())

procedure

(gen:filter g p [max-attempts])  gen?

  g : gen?
  p : (-> any/c boolean?)
  max-attempts : (or/c exact-positive-integer? +inf.0) = 1000
Produces a generator that repeatedly generates values using g until the result of p applied to one of those values is #t or the number of attempts exceeds max-attempts.

An exception is raised when the generator runs out of attempts.

This is a very brute-force way of generating values and you should avoid using it as much as possible, especially if the range of outputs is very small compared to the domain. Take a generator for non-empty strings as an example. Instead of:

> (gen:filter
   (gen:string gen:char-alphanumeric)
               (lambda (s)
                (not (string=? s ""))))

#<procedure:...eck/gen/core.rkt:78:3>

Write:

> (gen:let ([hd gen:char-alphanumeric]
            [tl (gen:string gen:char-alphanumeric)])
   (string-append (string hd) tl))

#<procedure:...eck/gen/core.rkt:67:3>

The latter takes a little more effort to write, but it doesn’t depend on the whims of the random number generator and will always generate a non-empty string on the first try.

procedure

(gen:choice g ...+)  gen?

  g : gen?
Produces a generator that generates values by randomly choosing one of the passed-in generators.

procedure

(gen:no-shrink g)  gen?

  g : gen?
Creates a generator based on g that never shrinks.

syntax

(gen:let ([id gen-expr] ...+) body ...+)

Provides a convenient syntax for creating generators that depend on one or more other generators.

> (define gen:list-of-trues-2
    (gen:let ([len gen:natural])
      (make-list len #t)))
> (sample gen:list-of-trues-2 5)

'(() () (#t #t #t #t) (#t #t #t #t #t) (#t #t #t #t #t #t #t #t))

> (shrink gen:list-of-trues-2 5)

'(#t)

'(())

syntax

(gen:delay gen-expr)

Creates a generator that wraps and delays the execution of gen-expr until it is called. This is handy for when you need to write recursive generators.

2.1.3 Basic Generators

value

gen:natural : gen?

Generates natural numbers.

> (sample gen:natural)

'(0 0 4 5 8 20 22 39 43 20)

> (shrink gen:natural)

9

'(4 2 1 0)

procedure

(gen:integer-in lo hi)  gen?

  lo : exact-integer?
  hi : exact-integer?
Creates a generator that produces exact integers between lo and hi, inclusive. A contract error is raised if lo is greater than hi.

> (sample (gen:integer-in 1 255))

'(75 46 239 152 124 200 155 202 172 65)

> (shrink (gen:integer-in 1 255))

75

'(37 18 9 4 2 1)

value

gen:real : gen?

Generates real numbers between 0 and 1, inclusive. Real numbers do not currently shrink, but this may change in the future.

> (sample gen:real)

'(0.2904158091187683

  0.17902984405826025

  0.9348212358175817

  0.592848361775386

  0.4846099332903666

  0.7816821100632378

  0.6078124617750272

  0.788902313469835

  0.6710271948421507

  0.25158978983077135)

> (shrink gen:real)

0.2904158091187683

'()

procedure

(gen:one-of xs)  gen?

  xs : (non-empty-listof any/c)
Creates a generator that produces values randomly selected from xs.

> (define gen:letters (gen:one-of '(a b c)))
> (sample gen:letters)

'(c a b a c a c c b b)

> (shrink gen:letters)

'c

'()

value

gen:boolean : gen?

Generates boolean values.

> (sample gen:boolean)

'(#f #f #t #t #f #t #t #t #t #f)

value

gen:char : gen?

Generates ASCII characters.

> (sample gen:char)

'(#\J #\- #\ï #\u0097 #\| #\È #\u009B #\É #\« #\@)

> (shrink gen:char)

#\J

'(#\% #\u0012 #\tab #\u0004 #\u0002 #\u0001 #\nul)

Generates ASCII letters.

> (sample gen:char-letter)

'(#\e #\P #\u #\U #\G #\O #\g #\E #\u #\o)

Generates ASCII digits.

> (sample gen:char-digit)

'(#\2 #\1 #\9 #\5 #\4 #\7 #\6 #\7 #\6 #\2)

Generates alphanumeric ASCII characters.

> (sample gen:char-alphanumeric)

'(#\1 #\M #\U #\q #\g #\d #\o #\H #\2 #\7)

procedure

(gen:tuple g ...+)  gen?

  g : gen?
Creates a generator that produces heterogeneous lists where the elements are created by generating values from each g in sequence.

> (sample (gen:tuple gen:natural gen:boolean))

'((0 #f) (1 #t) (2 #t) (6 #t) (11 #f) (16 #t) (9 #f) (46 #f) (8 #t) (38 #t))

procedure

(gen:list g [#:max-length max-len])  gen?

  g : gen?
  max-len : exact-nonnegative-integer? = 128
Creates a generator that produces lists of random lengths where every element is generated using g. Shrinks by reducing the size of the list by one element every time.

> (sample (gen:list gen:natural) 5)

'(() () (2 2 3 3) (6 2 6 5 2 2 9) (2 13))

> (shrink (gen:list gen:natural))

'(5 28 18 15 24 18 24 20 7)

'((5 28 18 15 24 18 24 20)

  (5 28 18 15 24 18 24)

  (5 28 18 15 24 18)

  (5 28 18 15 24)

  (5 28 18 15)

  (5 28 18)

  (5 28)

  (5)

  ())

procedure

(gen:vector g [#:max-length max-len])  gen?

  g : gen?
  max-len : exact-nonnegative-integer? = 128
Like gen:list but for vector?s.

> (sample (gen:vector gen:natural) 5)

'(#() #() #(2 2 3 3) #(6 2 6 5 2 2 9) #(2 13))

> (shrink (gen:vector gen:natural))

'#(5 28 18 15 24 18 24 20 7)

'(#(5 28 18 15 24 18 24 20)

  #(5 28 18 15 24 18 24)

  #(5 28 18 15 24 18)

  #(5 28 18 15 24)

  #(5 28 18 15)

  #(5 28 18)

  #(5 28)

  #(5)

  #())

procedure

(gen:bytes [g #:max-length max-len])  gen?

  g : gen? = (gen:integer-in 0 255)
  max-len : exact-nonnegative-integer? = 128
Like gen:list but for bytes?s. Raises a contract error if g produces anything other than integers in the range 0 to 255 inclusive.

> (sample (gen:bytes) 5)

'(#"" #"" #"\227|\310\233" #"\253@\237\214>A\354" #"#\312")

> (shrink (gen:bytes))

#"-\357\227|\310\233\311\253@"

'(#"-\357\227|\310\233\311\253"

  #"-\357\227|\310\233\311"

  #"-\357\227|\310\233"

  #"-\357\227|\310"

  #"-\357\227|"

  #"-\357\227"

  #"-\357"

  #"-"

  #"")

procedure

(gen:string [g #:max-length max-len])  gen?

  g : gen? = gen:char
  max-len : exact-nonnegative-integer? = 128
Like gen:list but for string?s. Raises a contract error if g produces anything other than char? values.

> (sample (gen:string gen:char-letter) 5)

'("" "" "MPRq" "gEuoX" "fuee")

> (shrink (gen:string gen:char-letter))

"yMPRqGydM"

'("yMPRqGyd" "yMPRqGy" "yMPRqG" "yMPRq" "yMPR" "yMP" "yM" "y" "")

procedure

(gen:symbol [g #:max-length max-len])  gen?

  g : gen? = gen:char
  max-len : exact-nonnegative-integer? = 128
Like gen:string but for symbol?s. Raises a contract error if g produces anything other than char? values.

> (sample (gen:symbol gen:char-letter) 5)

'(|| || MPRq gEuoX fuee)

> (shrink (gen:symbol gen:char-letter))

'yMPRqGydM

'(yMPRqGyd yMPRqGy yMPRqG yMPRq yMPR yMP yM y ||)

procedure

(gen:hash k g ...+ ...+)  gen?

  k : any/c
  g : gen?

procedure

(gen:hasheq k g ...+ ...+)  gen?

  k : any/c
  g : gen?

procedure

(gen:hasheqv k g ...+ ...+)  gen?

  k : any/c
  g : gen?
These functions create generators that produce hashes, hasheqs and hasheqvs, respectively, where each key maps to a value generated from its associated generator.

> (sample (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)) 5)

'(#hasheq((a . 0) (b . ""))

  #hasheq((a . 1) (b . "u"))

  #hasheq((a . 3) (b . "GOg"))

  #hasheq((a . 9) (b . "u"))

  #hasheq((a . 7) (b . "XafFfaeLg")))

> (shrink (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)))

'#hasheq((a . 9) (b . "PuUGO"))

'(#hasheq((a . 4) (b . "PuUGO"))

  #hasheq((a . 2) (b . "PuUGO"))

  #hasheq((a . 1) (b . "PuUGO"))

  #hasheq((a . 0) (b . "PuUGO"))

  #hasheq((a . 9) (b . "PuUG"))

  #hasheq((a . 4) (b . "PuUG"))

  #hasheq((a . 2) (b . "PuUG"))

  #hasheq((a . 1) (b . "PuUG"))

  #hasheq((a . 0) (b . "PuUG"))

  #hasheq((a . 9) (b . "PuU"))

  #hasheq((a . 4) (b . "PuU"))

  #hasheq((a . 2) (b . "PuU"))

  #hasheq((a . 1) (b . "PuU"))

  #hasheq((a . 0) (b . "PuU"))

  #hasheq((a . 9) (b . "Pu"))

  #hasheq((a . 4) (b . "Pu"))

  #hasheq((a . 2) (b . "Pu"))

  #hasheq((a . 1) (b . "Pu"))

  #hasheq((a . 0) (b . "Pu"))

  #hasheq((a . 9) (b . "P"))

  #hasheq((a . 4) (b . "P"))

  #hasheq((a . 2) (b . "P"))

  #hasheq((a . 1) (b . "P"))

  #hasheq((a . 0) (b . "P"))

  #hasheq((a . 9) (b . ""))

  #hasheq((a . 4) (b . ""))

  #hasheq((a . 2) (b . ""))

  #hasheq((a . 1) (b . ""))

  #hasheq((a . 0) (b . "")))

procedure

(gen:frequency frequencies)  gen?

  frequencies : (non-empty-listof (cons/c exact-positive-integer? gen?))
Creates a generator that generates values using a generator that is randomly picked from frequencies. Generators with a higher weight will get picked more often.

> (sample (gen:frequency `((5 . ,gen:natural)
                           (2 . ,gen:boolean))))

'(0 #t 3 7 4 14 9 #f 51 46)

> (shrink (gen:frequency `((5 . ,gen:natural)
                           (2 . ,gen:boolean))))

5

'(2 1 0)

2.1.4 Unicode Generators

 (require rackcheck/gen/unicode) package: rackcheck

These generators produce valid unicode char?s.

> (sample gen:unicode)

'(#\U0003C314

  #\耎

  #\U000D7AFB

  #\ꩧ

  #\㙗

  #\登

  #\U00050685

  #\⏲

  #\U000DA289

  #\U000A1C2C)

> (sample gen:unicode-letter)

'(#\醃 #\𨲠 #\𐍙 #\ꡩ #\𐙕 #\啖 #\項 #\𥌽 #\𥐖 #\𧻹)

> (sample gen:unicode-mark)

'(#\𑂀 #\᪶ #\︢ #\󠅼 #\󠅕 #\⃮ #\ौ #\͗ #\󠄕 #\َ)

> (sample gen:unicode-number)

'(#\𐋤 #\㊼ #\፪ #\𐮬 #\𞣊 #\꠰ #\🄉 #\𐋧 #\𖭓 #\𑁝)

> (sample gen:unicode-punctuation)

'(#\᪠ #\⸿ #\﹍ #\⸣ #\𐫶 #\⦆ #\⸓ #\* #\\ #\𑇍)

> (sample gen:unicode-symbol)

'(#\⣢ #\🢃 #\𝄯 #\🝛 #\♹ #\🌸 #\🝖 #\🎀 #\㌿ #\🐎)

> (sample gen:unicode-separator)

'(#\u202F

  #\u2001

  #\u202F

  #\u2000

  #\u2002

  #\u2008

  #\u2028

  #\u2007

  #\u00A0

  #\u2004)

2.2 Properties

procedure

(property? v)  boolean?

  v : any/c
Returns #t when v is a property.

syntax

(property ([id gen-expr] ...) body ...+)

Declares a property where the inputs are one or more generators.

> (property ([xs (gen:list gen:natural)])
    (check-equal? (reverse (reverse xs)) xs))

#<prop>

syntax

(define-property name
 ([id gen-expr] ...)
 body ...+)
A shorthand for (define name (property ...)).

syntax

(check-property maybe-config prop-expr)

 
maybe-config = 
  | config-expr
Tries to falsify the property p according to the config. If not provided, then a default configuration with a random seed value is used.

> (check-property
   (property ([xs (gen:list gen:natural)])
     (check-equal? (reverse (reverse xs)) xs)))

  ✓ property unnamed passed 100 tests.

> (check-property
   (property ([xs (gen:list gen:natural)])
     (check-equal? (reverse xs) xs)))

--------------------

FAILURE

location:   eval:54:0

name:       unnamed

seed:       106608640

actual:     '(7 1)

expected:   '(1 7)

Failed after 5 tests:

  xs = (1 7 5 4 9 6 7 9 4)

Shrunk:

  xs = (1 7)

--------------------

procedure

(label! s)  void?

  s : (or/c false/c string?)
Keeps track of how many times s appears in the current set of tests. Use this to classify and keep track of what categories the inputs to your properties fall under.

Does nothing when s is #f.

> (check-property
   (property
    ([a gen:natural]
     [b gen:natural])
    (label!
     (case a
      [(0)  "zero"]
      [else "non-zero"]))
    (+ a b)))

  ✓ property unnamed passed 100 tests.

  Labels:

  ├ 98.00% non-zero

    2.00% zero

procedure

(config? v)  boolean?

  v : any/c
Returns #t when v is a config value.

procedure

(make-config [#:seed seed    
  #:tests tests    
  #:size size    
  #:deadline deadline])  config?
  seed : (integer-in 0 (sub1 (expt 2 31))) = ...
  tests : exact-positive-integer? = 100
  size : (-> exact-positive-integer? exact-nonnegative-integer?)
   = (lambda (n) (expt (sub1 n) 2))
  deadline : (>=/c 0)
   = (+ (current-inexact-milliseconds) (* 60 1000))
Creates values that control the behavior of check-property.