2.2.1 Device resources🔗ℹ

Each call makes independent resources with their own stream:

> (define loader (device-resources))

> (define trainer (device-resources))

> (eq? loader trainer)

#f

> (map resources-device (list loader trainer))

'(0 0)

(device-count) is the first number that is not a device:

> (device-count)

1

> (device-resources #:device (device-count))

device-resources: no device 1 among 1

Accepting either a device number or resources:

> (define (as-resources where)     (if (device-resources? where)         where         (current-device-resources where)))

> (eq? (as-resources 0) (current-device-resources))

#t

> (eq? (as-resources loader) loader)

#t

Released resources still satisfy it:

> (define finished (with-device-resources ([r (device-resources)]) r))

> (device-resources? finished)

#t

Naming the device in a log line:

> (printf "training on cuda:~a\n" (resources-device trainer))

training on cuda:0

Checking that two resources share a device:

> (define (same-device? a b)     (= (resources-device a) (resources-device b)))

> (same-device? loader trainer)

#t

> (resources-device finished)

0

Timing GPU work, which returns once queued:

> (define (milliseconds-on r thunk)     (define start (current-inexact-milliseconds))     (thunk)     (resources-sync! r)     (- (current-inexact-milliseconds) start))

> (real? (milliseconds-on trainer void))

#t

Waiting in a thread of its own:

> (thread-wait    (thread (lambda ()              (resources-sync! trainer)              (displayln "the trainer's stream has drained"))))

the trainer's stream has drained

> (resources-sync! finished)

resources-sync!: the device resources on device 0 were

released

Each thread gets its own:

> (define (worker-resources)     (define answer (make-channel))     (thread (lambda () (channel-put answer (current-device-resources))))     (channel-get answer))

> (define a (worker-resources))

> (define b (worker-resources))

> (list (eq? a b) (eq? a (current-device-resources)))

'(#f #f)

Releasing the default makes the next call create a new one:

> (define before (current-device-resources))

> (with-device-resources ([r before])     (resources-sync! r))

> before

#<device-resources device 0 released>

> (eq? before (current-device-resources))

#f

The finalizer alone is correct: it releases unreachable resources at some collection. The form gives their lifetime a clear timeline, a release at a known point, because resources hold GPU and driver state the collector sees only as an estimate: a CUDA stream, a job worker thread and lazily created cuBLAS, cuSOLVER and cuSPARSE handles. Each resources object charges the collector about 1 MiB until it is released, so the defaults of short-lived threads are collected soon after the threads end; a cuBLAS handle alone holds several MiB of device memory. Scoped, they are released when the body exits:

> (define unscoped (device-resources))

> (define scoped (with-device-resources ([r (device-resources)]) r))

> (list unscoped scoped)

'(#<device-resources device 0> #<device-resources device 0 released>)

Released even when the body fails:

> (define (run-batch items)     (with-device-resources ([r (device-resources)])       (for ([item (in-list items)])         (when (negative? item)           (error 'run-batch "bad item ~a" item)))       (resources-sync! r)       (length items)))

> (run-batch '(1 2 3))

3

> (run-batch '(1 -2 3))

run-batch: bad item -2

Several bindings, released together:

> (define-values (left right)     (with-device-resources ([left (device-resources)]                             [right (device-resources)])       (values left right)))

> (list left right)

'(#<device-resources device 0 released> #<device-resources device 0 released>)

> (resources-sync! left)

resources-sync!: the device resources on device 0 were

released

2.2.2 Devices🔗ℹ

Spreading workers round robin:

> (define (device-for-worker i)     (modulo i (device-count)))

> (map device-for-worker '(0 1 2 3))

'(0 0 0 0)

Resources for every device:

> (define per-device     (for/list ([d (in-range (device-count))])       (device-resources #:device d)))

> (map resources-device per-device)

'(0)

2.2.3 The default memory resource🔗ℹ

RMM’s current memory resource is per device and shared by every library in the process. The first resources made on a device replace RMM’s default (cudaMalloc, which synchronizes the device) with its CUDA async memory resource, a stream-ordered cudaMallocAsync pool. Any other resource, installed before or after, is left alone; a plain rmm::mr::cuda_memory_resource set beforehand looks like the default and is replaced. A device without memory-pool support keeps the default.

2.2.4 Errors🔗ℹ

• 'out-of-memory: an allocation failed;

• 'cuda: a CUDA call failed, for example because no driver is installed;

• 'logic: the call was refused before reaching the GPU, for example a device that does not exist or resources already released;

• 'generic: anything else.

Dispatching on the kind:

> (define (describe-failure thunk)     (with-handlers ([exn:fail:raft?                      (lambda (e)                        (case (exn:fail:raft-kind e)                          [(out-of-memory) 'free-memory-and-retry]                          [(logic) 'fix-the-call]                          [else 'report]))])       (thunk)       'ok))

> (describe-failure (lambda () (current-device-resources)))

'ok

> (describe-failure (lambda () (device-resources #:device 99)))

'fix-the-call

Recovering the procedure name from the message:

> (with-handlers ([exn:fail:raft?                    (lambda (e)                      (match-define (list who _ ...)                        (string-split (exn-message e) ": "))                      who)])     (resources-sync! finished))

"resources-sync!"

2.2.5 Exiting while jobs run🔗ℹ

The value is read when a job is made, on the calling thread, so parameterize covers the jobs made in its body and in threads created there. The exit logs to the 'raft topic: the jobs in flight and the time waited at 'debug, and a skipped teardown, with the jobs and the releases still running, at 'warning. If only a place, and not the main place, instantiates raft, the exit still waits, but native destructors registered after the first job or release may run first. Instantiating it from a parallel thread of the main place counts as the main place. A release still running on the library threads that tear resources and arrays down holds the exit the same way, for up to one second after the exit begins or the last job in flight finishes, whichever is later: up to the largest bound in flight plus one second. The wait covers any release made before the exit decides. The exit decides as soon as no job and no release is running, so a release made after that, such as a place’s teardown that starts once everything else has finished, is not waited for. It runs on its caller and can run while the native static destructors run, or be cut off by the process exit. A job or a release that itself calls the C exit skips the teardown, but the C atexit handlers and static destructors registered after the first job or release run first, on that thread, while other jobs and releases keep running.

A thread created inside parameterize keeps the value for its jobs:

> (define seen (make-channel))

> (parameterize ([current-exit-wait +inf.0])     (thread (lambda () (channel-put seen (current-exit-wait)))))

#<thread>

> (channel-get seen)

+inf.0

A negative or non-real value is refused:

> (current-exit-wait -1)

current-exit-wait: expected a non-negative real number of

seconds, given -1

> (parameterize ([current-exit-wait +nan.0]) 'unreached)

current-exit-wait: expected a non-negative real number of

seconds, given +nan.0

2.2.6 Version and ABI🔗ℹ

Comparing releases numerically:

> (match-define (list year month _)     (map string->number (string-split (raft-version) ".")))

> (list year month)

'(26 8)

> (>= (+ (* 100 year) month) 2608)

#t

Refusing to start on another release:

> (define (require-raft-release! wanted)     (unless (string=? (raft-version) wanted)       (error 'my-pipeline "built for RAFT ~a, but this is RAFT ~a"              wanted (raft-version))))

> (require-raft-release! "26.08.00")

> (require-raft-release! "26.10.00")

my-pipeline: built for RAFT 26.10.00, but this is RAFT

26.08.00

In a match pattern, raft-abi takes the tag’s fields by keyword. Every keyword is optional and fields not named are ignored; an unknown keyword is a syntax error.

> (match-define (raft-abi #:raft raft #:cuda-runtime cuda-runtime) (raft-abi))

> (list raft cuda-runtime)

'("26.08.00" "13.2")

A program tested on one release can say so on another:

> (define (support-status abi)     (match abi       [(raft-abi #:raft "26.08.00") 'supported]       [(raft-abi #:raft other) (list 'untested other)]))

> (support-status (raft-abi))

'supported

> (support-status (struct-copy raft-abi (raft-abi) [raft "26.10.00"]))

'(untested "26.10.00")

Checking a recorded tag: a match, a different RAFT release, and a handle layout change a version check would miss:

> (define (abi-mismatches built-against)     (for/list ([field (in-list (list raft-abi-version raft-abi-raft raft-abi-rmm                                      raft-abi-cccl raft-abi-handle-size))]                [name (in-list '(version raft rmm cccl handle-size))]                #:unless (equal? (field built-against) (field (raft-abi))))       name))

> (abi-mismatches (raft-abi))

'()

> (abi-mismatches (struct-copy raft-abi (raft-abi) [raft "26.10.00"]))

'(raft)

> (abi-mismatches (struct-copy raft-abi (raft-abi) [handle-size 40]))

'(handle-size)

A one-line support report:

> (match-let ([(raft-abi #:raft raft #:rmm rmm #:cccl cccl) (raft-abi)])     (format "raft=~a rmm=~a cccl=~a" raft rmm cccl))

"raft=26.08.00 rmm=26.08.00 cccl=3.4.3"