nanoarrow: Arrow IPC for Racket🔗ℹ

1 Installation and platform notes🔗ℹ

Install the package from a checkout with:

  raco pkg install --auto --link /path/to/nanoarrow

The Racket package itself is pure FFI; you also need the nanoarrow shared libraries (libnanoarrow_shared and libnanoarrow_ipc_shared, version 0.9.x):

• macOS (Homebrew):
    brew install nanoarrow
  The loader automatically probes /opt/homebrew/opt/nanoarrow/lib and /usr/local/opt/nanoarrow/lib when plain dlopen fails, so no environment variable is needed in the common case. Setting DYLD_FALLBACK_LIBRARY_PATH also works.

• Linux: nanoarrow is generally not in distribution package managers; build it from the apache-arrow-nanoarrow-0.9.0 release:

    cmake -S apache-arrow-nanoarrow-apache-arrow-nanoarrow-0.9.0 -B build -DCMAKE_BUILD_TYPE=Release
    cmake --build build && cmake --install build --prefix ~/.nanoarrow

  Make the libraries findable via LD_LIBRARY_PATH, or point the NANOARROW_LIBRARY_DIR environment variable at the install prefix’s lib directory.

The library search order is: NANOARROW_LIBRARY_DIR (if set), the plain platform loader (which honors DYLD_FALLBACK_LIBRARY_PATH and LD_LIBRARY_PATH), then the well-known prefixes listed above. The fallback probing exists because processes spawned by tooling (for example raco test workers on macOS) do not always inherit DYLD_* variables.

2 Data model🔗ℹ

A record batch is described by two transparent structs:

3 Type mapping🔗ℹ

Floating-point columns are flvectors of IEEE doubles; a 'float32 column holds values rounded through IEEE single precision. Integer values must fit their column type (e.g. an 'int8 column rejects 1000). Everything else—decimals, lists, structs, timestamps, dictionaries, and so on—is rejected with a clear "unsupported Arrow type" error when encountered in an input file.

4 Nulls policy (v0.1)🔗ℹ

Nullability is representable in the Arrow schema: each arrow-field carries a nullable flag that round-trips through the IPC format. A column value that actually contains a null, however, is not representable in this version: reading or writing one raises exn:fail with a message naming the column. Columns must be fully populated. (A single sentinel-based representation cannot preserve the exact-integer typing of integer columns or the flvector representation of float columns; both options were considered and rejecting nulls was chosen for v0.1.)

5 API🔗ℹ

6 Worked example🔗ℹ

(require nanoarrow)

(define batch

(record-batch

3

(list (arrow-field "name" 'string #t)

(arrow-field "score" 'float64 #f)

(arrow-field "count" 'int32 #t))

(vector (vector "ada" "grace" "linus")

(flvector 9.5 8.0 7.25)

(vector 1 2 3))))

(define file-bytes (record-batch->ipc-file batch))

(define batch-2 (ipc-file->record-batch file-bytes))

(record-batch-num-rows batch-2)

(vector-ref (record-batch-columns batch-2) 1)

To write a real .arrow file, just write the bytes:

(with-output-to-file "example.arrow" #:exists 'truncate

(lambda () (write-bytes file-bytes)))

7 Documented limits🔗ℹ

• Primitive column types only (see Type mapping); any other type raises a clear error rather than crashing.

• Exactly one record batch per payload is read; extra record batches in the input are ignored.

• Columns containing null values are rejected (see Nulls policy (v0.1)).

• Little-endian hosts (all common current platforms).

• nanoarrow 0.9.x shared libraries; the FFI layer checks the library version at load time.