FlatBuffers: Generic Reader and Builder🔗ℹ

1 A tour of the format🔗ℹ

A FlatBuffers buffer starts with a uoffset (a uint32) pointing forward to the root table. A table starts with an soffset (a signed int32) such that vtable position = table position - soffset; because the value is signed, the vtable may lie before or after the table. A vtable holds a uint16 vtable size, a uint16 table inline size, and one uint16 per field: the field’s offset from the table start, or 0 if the field is absent. Strings are uint32-length-prefixed, null-terminated UTF-8 byte vectors; vectors are uint32-length-prefixed sequences of aligned elements. Strings, vectors, sub-tables and structs are reached through unsigned offsets stored at the field’s position, each relative to its own location.

Two consequences drive this API:

• Default elision. A field whose value equals its default is simply not stored. A reader must return the default for absent fields — which is why every accessor in this package takes an optional default.

• Vtable deduplication. Tables with identical layouts share one vtable; the shared vtable may sit after the table (the soffset is then negative). The builder performs this dedup automatically, and the reader handles both directions.

2 Quick start🔗ℹ

Take this schema (the one used by the package’s own tests):

Building the equivalent buffer with the low-level builder. Strings and vectors must be created before the object that references them; vector elements are prepended in reverse order; field ids are the vtable slot numbers:

> (define b (make-builder))

> (define label-off (builder-create-string b "hello"))

> (define name-off (builder-create-string b "in"))

> (define a-off (builder-create-string b "a"))

> (define b-off (builder-create-string b "b"))

> (define inner-off     (let ()       (builder-start-object b 3)       (builder-add-i32-slot b 0 1)       (builder-add-i32-slot b 1 -2)       (builder-add-slot b 2 name-off)       (builder-end-object b)))

> (define inner-a-off     (let ()       (builder-start-object b 3)       (builder-add-i32-slot b 0 10)       (builder-add-slot b 2 a-off)       (builder-end-object b)))

> (builder-start-vector b 2 4)

> (builder-prepend-uoffset b inner-a-off)

> (builder-prepend-uoffset b inner-off)

> (define inners-off (builder-end-vector b 2))

> (define tags-off     (let ()       (builder-start-vector b 2 4)       (builder-prepend-uoffset b b-off)       (builder-prepend-uoffset b a-off)       (builder-end-vector b 2)))

> (define root-off     (let ()       (builder-start-object b 5)       (builder-add-slot b 0 label-off)       (builder-add-slot b 1 inner-off)       (builder-add-slot b 2 inners-off)       (builder-add-slot b 3 tags-off)       (builder-add-slot b 4 0)       (builder-end-object b)))

> (define buf (builder-finish b root-off))

> (bytes-length buf)

152

Reading it back with the generic reader. Missing fields return the caller’s default (or the type default 0 / 0.0 / #f):

> (define t (fb-root (open-flatbuffer buf)))

> (table-field-string t 0)

"hello"

> (table-field-i32 (table-field-table t 1) 1)

-2

> (define inners (table-field-vector t 2))

> (fb-vector-length inners)

2

> (table-field-i32 (fb-vector-ref-table inners 1) 0)

10

> (table-field-table t 4)

#f

> (table-field-i32 t 4 99)

99

> (table-field-present? t 4)

#f

Buffers built by this builder are ordinary FlatBuffers buffers: they can be verified with flatc -t, read by the official C++/Java/Python runtimes, and so on. The package’s test suite does exactly that (see Interoperability with flatc).

3 Reading buffers🔗ℹ

A table pairs a buffer with the table’s absolute byte position; table-pos exposes it. Fields are accessed by numeric field id (the vtable slot, normally the field’s declaration order in the schema):

4 Building buffers🔗ℹ

5 Interoperability with flatc🔗ℹ

The test suite ("flatbuffers/test/flatbuffers-test.rkt") checks:

• binaries produced by flatc -b from JSON test data (including a default-elision case and an empty-vector/empty-string edge case) are decoded correctly by the reader;

• buffers built by the builder are converted back to JSON with flatc -t and compared field-by-field semantically — byte-for-byte equality is not required, because the format deliberately leaves field order, object placement and vtable sharing unspecified;

• an alignment fuzz loop (strings of length 1..40 interleaved with uint64 fields) round-trips through both our reader and, for one case, flatc;

• vtable deduplication produces buffers flatc accepts (soffset pointing forward to a shared vtable).

The interop tests are skipped gracefully at runtime when flatc is not on PATH (e.g. on Windows CI), so the pure-Racket round-trip suite always runs.

6 Limitations🔗ℹ

• Little-endian, 32-bit-offset, non-size-prefixed buffers only.

• No schema verification: a generic reader trusts the schema knowledge of its caller (a vector’s element type, a struct’s size, ...). Malformed or hostile buffers raise exn:fail on out-of-range access rather than guaranteeing memory safety properties beyond Racket’s own byte-string bounds. A formal verifier is future work.

• The builder deduplicates vtables byte-exactly (like the reference implementation); tables built with different slot counts produce distinct vtables even if some slots are unused.