Racket

LZF: Pure-Racket LZF Compression🔗ℹ

package: lzf

This package is a codec for the liblzf byte format, not a binding to the C library: no FFI is used and no system liblzf installation is required.

The LZF format is a lightweight, byte-oriented LZ77 variant designed by Marc Alexander Lehmann for his liblzf library. Its simplicity—one control byte per token, byte-granular copies, no bit-planes—makes it extremely fast in C, which is why it is the string-compression codec embedded in Redis RDB persistence files and the interchange format of Python’s python-lzf module.

This package implements the format in pure Racket, written against the reference C sources in the Redis repository (src/lzf_c.c and src/lzf_d.c):

lzf-decompress reads streams produced by liblzf, including payloads found in Redis RDB files and python-lzf output;
lzf-compress produces streams decodable by liblzf.

The wire format, for reference (from the comment in lzf_c.c):

000LLLLL <L+1> ; literal run, L+1 = 1..33 octetsLLLooooo oooooooo ; back reference, L+2 octets, offset o+1111ooooo LLLLLLLL oooooooo ; back reference, L+9 octets, offset o+1

1 API🔗ℹ

procedure
(lzf-compress bs) → bytes?
bs : bytes?

Compresses bs with a greedy encoder and returns a liblzf-format byte string.

The encoder uses a single-slot hash table of 2^13 slots (liblzf’s low-memory configuration, HLOG = 13) with liblzf’s default (VERY_FAST) post-match insertion policy, and it never emits a back reference into position 0, exactly like the C encoder. The output is deterministic: the same input always produces the same bytes.

Back references span at most 2^13 bytes of distance and copy at most 264 bytes; literal runs hold at most 33 bytes. Compressing the empty byte string returns the empty byte string.

procedure
(lzf-decompress bs [#:hint hint]) → bytes?
bs : bytes?
hint : natural? = (max 64 (* 4 (bytes-length bs)))

Decompresses the liblzf stream bs and returns a freshly allocated, exact-size byte string.

hint is an initial guess at the output size. Unlike the C decoder—which fails with E2BIG when the caller’s buffer is too small—this decoder grows its output buffer automatically whenever the hint is too small, so every valid stream decodes successfully regardless of the hint. Supplying an accurate hint merely avoids a few reallocations.

Decompressing the empty byte string returns the empty byte string.

Corrupt input raises exn:fail with a descriptive message. Two conditions are detected:

a truncated stream—a literal run, offset byte, or length byte running past the end of bs (the C decoder reports EINVAL for these under CHECK_INPUT);
a back reference reaching before the start of the output (the C decoder’s ref < out_data EINVAL case).

2 Examples🔗ℹ

> (define data (bytes-append (make-bytes 40 65) #" and some real text..."))
> (define compressed (lzf-compress data))
> (bytes-length compressed)
29
> (bytes=? (lzf-decompress compressed) data)
#t

Both directions treat the empty byte string as empty:

> (lzf-compress "")
lzf-compress: contract violation
  expected: bytes?
  given: ""
  in: the 1st argument of
      (-> bytes? bytes?)
  contract from: <pkgs>/lzf/lzf/main.rkt
  blaming: program
   (assuming the contract is correct)
  at: <pkgs>/lzf/lzf/main.rkt:17:3
> (lzf-decompress "")
lzf-decompress: contract violation
  expected: bytes?
  given: ""
  in: the 1st argument of
      (->* (bytes?) (#:hint natural?) bytes?)
  contract from: <pkgs>/lzf/lzf/main.rkt
  blaming: program
   (assuming the contract is correct)
  at: <pkgs>/lzf/lzf/main.rkt:18:3

Decoding corrupt data raises exn:fail; it never crashes:

> (lzf-decompress #" \5")
lzf-decompress: corrupt LZF stream: back-reference reaches
before start of output (offset 6 at output position 0)

3 Implementation notes🔗ℹ

The hash-table configuration (table size, insertion policy) affects compression ratio only, never the emitted format, so this codec is bit-compatible with every liblzf build.
Unlike liblzf’s default build—which leaves its hash table uninitialized on the stack—this implementation zero-initializes the table, which is what makes the output deterministic.
The decoder validates back-reference reach and stream completeness but, like the C decoder, does not attempt checksums; LZF has no integrity metadata. Pair it with an external checksum if corruption matters.

4 License🔗ℹ

MIT. See the LICENSE file in the package distribution.