1.5.2.1 Read🔗ℹ

1.5.2.2 Write🔗ℹ

> (define df     (dataframe (list (series '(1 2 3) #:name "foo")                      (series (list polars-null "bak" "baz") #:name "bar"))))

> (write-json df path)

> (file->string path)

"[{\"foo\":1,\"bar\":null},{\"foo\":2,\"bar\":\"bak\"},{\"foo\":3,\"bar\":\"baz\"}]"

> (read-json path)

shape: (3, 2) ┌─────┬──────┐ │ foo ┆ bar  │ │ --- ┆ ---  │ │ i64 ┆ str  │ ╞═════╪══════╡ │ 1   ┆ null │ │ 2   ┆ bak  │ │ 3   ┆ baz  │ └─────┴──────┘

> (write-ndjson df nd-path)

> (file->string nd-path)

"{\"foo\":1,\"bar\":null}\n{\"foo\":2,\"bar\":\"bak\"}\n{\"foo\":3,\"bar\":\"baz\"}\n"

> (read-ndjson nd-path)

shape: (3, 2) ┌─────┬──────┐ │ foo ┆ bar  │ │ --- ┆ ---  │ │ i64 ┆ str  │ ╞═════╪══════╡ │ 1   ┆ null │ │ 2   ┆ bak  │ │ 3   ┆ baz  │ └─────┴──────┘

Racket’s json library has a read-json and write-json of its own; read-json’s reference entry shows how to require both libraries.

1.5.2.3 Scan🔗ℹ

Polars allows you to scan an NDJSON input. Scanning delays the actual parsing of the file and instead returns a lazy computation holder called a lazyframe.

> (scan-ndjson "stations.ndjson")

#<lazyframe>

> (~> (scan-ndjson "stations.ndjson")       (filter (> (col "reading") 3))       collect)

shape: (2, 5) ┌─────────┬────────────┬─────────┬──────┬──────┐ │ station ┆ day        ┆ reading ┆ flag ┆ note │ │ ---     ┆ ---        ┆ ---     ┆ ---  ┆ ---  │ │ str     ┆ str        ┆ f64     ┆ str  ┆ str  │ ╞═════════╪════════════╪═════════╪══════╪══════╡ │ south   ┆ 2024-01-01 ┆ 4.5     ┆ low  ┆ null │ │ south   ┆ 2024-01-02 ┆ 5.25    ┆ high ┆ late │ └─────────┴────────────┴─────────┴──────┴──────┘

1.5.2.4 Reading options🔗ℹ

Upstream documents read_json’s and read_ndjson’s options on its reference pages; here they are with their Racket spellings. Both files hold the four objects shown above: the first reading is an integer, and only the last object has a note.

Types. #:schema gives the columns and their types outright; #:schema-overrides changes the types of some inferred columns. Either reads an ISO date string as a date.

> (read-json "stations.json"              #:schema '(("day" . date) ("station" . categorical) ("reading" . f32)))

shape: (4, 3) ┌────────────┬─────────┬─────────┐ │ day        ┆ station ┆ reading │ │ ---        ┆ ---     ┆ ---     │ │ date       ┆ cat     ┆ f32     │ ╞════════════╪═════════╪═════════╡ │ 2024-01-01 ┆ north   ┆ 3.0     │ │ 2024-01-01 ┆ south   ┆ 4.5     │ │ 2024-01-02 ┆ north   ┆ 2.0     │ │ 2024-01-02 ┆ south   ┆ 5.25    │ └────────────┴─────────┴─────────┘

> (~> (read-json "stations.json" #:schema-overrides '(("day" . date)))       (select "day" "reading"))

shape: (4, 2) ┌────────────┬─────────┐ │ day        ┆ reading │ │ ---        ┆ ---     │ │ date       ┆ f64     │ ╞════════════╪═════════╡ │ 2024-01-01 ┆ 3.0     │ │ 2024-01-01 ┆ 4.5     │ │ 2024-01-02 ┆ 2.0     │ │ 2024-01-02 ┆ 5.25    │ └────────────┴─────────┘

Inference. Types are inferred from the first 100 objects; #:infer-schema-length changes that, and #f reads them all. A key that first appears later is an error:

> (read-json "stations.json" #:infer-schema-length 1)

read-json: failed to read json from stations.json: extra

field in struct data: note, consider increasing

#:infer-schema-length, or passing the full #:schema to

ignore extra fields

> (shape (read-json "stations.json" #:infer-schema-length #f))

'(4 5)

read-ndjson and scan-ndjson take the same three keywords, but leave a late key out, and #:ignore-errors reads a value that does not parse as null:

> (column-names (read-ndjson "stations.ndjson" #:infer-schema-length 2))

'("station" "day" "reading" "flag")

> (read-ndjson "stations.ndjson" #:infer-schema-length 1)

read-ndjson: failed to read ndjson from stations.ndjson:

cannot parse '4.5' (f64) as Int64

> (~> (read-ndjson "stations.ndjson" #:infer-schema-length 1 #:ignore-errors #t)       (select "station" "reading"))

shape: (4, 2) ┌─────────┬─────────┐ │ station ┆ reading │ │ ---     ┆ ---     │ │ str     ┆ i64     │ ╞═════════╪═════════╡ │ north   ┆ 3       │ │ south   ┆ null    │ │ north   ┆ 2       │ │ south   ┆ null    │ └─────────┴─────────┘

Rows and files. #:n-rows caps the rows read, #:row-index-name adds a column of row numbers and #:include-file-paths one naming each row’s file. The path is a glob pattern, as for Parquet:

> (for ([i '(1 2 3)])     (write-ndjson (read-csv (format "parts/part-~a.csv" i))                   (build-path dir (format "part-~a.ndjson" i))))

> (read-ndjson (build-path dir "part-*.ndjson") #:n-rows 4 #:row-index-name "row")

shape: (4, 4) ┌─────┬────────┬──────┬───────────┐ │ row ┆ origin ┆ dest ┆ dep_delay │ │ --- ┆ ---    ┆ ---  ┆ ---       │ │ u32 ┆ str    ┆ str  ┆ i64       │ ╞═════╪════════╪══════╪═══════════╡ │ 0   ┆ EWR    ┆ IAH  ┆ 2         │ │ 1   ┆ LGA    ┆ IAH  ┆ 4         │ │ 2   ┆ JFK    ┆ MIA  ┆ 2         │ │ 3   ┆ JFK    ┆ BQN  ┆ -1        │ └─────┴────────┴──────┴───────────┘

API gaps: a path only, no file object or in-memory string in or out; an Enum in #:schema (read as 'categorical and cast); no #:glob #f to read a path literally; one path or pattern, not a list; no cloud storage_options, credential_provider or retries.