1.5.3.1 Read🔗ℹ

read-parquet loads a Parquet file into a dataframe. "flights.parquet" holds the 102 rows of "flights.tsv":

> (define flights (read-parquet "flights.parquet"))

> (shape flights)

'(102 19)

> (~> flights (select "carrier" "dep_delay" "time_hour") (head 3))

shape: (3, 3) ┌─────────┬───────────┬─────────────────────┐ │ carrier ┆ dep_delay ┆ time_hour           │ │ ---     ┆ ---       ┆ ---                 │ │ str     ┆ i64       ┆ datetime[μs]        │ ╞═════════╪═══════════╪═════════════════════╡ │ UA      ┆ 2         ┆ 2013-01-01 05:00:00 │ │ UA      ┆ 4         ┆ 2013-01-01 05:00:00 │ │ AA      ┆ 2         ┆ 2013-01-01 05:00:00 │ └─────────┴───────────┴─────────────────────┘

1.5.3.2 Write🔗ℹ

write-parquet saves a dataframe to Parquet:

> (define df     (dataframe (list (series '(1 2 3) #:name "foo")                      (series (list polars-null "bak" "baz") #:name "bar"))))

> (write-parquet df path)

> (read-parquet path)

shape: (3, 2) ┌─────┬──────┐ │ foo ┆ bar  │ │ --- ┆ ---  │ │ i64 ┆ str  │ ╞═════╪══════╡ │ 1   ┆ null │ │ 2   ┆ bak  │ │ 3   ┆ baz  │ └─────┴──────┘

1.5.3.3 Scan🔗ℹ

Polars can also scan a Parquet input. Scanning delays the actual parsing of the file and returns a lazyframe, a lazy computation holder; collect runs it. See Lazy API for why that is desirable.

> (scan-parquet path)

#<lazyframe>

> (collect (scan-parquet path))

shape: (3, 2) ┌─────┬──────┐ │ foo ┆ bar  │ │ --- ┆ ---  │ │ i64 ┆ str  │ ╞═════╪══════╡ │ 1   ┆ null │ │ 2   ┆ bak  │ │ 3   ┆ baz  │ └─────┴──────┘

1.5.3.4 Dtypes🔗ℹ

A CSV file holds text, so a column read back from one is whatever the reader infers. Parquet records each column’s dtype, so Categorical, Enum, Decimal and datetime columns, zoned or not, come back as they went out. "produce.parquet" has one of each of the first three:

> (define produce (read-parquet "produce.parquet"))

> produce

shape: (4, 3) ┌───────┬───────┬───────────────┐ │ item  ┆ grade ┆ price         │ │ ---   ┆ ---   ┆ ---           │ │ cat   ┆ enum  ┆ decimal[10,2] │ ╞═══════╪═══════╪═══════════════╡ │ apple ┆ high  ┆ 1.25          │ │ pear  ┆ low   ┆ 0.80          │ │ apple ┆ null  ┆ null          │ │ fig   ┆ mid   ┆ 12.00         │ └───────┴───────┴───────────────┘

> (write-csv produce (build-path dir "produce.csv"))

> (read-csv (build-path dir "produce.csv"))

shape: (4, 3) ┌───────┬───────┬───────┐ │ item  ┆ grade ┆ price │ │ ---   ┆ ---   ┆ ---   │ │ str   ┆ str   ┆ f64   │ ╞═══════╪═══════╪═══════╡ │ apple ┆ high  ┆ 1.25  │ │ pear  ┆ low   ┆ 0.8   │ │ apple ┆ null  ┆ null  │ │ fig   ┆ mid   ┆ 12.0  │ └───────┴───────┴───────┘

> (write-parquet produce (build-path dir "produce.parquet"))

> (read-parquet (build-path dir "produce.parquet"))

shape: (4, 3) ┌───────┬───────┬───────────────┐ │ item  ┆ grade ┆ price         │ │ ---   ┆ ---   ┆ ---           │ │ cat   ┆ enum  ┆ decimal[10,2] │ ╞═══════╪═══════╪═══════════════╡ │ apple ┆ high  ┆ 1.25          │ │ pear  ┆ low   ┆ 0.80          │ │ apple ┆ null  ┆ null          │ │ fig   ┆ mid   ┆ 12.00         │ └───────┴───────┴───────────────┘

A naive and a zoned datetime column (Time zones) lose their dtype through CSV and keep it through Parquet:

> (define stamps     (select flights "time_hour"             (~> (col "time_hour")                 (dt-replace-time-zone "America/New_York")                 (alias "local"))))

> (write-csv stamps (build-path dir "stamps.csv"))

> (read-csv (build-path dir "stamps.csv") #:n-rows 2)

shape: (2, 2) ┌────────────────────────────┬─────────────────────────────────┐ │ time_hour                  ┆ local                           │ │ ---                        ┆ ---                             │ │ str                        ┆ str                             │ ╞════════════════════════════╪═════════════════════════════════╡ │ 2013-01-01T05:00:00.000000 ┆ 2013-01-01T05:00:00.000000-050… │ │ 2013-01-01T05:00:00.000000 ┆ 2013-01-01T05:00:00.000000-050… │ └────────────────────────────┴─────────────────────────────────┘

> (read-csv (build-path dir "stamps.csv") #:n-rows 2 #:try-parse-dates #t)

shape: (2, 2) ┌─────────────────────┬─────────────────────────┐ │ time_hour           ┆ local                   │ │ ---                 ┆ ---                     │ │ datetime[μs]        ┆ datetime[μs, UTC]       │ ╞═════════════════════╪═════════════════════════╡ │ 2013-01-01 05:00:00 ┆ 2013-01-01 10:00:00 UTC │ │ 2013-01-01 05:00:00 ┆ 2013-01-01 10:00:00 UTC │ └─────────────────────┴─────────────────────────┘

> (write-parquet stamps (build-path dir "stamps.parquet"))

> (read-parquet (build-path dir "stamps.parquet") #:n-rows 2)

shape: (2, 2) ┌─────────────────────┬────────────────────────────────┐ │ time_hour           ┆ local                          │ │ ---                 ┆ ---                            │ │ datetime[μs]        ┆ datetime[μs, America/New_York] │ ╞═════════════════════╪════════════════════════════════╡ │ 2013-01-01 05:00:00 ┆ 2013-01-01 05:00:00 EST        │ │ 2013-01-01 05:00:00 ┆ 2013-01-01 05:00:00 EST        │ └─────────────────────┴────────────────────────────────┘

> (dtype (ref (read-parquet (build-path dir "stamps.parquet")) "local"))

'(datetime microseconds "America/New_York")

1.5.3.5 Projection and predicate pushdown🔗ℹ

A scan reads only what its plan needs. explain shows the plan: written, the filter and the selection sit above the scan; optimised, both move into it, so Polars reads two of the 19 columns (PROJECT) and applies the filter as it reads (SELECTION), skipping any row group whose statistics rule it out.

> (define late     (~> (scan-parquet "flights.parquet")         (filter (> (col "dep_delay") 20))         (select "carrier" "dep_delay")))

> (displayln (explain late #:optimized #f))

SELECT [col("carrier"), col("dep_delay")]   FILTER col("dep_delay") > 20   FROM     Parquet SCAN [/home/root/user/.local/share/racket/9.3/pkgs/polars/scribblings/data/flights.parquet]     PROJECT */19 COLUMNS     ESTIMATED ROWS: 102

> (displayln (explain late))

Parquet SCAN [/home/root/user/.local/share/racket/9.3/pkgs/polars/scribblings/data/flights.parquet] PROJECT 2/19 COLUMNS SELECTION: col("dep_delay") > 20 ESTIMATED ROWS: 102

> (collect late)

shape: (3, 2) ┌─────────┬───────────┐ │ carrier ┆ dep_delay │ │ ---     ┆ ---       │ │ str     ┆ i64       │ ╞═════════╪═══════════╡ │ EV      ┆ 24        │ │ UA      ┆ 47        │ │ MQ      ┆ 39        │ └─────────┴───────────┘

1.5.3.6 Options🔗ℹ

Upstream documents the readers’ and the writer’s options on their reference pages; each keyword here keeps Python’s name. #:columns, #:n-rows and #:row-index-name frame what read-parquet returns; #:compression, #:compression-level, #:statistics and #:row-group-size set how write-parquet lays the file out.

> (read-parquet "flights.parquet" #:columns '("row" "carrier" "dep_delay") #:n-rows 3                 #:row-index-name "row")

shape: (3, 3) ┌─────┬─────────┬───────────┐ │ row ┆ carrier ┆ dep_delay │ │ --- ┆ ---     ┆ ---       │ │ u32 ┆ str     ┆ i64       │ ╞═════╪═════════╪═══════════╡ │ 0   ┆ UA      ┆ 2         │ │ 1   ┆ UA      ┆ 4         │ │ 2   ┆ AA      ┆ 2         │ └─────┴─────────┴───────────┘

> (write-parquet flights (build-path dir "flights.parquet")                  #:compression 'gzip #:compression-level 9 #:statistics 'full                  #:row-group-size 50)

> (shape (read-parquet (build-path dir "flights.parquet")))

'(102 19)

API gaps: no schema, extra_columns or metadata.