2 Reference
polars is written to read as ordinary Racket. The operators
you already know —
Underneath sits a monomorphic, dtype-suffixed layer (series-new-i32,
series-sum-f64, expr-add and friends). It is documented here
for completeness —
2.1 Operators and pipelines
This is the surface to write polars in. An
expression describes a column computation without running it, and
the operators below are the ordinary Racket ones —
Every operation takes the frame —
(~> df (filter (> (col "value") 15)) (group-by "group") (agg (alias (sum (col "value")) "sum_value")))
procedure
spec : (or/c string? regexp? dtype-spec?)
procedure
v :
(or/c boolean? (integer-in (- (expt 2 63)) (sub1 (expt 2 63))) (and/c real? (not/c exact-integer?)) string? symbol? polars-null? gregor-value)
procedure
(dtype-spec? v) → boolean?
v : any/c
A string is the column of that name (pl.col("name")). A string of the form ^...$ is a Polars regex, in the syntax of Rust’s regex crate, as in Python.
A dtype is every column of that dtype (pl.col(pl.Float64)). dtype-spec? is any spelling series’ #:dtype accepts, so 'float64 and 'f64 alike. A bare 'datetime means microseconds: the unit #:try-parse-dates gives parsed datetimes, and the unit series gives gregor datetimes unless one carries a sub-microsecond part and a nanosecond column holds them all, which makes the column nanoseconds (Dates, times, datetimes and durations). Match another unit with its dtype. A datetime dtype without a zone matches only naive columns, and '(datetime unit "Zone/Name") only the columns of that unit in exactly that zone, as pl.col(pl.Datetime("us", "Europe/Brussels")) does. API gap: no spelling selects every datetime column whatever its unit or zone, as pl.col(pl.Datetime) and pl.Datetime("us", "*") do (#171). 'categorical is every categorical column, and an '(enum ....) dtype the columns of exactly that Enum (Categorical, Enum and Decimal).
A regexp is every column whose name matches (pl.col("^sepal_.*$")), keeping the regexp’s Racket meaning: (col rx) selects exactly the names (regexp-match? rx name) accepts, in #rx and #px syntax alike. There are two exceptions. A \p{...} property class follows each side’s own version of the Unicode tables. And Racket’s own matcher misjudges some classes containing characters above U+00FF; there the selection follows the class as written. The crate has no lookaround, backreferences, atomic groups or conditionals, so col (and exclude) rejects a regexp using them with a contract violation naming the construct. To select by such a regexp, match column-names in Racket and select the names, as in the last example below.
A multi-column col expands inside any expression to one output per matched column, in the frame’s column order, each keeping the matched column’s name; a frame with no match yields no columns.
lit lifts a Racket scalar to a literal expression: booleans, exact integers (32-bit when they fit, 64-bit otherwise; one outside 64 bits is a contract error), other reals (as 'float64) and strings. A symbol is the string of its name, so a categorical column compares with the symbols it reads back as. A gregor-value is a gregor date, time, datetime, moment or period, and becomes a literal of the dtype series infers for it (Dates, times, datetimes and durations), so a temporal column compares with gregor values as Python’s compares with date, time, datetime and timedelta. A moment is a zoned literal: in its named zone, or in UTC for a fixed offset. A zoned column compares only with a literal in its own zone, as in Python. polars-null is a null of dtype 'null, as pl.lit(None) is: it broadcasts to a 'null column and compares as null, so (= (col "x") polars-null) is null on every row (test with is-null instead). In arithmetic it takes a numeric operand’s dtype, and with a temporal operand it raises, as in Python. Every operator below lifts a non-expression operand with lit automatically, so it is rarely needed explicitly.
> (define people (dataframe (list (series '(1 2 3) #:name "id" #:dtype 'i32) (series '(57.9 72.5 53.6) #:name "weight") (series '(1.56 1.77 1.65) #:name "height")))) > (select people (* (col 'float64) 1.1))
shape: (3, 2)
┌────────┬────────┐
│ weight ┆ height │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪════════╡
│ 63.69 ┆ 1.716 │
│ 79.75 ┆ 1.947 │
│ 58.96 ┆ 1.815 │
└────────┴────────┘
> (list (dtype-spec? 'f64) (dtype-spec? 'float)) '(#t #f)
> (select people (col "^.*ght$"))
shape: (3, 2)
┌────────┬────────┐
│ weight ┆ height │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪════════╡
│ 57.9 ┆ 1.56 │
│ 72.5 ┆ 1.77 │
│ 53.6 ┆ 1.65 │
└────────┴────────┘
> (select people (col #rx"^he"))
shape: (3, 1)
┌────────┐
│ height │
│ --- │
│ f64 │
╞════════╡
│ 1.56 │
│ 1.77 │
│ 1.65 │
└────────┘
> (select people (col #px"^\\w+t$"))
shape: (3, 2)
┌────────┬────────┐
│ weight ┆ height │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪════════╡
│ 57.9 ┆ 1.56 │
│ 72.5 ┆ 1.77 │
│ 53.6 ┆ 1.65 │
└────────┴────────┘
> (select people (~> (col "id") (* 10) (alias "id10")) (alias (lit 0) "zero"))
shape: (3, 2)
┌──────┬──────┐
│ id10 ┆ zero │
│ --- ┆ --- │
│ i32 ┆ i32 │
╞══════╪══════╡
│ 10 ┆ 0 │
│ 20 ┆ 0 │
│ 30 ┆ 0 │
└──────┴──────┘
> (lit (date 2013 6 1)) 2013-06-01
> (select people (alias (lit (datetime 2013 6 1 12 30)) "noon"))
shape: (1, 1)
┌─────────────────────┐
│ noon │
│ --- │
│ datetime[μs] │
╞═════════════════════╡
│ 2013-06-01 12:30:00 │
└─────────────────────┘
> (select people (alias (lit (moment 2013 6 1 12 30 #:tz "Europe/Paris")) "noon"))
shape: (1, 1)
┌────────────────────────────┐
│ noon │
│ --- │
│ datetime[μs, Europe/Paris] │
╞════════════════════════════╡
│ 2013-06-01 12:30:00 CEST │
└────────────────────────────┘
> (select people (alias (lit polars-null) "nothing") (= (col "id") polars-null) (+ (col "weight") polars-null))
shape: (3, 3)
┌─────────┬──────┬────────┐
│ nothing ┆ id ┆ weight │
│ --- ┆ --- ┆ --- │
│ null ┆ bool ┆ f64 │
╞═════════╪══════╪════════╡
│ null ┆ null ┆ null │
│ null ┆ null ┆ null │
│ null ┆ null ┆ null │
└─────────┴──────┴────────┘
> (col #px"^(?!id)") col: contract violation;
Polars' regex engine has no lookahead; to select by this
regexp, match column-names with regexp-match? and select the
names
given: #px"^(?!id)"
in: the 1st argument of
(->
(or/c string? regexp? dtype-spec?)
Expr-ptr?)
contract from:
<pkgs>/polars/private/expr-core.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/expr-core.rkt:39:11
> (select people (filter (lambda (name) (regexp-match? #px"^(?!id)" name)) (column-names people)))
shape: (3, 2)
┌────────┬────────┐
│ weight ┆ height │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪════════╡
│ 57.9 ┆ 1.56 │
│ 72.5 ┆ 1.77 │
│ 53.6 ┆ 1.65 │
└────────┴────────┘
> (lit (expt 2 63)) lit: contract violation
expected: (or/c boolean? (integer-in -9223372036854775808
9223372036854775807) (and/c real? (not/c exact-integer?))
string? symbol? polars-null? gregor-value)
given: 9223372036854775808
in: the 1st argument of
(->
(or/c
boolean?
(integer-in
-9223372036854775808
9223372036854775807)
(and/c real? (not/c exact-integer?))
string?
symbol?
polars-null?
gregor-value)
Expr-ptr?)
contract from:
<pkgs>/polars/private/expr-core.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/expr-core.rkt:38:11
procedure
procedure
e : multi-column-expr? name : (or/c string? regexp?)
procedure
(multi-column-expr? v) → boolean?
v : any/c
> (select people (all))
shape: (3, 3)
┌─────┬────────┬────────┐
│ id ┆ weight ┆ height │
│ --- ┆ --- ┆ --- │
│ i32 ┆ f64 ┆ f64 │
╞═════╪════════╪════════╡
│ 1 ┆ 57.9 ┆ 1.56 │
│ 2 ┆ 72.5 ┆ 1.77 │
│ 3 ┆ 53.6 ┆ 1.65 │
└─────┴────────┴────────┘
> (select people (exclude (all) "id"))
shape: (3, 2)
┌────────┬────────┐
│ weight ┆ height │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪════════╡
│ 57.9 ┆ 1.56 │
│ 72.5 ┆ 1.77 │
│ 53.6 ┆ 1.65 │
└────────┴────────┘
> (select people (exclude (all) #rx"^w" "id"))
shape: (3, 1)
┌────────┐
│ height │
│ --- │
│ f64 │
╞════════╡
│ 1.56 │
│ 1.77 │
│ 1.65 │
└────────┘
> (select people (exclude (all) "^h.*$"))
shape: (3, 2)
┌─────┬────────┐
│ id ┆ weight │
│ --- ┆ --- │
│ i32 ┆ f64 │
╞═════╪════════╡
│ 1 ┆ 57.9 │
│ 2 ┆ 72.5 │
│ 3 ┆ 53.6 │
└─────┴────────┘
> (select people (~> (col 'float64) (exclude "height") (* 2)))
shape: (3, 1)
┌────────┐
│ weight │
│ --- │
│ f64 │
╞════════╡
│ 115.8 │
│ 145.0 │
│ 107.2 │
└────────┘
> (select people (~> (all) (exclude "id") (exclude "weight")))
shape: (3, 1)
┌────────┐
│ height │
│ --- │
│ f64 │
╞════════╡
│ 1.56 │
│ 1.77 │
│ 1.65 │
└────────┘
> (~> people (with-columns (~> (col "height") (> 1.6) (alias "tall"))) (group-by "tall") (agg (~> (all) (exclude "id") mean)) (sort "tall"))
shape: (2, 3)
┌───────┬────────┬────────┐
│ tall ┆ weight ┆ height │
│ --- ┆ --- ┆ --- │
│ bool ┆ f64 ┆ f64 │
╞═══════╪════════╪════════╡
│ false ┆ 57.9 ┆ 1.56 │
│ true ┆ 63.05 ┆ 1.71 │
└───────┴────────┴────────┘
> (multi-column-expr? (col "id")) #f
> (~> (col 'float64) (* 2) multi-column-expr?) #t
> (exclude (col "id") "weight") exclude: contract violation
expected: multi-column-expr?
given: col("id")
in: the 1st argument of
(->
multi-column-expr?
(or/c string? regexp?)
(or/c string? regexp?)
...
Expr-ptr?)
contract from:
<pkgs>/polars/private/generic/selectors.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/selectors.rkt:9:11
procedure
(expr->string e) → string?
e : Expr-ptr?
> (col "weight") col("weight")
> (alias (* (col "v") 10) "v10") [(col("v")) * (dyn int: 10)].alias("v10")
> (expr->string (> (col "v") 2)) "[(col(\"v\")) > (dyn int: 2)]"
> (~> (col "v") sum (over "k")) col("v").sum().over([col("k")])
> (exclude (all) "id") [cs.all() - cs.by_name('id', require_all=false)]
> (col 'float64) cs.by_dtype([Float64])
> (col #rx"^he") cs.matches("^(?s).*(?:^he).*$")
> (define far-date (cast (series '(2000000000) #:dtype 'i32 #:name "d") 'date)) > (is-in "d" far-date) #<expr: cannot print: polars panicked: out-of-range date>
> (expr->string (is-in "d" far-date)) expr->string: cannot print the expression: polars panicked:
out-of-range date
> (define threshold (> (col "v") 2)) > (expr-drop threshold) > threshold #<expr: dropped>
> (filter (dataframe (list (series '(1 3) #:name "v"))) threshold) expr: expression has been dropped
> (expr-drop threshold) expr-drop: expression has been dropped
procedure
(meta-output-name e) → string?
e : (or/c Expr-ptr? string?)
procedure
(meta-root-names e) → (listof string?)
e : (or/c Expr-ptr? string?)
procedure
a : (or/c Expr-ptr? string?) b : (or/c Expr-ptr? string?)
A multi-column expression is read off the plan too, before any frame says which columns it will match, so a regexp or dtype col or (all) has no root names and no output name, as in Python.
> (define total (alias (sum (+ (col "a") (col "b"))) "total")) > total [(col("a")) + (col("b"))].sum().alias("total")
> (meta-output-name total) "total"
> (meta-root-names total) '("a" "b")
> (meta-eq? total (alias (sum (+ (col "a") (col "b"))) "total")) #t
> (meta-eq? total (col "a")) #f
> (equal? total (~> (+ (col "a") (col "b")) sum (alias "total"))) #f
> (meta-output-name (+ (col "a") (col "b"))) "a"
> (meta-output-name (lit 25)) "literal"
> (meta-root-names "a") '("a")
> (meta-root-names (col #rx"^he")) '()
> (meta-output-name (col #rx"^he")) meta-output-name: cannot determine the output name of
cs.matches("^(?s).*(?:^he).*$")
> (~> (col 'float64) (* 2) meta-root-names) '()
> (meta-eq? (col 'float64) (col 'f64)) #t
> (meta-output-name (col 'float64)) meta-output-name: cannot determine the output name of
cs.by_dtype([Float64])
> (meta-output-name "*") meta-output-name: cannot determine the output name of
cs.all()
> (meta-output-name 5) meta-output-name: contract violation
expected: col-expr/c
given: 5
in: the 1st argument of
(-> col-expr/c string?)
contract from:
<pkgs>/polars/private/generic/meta.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/meta.rkt:10:11
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
> (= (series (list polars-null polars-null)) 1)
shape: (2,)
Series: '' [bool]
[
null
null
]
> (> (series '(1 2)) 1.5)
shape: (2,)
Series: '' [bool]
[
#f
#t
]
> (> (series '(1 2)) "a") >: the value cannot be an element of a series of this dtype
value: "a"
series dtype: 'int64
> (and polars-null (series (list #t #f)))
shape: (2,)
Series: '' [bool]
[
null
#f
]
> (or polars-null (series (list #t #f)))
shape: (2,)
Series: '' [bool]
[
#t
null
]
> (xor (series '(#t #f)) #t)
shape: (2,)
Series: '' [bool]
[
#f
#t
]
> (not (series '(1 2))) not: invalid series dtype: expected `Boolean`, got `i64` for
series with name ``
procedure
v : any/c
procedure
v : any/c
procedure
v : any/c
procedure
v : any/c
procedure
a : any/c b : any/c
With one argument, - negates (expr-neg) and / is (/ 1 v), each output keeping its own name; + and * return an expression or series operand unchanged, and mod needs two arguments.
On integer columns / is integer division, rounding toward negative infinity, with a null where the divisor is 0: (/ (col "v") 2) gives -4 for -7, unlike quotient, and the one-argument reciprocal of an integer column is 1, -1, 0 or null. Python’s / gives floats there; cast an operand to 'f64 first for the same result.
> (- 5) -5
> (/ 4) 1/4
> (mod 10 4 3) 2
> (- (col "delay")) col("delay").negate()
> (/ (col "speed")) [(dyn int: 1) rust_div (col("speed"))].alias("speed")
> (- (series '(1 -2 3) #:name "v"))
shape: (3,)
Series: 'v' [i64]
[
-1
2
-3
]
> (/ (series '(1.0 2.0 4.0) #:name "v"))
shape: (3,)
Series: 'v' [f64]
[
1.0
0.5
0.25
]
> (/ (series '(1 2 4)))
shape: (3,)
Series: '' [i64]
[
1
0
0
]
> (/ (series '(1 -1 2 -2 0)))
shape: (5,)
Series: '' [i64]
[
1
-1
0
-1
null
]
> (/ (series '(-7 7 1)) (series '(2 2 0)))
shape: (3,)
Series: '' [i64]
[
-4
3
null
]
> (/ (cast (series '(1 2 4)) 'f64))
shape: (3,)
Series: '' [f64]
[
1.0
0.5
0.25
]
> (+ (series (list polars-null polars-null)) 1)
shape: (2,)
Series: '' [i32]
[
null
null
]
> (+ (series '(1 2)) 2.5)
shape: (2,)
Series: '' [f64]
[
3.5
4.5
]
> (+ (series '(1 2)) "a") +: the value cannot be an element of a series of this dtype
value: "a"
series dtype: 'int64
> (+ (series '(1 2)) (date 2024 1 1)) +: add operation not supported for dtypes `i64` and `date`
> (~> (dataframe (list (series '(1.0 2.0) #:name "c1") (series '(4.0 8.0) #:name "c2"))) (select (/ (col #rx"^c"))))
shape: (2, 2)
┌─────┬───────┐
│ c1 ┆ c2 │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪═══════╡
│ 1.0 ┆ 0.25 │
│ 0.5 ┆ 0.125 │
└─────┴───────┘
> (- (series '(1 2) #:dtype 'u8)) -: cannot negate a series of this dtype
dtype: 'uint8
> (/ (series '("a"))) /: cannot take the reciprocal of a series of this dtype
dtype: 'string
> (mod (col "v")) mod: arity mismatch;
the expected number of arguments does not match the given
number
expected: at least 2
given: 1
arguments...:
col("v")
> (~> (dataframe (list (series '(10 25 7) #:name "v"))) (with-columns (/ (col "v"))))
shape: (3, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 0 │
│ 0 │
│ 0 │
└─────┘
procedure
(filter d predicate) → dataframe?
d : dataframe? predicate : any/c
procedure
(sort d by [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → dataframe? d : dataframe? by : (or/c string? (non-empty-listof string?)) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
(sort lf by [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → lazyframe? lf : lazyframe? by : (or/c string? (non-empty-listof string?)) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
(sort s [ #:descending descending #:nulls-last nulls-last]) → series? s : series? descending : boolean? = #f nulls-last : boolean? = #f
(sort e [ #:descending descending #:nulls-last nulls-last]) → Expr-ptr? e : (or/c Expr-ptr? string?) descending : boolean? = #f nulls-last : boolean? = #f
(sort lst less-than? [ #:descending descending #:key extract-key #:cache-keys? cache-keys?]) → list? lst : list? less-than? : (any/c any/c . -> . any/c) descending : boolean? = #f extract-key : (or/c #f (any/c . -> . any/c)) = #f cache-keys? : boolean? = #f
Inside agg or over, Polars’ own sort reads out of bounds when its input holds one value per group, and Python crashes on the same query. So Expr.sort gets only an input whose form keeps a value per row: a column, or an alias, cast, sort, operator or when with one among its operands. Any other expression, a function such as round of (mean "delay") among them, is sorted by itself with sort-by (Expr.sort_by), which handles one value per group. It evaluates the expression twice and prints as a sort_by.
Nulls come first, whatever the direction, unless nulls-last is true; NaN sorts above every other float. For a frame, descending and nulls-last are each one boolean for every key or a list of one boolean per key. Rows that tie on every key keep their input order when maintain-order is true, and are in no particular order otherwise.
A sorted expression reorders its own column only, so in with-columns it no longer lines up with the rest of its row. Use it in select or agg, and sort-by to reorder one column by others. A frame sorts by column names only: sorting by an expression (df.sort(pl.col("a") * -1)) has no spelling yet.
On a list, sort is racket/base’s, plus descending, which reverses the order and keeps equal elements in their input order, as Python’s sorted(xs, reverse=True) does. The other Polars keywords are a contract violation there.
> (define flights (dataframe (list (series '("UA" "AA" "UA" "AA" "B6") #:name "carrier") (series (list 12 polars-null 340 -3 polars-null) #:name "delay")))) > (sort flights "delay" #:descending #t)
shape: (5, 2)
┌─────────┬───────┐
│ carrier ┆ delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════╡
│ AA ┆ null │
│ B6 ┆ null │
│ UA ┆ 340 │
│ UA ┆ 12 │
│ AA ┆ -3 │
└─────────┴───────┘
> (~> flights (sort "delay" #:descending #t #:nulls-last #t) (head 2))
shape: (2, 2)
┌─────────┬───────┐
│ carrier ┆ delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════╡
│ UA ┆ 340 │
│ UA ┆ 12 │
└─────────┴───────┘
> (sort flights '("carrier" "delay") #:descending '(#f #t) #:nulls-last '(#f #t) #:maintain-order #t)
shape: (5, 2)
┌─────────┬───────┐
│ carrier ┆ delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════╡
│ AA ┆ -3 │
│ AA ┆ null │
│ B6 ┆ null │
│ UA ┆ 340 │
│ UA ┆ 12 │
└─────────┴───────┘
> (~> flights lazy (sort "delay" #:nulls-last #t) collect)
shape: (5, 2)
┌─────────┬───────┐
│ carrier ┆ delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════╡
│ AA ┆ -3 │
│ UA ┆ 12 │
│ UA ┆ 340 │
│ AA ┆ null │
│ B6 ┆ null │
└─────────┴───────┘
> (sort (ref flights #:columns "delay") #:descending #t #:nulls-last #t)
shape: (5,)
Series: 'delay' [i64]
[
340
12
-3
null
null
]
> (sort "delay" #:descending #t #:nulls-last #t) col("delay").sort(desc)
> (sort (max "delay") #:descending #t) col("delay").max()
> (sort (round (mean "delay")) #:descending #t) col("delay").mean().round().sort_by(by=[col("delay").mean().round()], sort_option=SortMultipleOptions { descending: [true], nulls_last: [false], multithreaded: true, maintain_order: false, limit: None })
> (~> flights (group-by "carrier") (agg (sort (round (mean "delay")) #:descending #t)) (sort "carrier"))
shape: (3, 2)
┌─────────┬───────┐
│ carrier ┆ delay │
│ --- ┆ --- │
│ str ┆ f64 │
╞═════════╪═══════╡
│ AA ┆ -3.0 │
│ B6 ┆ null │
│ UA ┆ 176.0 │
└─────────┴───────┘
> (select flights (sort "delay" #:nulls-last #t))
shape: (5, 1)
┌───────┐
│ delay │
│ --- │
│ i64 │
╞═══════╡
│ -3 │
│ 12 │
│ 340 │
│ null │
│ null │
└───────┘
> (sort '(3 1 2) <) '(1 2 3)
> (sort '(3 1 2) < #:descending #t) '(3 2 1)
> (sort '((a . 1) (b . 2) (c . 1)) < #:key cdr #:descending #t) '((b . 2) (a . 1) (c . 1))
> (sort '(3 1 2) < #:nulls-last #t) sort: contract violation
expected: polars-only/c
given: #t
in: the nulls-last argument of
sort/c
contract from:
<pkgs>/polars/private/generic/ordering.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/ordering.rkt:17:24
procedure
(sort-by x #:by by [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → Expr-ptr? x : (or/c Expr-ptr? string?) by : (or/c Expr-ptr? string? (non-empty-listof (or/c Expr-ptr? string?))) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
> (define scores (dataframe (list (series '("ann" "bob" "cy" "dee") #:name "name") (series (list 3 polars-null 1 3) #:name "score"))))
> (select scores (sort-by "name" #:by "score" #:descending #t #:nulls-last #t #:maintain-order #t))
shape: (4, 1)
┌──────┐
│ name │
│ --- │
│ str │
╞══════╡
│ ann │
│ dee │
│ cy │
│ bob │
└──────┘
> (select scores (sort-by "name" #:by '("score" "name") #:descending '(#t #f)))
shape: (4, 1)
┌──────┐
│ name │
│ --- │
│ str │
╞══════╡
│ bob │
│ ann │
│ dee │
│ cy │
└──────┘
procedure
(select d spec ...) → (or/c dataframe? lazyframe?)
d : (or/c dataframe? lazyframe?) spec : any/c
procedure
(with-columns d spec ...) → (or/c dataframe? lazyframe?)
d : (or/c dataframe? lazyframe?) spec : any/c
> (~> (dataframe (list (series '(1 2 3) #:name "a") (series '(4 5 6) #:name "b"))) (select "a" (alias (+ (col "a") (col "b")) "sum")))
shape: (3, 2)
┌─────┬─────┐
│ a ┆ sum │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 5 │
│ 2 ┆ 7 │
│ 3 ┆ 9 │
└─────┴─────┘
> (~> (dataframe (list (series '(1 2 3) #:name "a"))) (with-columns (alias (* (col "a") 2) "double")))
shape: (3, 2)
┌─────┬────────┐
│ a ┆ double │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪════════╡
│ 1 ┆ 2 │
│ 2 ┆ 4 │
│ 3 ┆ 6 │
└─────┴────────┘
A cast to a zoned '(datetime unit "Zone/Name") keeps each instant: a naive datetime, or an integer count since the epoch, is read as UTC, as Python reads it. A cast from a zoned datetime to a naive one gives its UTC clock time; dt-replace-time-zone keeps the local clock instead. A zone polars does not know raises, with the nearest name it does.
A cast can make a date or datetime outside the years -262143 to 262142, which series and lit refuse. ref, the conversions and a series’ printing read it as the gregor value, but for a datetime in a zone other than UTC, whose local clock Polars cannot give: ref and the conversions raise with Polars’ reason, and the series prints that value as a placeholder. Polars cannot print one: a dataframe or expression holding one prints as a placeholder with Polars’ reason, and display-dataframe, expr->string and a cast to 'string raise it.
> (~> (dataframe (list (series '(1 2 3) #:name "v"))) (with-columns (cast "v" 'float64)))
shape: (3, 1)
┌─────┐
│ v │
│ --- │
│ f64 │
╞═════╡
│ 1.0 │
│ 2.0 │
│ 3.0 │
└─────┘
> (cast (series '("UA" "AA" "UA")) 'categorical)
shape: (3,)
Series: '' [cat]
[
"UA"
"AA"
"UA"
]
> (cast (series '(1 2 3)) 'f64)
shape: (3,)
Series: '' [f64]
[
1.0
2.0
3.0
]
> (define-enum carriers UA AA) > (cast (series '("UA" "B6")) carriers) cast: 1 of 2 values is not in '(enum UA AA): B6
> (cast (series (list (datetime 2021 3 27 12))) '(datetime microseconds "Europe/Brussels"))
shape: (1,)
Series: '' [datetime[us, Europe/Brussels]]
[
2021-03-27 13:00:00 CET
]
> (cast (series (list (datetime 2021 3 27 12))) '(datetime microseconds "Europe/Brusels")) cast: cannot convert to '(datetime microseconds
"Europe/Brusels"): unable to parse time zone:
'Europe/Brusels'. Please check the Time Zone Database for a
list of available time zones.
Hint: did you mean 'Europe/Brussels' instead?
> (define far (cast (series '(2000000000) #:dtype 'i32 #:name "d") 'date)) > (ref far 0) #<date 5477784-01-06>
> (dataframe (list far)) #<dataframe: cannot print: polars panicked: out-of-range date>
> (display-dataframe (dataframe (list far))) display-dataframe: cannot print the dataframe: polars
panicked: out-of-range date
> (define far-zoned (cast (series '(9000000000000000000) #:dtype 'i64 #:name "k") '(datetime microseconds "Europe/Brussels"))) > far-zoned
shape: (1,)
Series: 'k' [datetime[us, Europe/Brussels]]
[
#<cannot print: polars panicked: invalid or out-of-range datetime>
]
> (ref far-zoned 0) ref: cannot read a datetime in "Europe/Brussels": polars
panicked: invalid or out-of-range datetime
procedure
(vstack top bottom) → dataframe?
top : dataframe? bottom : dataframe?
> (define top (dataframe (list (series '(1 2) #:name "v")))) > (vstack top (dataframe (list (series '(3) #:name "v"))))
shape: (3, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 3 │
└─────┘
> (vstack top (dataframe (list (series '("3") #:name "v")))) vstack: type String is incompatible with expected type
Int64: failed to vstack column 'v'
procedure
(hstack d column ...) → dataframe?
d : dataframe? column : series?
> (hstack top (series '(10 20) #:name "w"))
shape: (2, 2)
┌─────┬─────┐
│ v ┆ w │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 10 │
│ 2 ┆ 20 │
└─────┴─────┘
> (hstack top (series '(10) #:name "w")) hstack: lengths don't match: could not create a new
DataFrame: height of column 'w' (1) does not match height of
column 'v' (2)
procedure
(with-column d column) → dataframe?
d : dataframe? column : series?
> (with-column top (series '(10 20) #:name "w"))
shape: (2, 2)
┌─────┬─────┐
│ v ┆ w │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 10 │
│ 2 ┆ 20 │
└─────┴─────┘
> (with-column top (series '(0) #:name "v"))
shape: (2, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 0 │
│ 0 │
└─────┘
> (shape (with-column (dataframe '()) (series '(1 2) #:name "w"))) '(2 1)
> (with-column top (series '(1 2 3) #:name "w")) with-column: unable to add a column of length 3 to a
DataFrame of height 2
procedure
x : (or/c series? dataframe? lazyframe? Expr-ptr?) n : (integer-in 0 18446744073709551615)
procedure
x : (or/c series? dataframe? lazyframe? Expr-ptr?) n : (integer-in 0 18446744073709551615)
procedure
x : (or/c series? dataframe? lazyframe? Expr-ptr?) offset : (integer-in -9223372036854775808 9223372036854775807) length : (integer-in 0 18446744073709551615)
> (define nums (dataframe (list (series '(1 2 3 4 5) #:name "v")))) > (head nums 2)
shape: (2, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
└─────┘
> (tail nums 2)
shape: (2, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 4 │
│ 5 │
└─────┘
> (slice nums -2 (expt 2 40))
shape: (2, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 4 │
│ 5 │
└─────┘
> (head nums -1) head: contract violation
expected: (integer-in 0 18446744073709551615)
given: -1
in: the 2nd argument of
(->
(or/c
series?
dataframe?
lazyframe?
Expr-ptr?)
(integer-in 0 18446744073709551615)
any/c)
contract from:
<pkgs>/polars/private/generic/reshape.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/reshape.rkt:25:11
procedure
(unique x) → (or/c series? dataframe?)
x : (or/c series? dataframe?)
procedure
(drop-nulls x) → any/c
x : (or/c series? dataframe? Expr-ptr?)
> (define dups (dataframe (list (series (list 1 1 2 polars-null) #:name "v") (series '("a" "a" "b" "c") #:name "w")))) > (sort (unique dups) "v")
shape: (3, 2)
┌──────┬─────┐
│ v ┆ w │
│ --- ┆ --- │
│ i64 ┆ str │
╞══════╪═════╡
│ null ┆ c │
│ 1 ┆ a │
│ 2 ┆ b │
└──────┴─────┘
> (series->list (drop-nulls (ref dups "v"))) '(1 1 2)
> (drop-nulls dups)
shape: (3, 2)
┌─────┬─────┐
│ v ┆ w │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ a │
│ 1 ┆ a │
│ 2 ┆ b │
└─────┴─────┘
> (unique '(1 1 2)) unique: expected a series or dataframe, got '(1 1 2)
procedure
(drop d names) → dataframe?
d : dataframe? names : (or/c string? (listof string?))
> (define pair (dataframe (list (series '(1) #:name "a") (series '(2) #:name "b")))) > (drop pair "a")
shape: (1, 1)
┌─────┐
│ b │
│ --- │
│ i64 │
╞═════╡
│ 2 │
└─────┘
> (drop pair '("a" "nope")) drop: "nope" not found
> (drop '(1 2 3) 1) '(2 3)
procedure
(pivot d #:on on #:index index #:values values [ #:agg agg]) → dataframe? d : dataframe? on : (listof string?) index : (listof string?) values : (listof string?)
agg : (or/c #f 'none 'first 'sum 'min 'max 'mean 'count) = 'first
> (define sales (dataframe (list (series '("a" "a" "b") #:name "store") (series '("q1" "q2" "q1") #:name "quarter") (series '(10 20 30) #:name "sales"))))
> (define by-quarter (sort (pivot sales #:on '("quarter") #:index '("store") #:values '("sales") #:agg 'sum) "store")) > by-quarter
shape: (2, 3)
┌───────┬─────┬─────┐
│ store ┆ q1 ┆ q2 │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ a ┆ 10 ┆ 20 │
│ b ┆ 30 ┆ 0 │
└───────┴─────┴─────┘
> (pivot sales #:on '("nope") #:index '("store") #:values '("sales")) pivot: cannot pivot the dataframe: not found: "nope" not
found
procedure
(unpivot d #:on on #:index index) → dataframe?
d : dataframe? on : (listof string?) index : (listof string?)
> (unpivot by-quarter #:on '("q1" "q2") #:index '("store"))
shape: (4, 3)
┌───────┬──────────┬───────┐
│ store ┆ variable ┆ value │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞═══════╪══════════╪═══════╡
│ a ┆ q1 ┆ 10 │
│ b ┆ q1 ┆ 30 │
│ a ┆ q2 ┆ 20 │
│ b ┆ q2 ┆ 0 │
└───────┴──────────┴───────┘
> (unpivot by-quarter #:on '("nope") #:index '("store")) unpivot: cannot unpivot the dataframe: not found: "nope" not
found
procedure
(join left right [ #:on on #:left-on left-on #:right-on right-on #:how how]) → (or/c dataframe? lazyframe?) left : (or/c dataframe? lazyframe?) right : (or/c dataframe? lazyframe?) on : (or/c #f (non-empty-listof string?)) = #f left-on : (or/c #f (non-empty-listof string?)) = #f right-on : (or/c #f (non-empty-listof string?)) = #f
how : (or/c 'inner 'left 'outer 'full 'cross 'semi 'anti) = 'inner
> (define left (dataframe (list (series '("a" "b") #:name "k") (series '(1 2) #:name "v"))))
> (define right (dataframe (list (series '("a" "c") #:name "k") (series '(10 30) #:name "w")))) > (~> (join left right #:on '("k") #:how 'left) (sort "k"))
shape: (2, 3)
┌─────┬─────┬──────┐
│ k ┆ v ┆ w │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪═════╪══════╡
│ a ┆ 1 ┆ 10 │
│ b ┆ 2 ┆ null │
└─────┴─────┴──────┘
> (join left right #:on '("k") #:how 'inner)
shape: (1, 3)
┌─────┬─────┬─────┐
│ k ┆ v ┆ w │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ a ┆ 1 ┆ 10 │
└─────┴─────┴─────┘
> (join left (dataframe (list (series '(1 2) #:name "k"))) #:on '("k")) join: datatypes of join keys don't match - `k`: str on left
does not match `k`: i64 on right
> (join left right) join: contract violation
must supply #:on, or #:left-on and #:right-on
in: join/c
contract from:
<pkgs>/polars/private/generic/combine.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/combine.rkt:14:11
procedure
(join-asof left right [ #:on on #:left-on left-on #:right-on right-on #:by by #:left-by left-by #:right-by right-by #:strategy strategy #:tolerance tolerance]) → dataframe? left : dataframe? right : dataframe? on : (or/c #f string?) = #f left-on : (or/c #f string?) = #f right-on : (or/c #f string?) = #f by : (or/c #f (listof string?)) = #f left-by : (or/c #f (listof string?)) = #f right-by : (or/c #f (listof string?)) = #f strategy : (or/c 'backward 'forward 'nearest) = 'backward
tolerance :
(or/c #f (integer-in (- (expt 2 63)) (sub1 (expt 2 63))) (and/c real? (not/c exact-integer?))) = #f
> (define readings (dataframe (list (series '(1 3 5) #:name "t") (series '(100 300 500) #:name "v"))))
> (define offsets (dataframe (list (series '(1 2 4) #:name "t") (series '(10 20 40) #:name "o")))) > (join-asof readings offsets #:on "t")
shape: (3, 3)
┌─────┬─────┬─────┐
│ t ┆ v ┆ o │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 100 ┆ 10 │
│ 3 ┆ 300 ┆ 20 │
│ 5 ┆ 500 ┆ 40 │
└─────┴─────┴─────┘
> (join-asof readings offsets #:on "t" #:tolerance 0)
shape: (3, 3)
┌─────┬─────┬──────┐
│ t ┆ v ┆ o │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪══════╡
│ 1 ┆ 100 ┆ 10 │
│ 3 ┆ 300 ┆ null │
│ 5 ┆ 500 ┆ null │
└─────┴─────┴──────┘
> (join-asof readings offsets #:on "t" #:left-by '("v")) join-asof: contract violation
must supply both #:left-by and #:right-by
in: join-asof/c
contract from:
<pkgs>/polars/private/generic/combine.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/combine.rkt:15:11
procedure
(read-csv path [ #:has-header has-header #:columns columns #:new-columns new-columns #:separator separator #:comment-prefix comment-prefix #:quote-char quote-char #:skip-rows skip-rows #:skip-lines skip-lines #:schema-overrides schema-overrides #:null-values null-values #:missing-utf8-is-empty-string missing-utf8-is-empty-string #:ignore-errors ignore-errors #:try-parse-dates try-parse-dates #:infer-schema infer-schema #:infer-schema-length infer-schema-length #:n-rows n-rows #:encoding encoding #:skip-rows-after-header skip-rows-after-header #:row-index-name row-index-name #:row-index-offset row-index-offset #:eol-char eol-char #:raise-if-empty raise-if-empty #:truncate-ragged-lines truncate-ragged-lines #:decimal-comma decimal-comma #:glob glob]) → dataframe? path : path-string? has-header : boolean? = #t
columns : (or/c (and/c (listof column-name/c) distinct?) (and/c (listof size/c) distinct?)) = '() new-columns : (and/c (listof column-name/c) distinct?) = '() separator : (or/c csv-char/c #f) = #f
comment-prefix : (or/c (and/c non-empty-string? c-string/c) #f) = #f quote-char : (or/c csv-char/c #f) = #\" skip-rows : size/c = 0 skip-lines : size/c = 0
schema-overrides : (and/c (listof (cons/c column-name/c csv-dtype/c)) distinct-names?) = '()
null-values :
(or/c c-string/c (listof c-string/c) (and/c (listof (cons/c column-name/c c-string/c)) distinct-names?) #f) = #f missing-utf8-is-empty-string : boolean? = #f ignore-errors : boolean? = #f try-parse-dates : boolean? = #f infer-schema : boolean? = #t infer-schema-length : (or/c size/c #f) = 100 n-rows : (or/c size/c #f) = #f encoding : (or/c 'utf8 'utf8-lossy) = 'utf8 skip-rows-after-header : size/c = 0 row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 eol-char : eol-char/c = #\newline raise-if-empty : boolean? = #t truncate-ragged-lines : boolean? = #f decimal-comma : boolean? = #f glob : boolean? = #t
A csv-char/c is one ASCII character other than newline or return. A size/c is a count or position Polars takes as a native size, an unsigned 64-bit integer, a positive-size/c one from 1, and a row-index/c an unsigned 32-bit integer. A c-string/c is a string without a NUL, and so is a column-name/c. separator defaults to #\,; quote-char must differ from it, and #:quote-char #f turns quoting off. An eol-char/c is any ASCII character; eol-char ends a line and must differ from both (a return before a newline is dropped, so \r\n files need no option). Lines that start with comment-prefix are skipped. The first skip-rows rows of each file are skipped, quoting respected; skip-lines skips lines instead, ignoring quotes, and only one of the two may be set. skip-rows-after-header skips rows after the header, and n-rows caps the rows read. A field equal to one of null-values reads as null; given as an association list of column names and markers, each marker applies to its column only, and naming a column the file lacks is an error. A missing field reads as null, or as "" in a string column with #:missing-utf8-is-empty-string #t. Column types are inferred from the first infer-schema-length rows: #f reads every row, and 0, or #:infer-schema #f, makes every column a string. A count whose reservation, 24 bytes a row, exceeds the host’s memory (a container’s cgroup limit where one is set, or POLARS_OVERRIDE_TOTAL_MEMORY bytes, a malformed value counting as 0), or above the rows a local input could hold (one a byte, after the most its compression can expand), reads every row, as #f does. Deciding the second stats the inputs when the read or scan is built, so a pipe, a device or a file created after scan-csv never bounds the count, and a count past a file’s size then reads every row even if the file grows (or reports a size of 0 though it has content). Polars reserves room for the count before it reads, so a smaller count that is still enormous can exhaust memory and abort the process. schema-overrides fixes the named columns’ types. A csv-dtype/c is any spelling series’ #:dtype accepts except a duration and 'null, which Polars cannot read from CSV, an Enum and a zoned datetime (#170); API gap: read the column as 'categorical, 'string or a naive datetime and cast it. Each column appears at most once (distinct-names?), and naming a column the file lacks is an error. With #:ignore-errors #t a field that does not parse reads as null. #:try-parse-dates #t reads ISO dates, times of day and datetimes as 'date, 'time and 'datetime columns, and datetimes with a UTC offset as '(datetime microseconds "UTC"), as Python does. #:decimal-comma #t reads 1,5 as 1.5. 'utf8-lossy replaces invalid UTF-8 with U+FFFD. A line with more fields than the header is an error, unless #:truncate-ragged-lines #t drops the extra fields; an empty file is an error, unless #:raise-if-empty #f reads it as an empty frame.
new-columns renames the file’s first columns, header or
"column_1"-style names alike; more names than the file has
columns, or a name that ends up twice, is an error. columns reads
only the named columns, or those at the given zero-based positions, in the
file’s order; the empty list reads them all. row-index-name adds
a first column, 'uint32, numbering the rows from
row-index-offset; a name the file already uses is an error, and so
is a row number past 4294967295. Every keyword that names a
column —
A gzip, zlib or zstd file is decompressed as it is read, judged by its first four bytes, so a plain file that starts like a zlib stream (a header beginning x^) fails to read or reads as garbage; a UTF-8 byte order mark at the start of the file avoids it. API gaps: no schema, with_column_names, include_file_paths or missing_columns.
The result is (collect (scan-csv path ....)) with the same
keywords but #:columns, which scan-csv lacks:
select from the plan instead. A single file is read eagerly
rather than through a plan, which is faster. One check is
read-csv’s own. When separator is
#f, new-columns and columns are empty and the
file reads as one column, a row index aside, whose header splits on a tab, ; or
|, read-csv raises an error naming the separator to
pass —
A failure names the operation, the path and the cause: the operating system’s for a file that cannot be opened, Polars’ own for input it cannot parse.
The examples read "flights.tsv", 102 rows of the nycflights13 data with NA for a missing value. Without #:separator the separator check stops the read; with it, the first NA fails to parse, and #:null-values fixes that:
> (read-csv "flights.tsv") read-csv: flights.tsv reads as one column whose header
splits on #\tab into 19 fields; pass #:separator #\tab, or
#:separator #\, to keep one column
> (read-csv "flights.tsv" #:separator #\tab) read-csv: failed to read csv from flights.tsv: could not
parse `NA` as dtype `i64` at column 'arr_delay' (column
number 9)
The current offset in the file is 648 bytes.
You might want to try:
- increasing #:infer-schema-length (e.g.
#:infer-schema-length 10000, or #f for every row),
- specifying correct dtype with #:schema-overrides
- setting #:ignore-errors to #t,
- adding `NA` to #:null-values.
Original error: ```invalid primitive value found during CSV
parsing```
> (define flights (read-csv "flights.tsv" #:separator #\tab #:null-values "NA")) > (shape flights) '(102 19)
> (null-count (ref flights #:columns "dep_delay")) 1
> (~> (read-csv "flights.tsv" #:separator #\tab #:null-values '("NA" "")) (ref #:columns "arr_delay") null-count) 2
Failures. A missing file or an unwritable path reports the operating system’s reason; a file in the wrong format reports Polars’:
> (read-csv "/no/such/file.csv") read-csv: failed to read csv from /no/such/file.csv: cannot
open file: No such file or directory (os error 2)
> (define small (dataframe (list (series '(1 2) #:name "v")))) > (write-parquet small "/no/such/dir/out.parquet") write-parquet: failed to write parquet to
/no/such/dir/out.parquet: cannot create file: No such file
or directory (os error 2)
Here not-parquet is a path in the temporary directory:
> (write-csv small not-parquet) > (read-parquet not-parquet) read-parquet: failed to read parquet from
/var/tmp/polars-not-parquet.csv: parquet: File out of
specification: A Parquet file must contain a header and
footer with at least 12 bytes
> (read-ndjson not-parquet) read-ndjson: failed to read ndjson from
/var/tmp/polars-not-parquet.csv: InternalError(TapeError) at
character 0 ('v')
Types. #:ignore-errors turns what does not parse into nulls; #:infer-schema-length widens or narrows the rows types are inferred from; #:schema-overrides and #:try-parse-dates set them outright. An override for a column the file lacks is an error:
> (~> (read-csv "flights.tsv" #:separator #\tab #:ignore-errors #t) (ref #:columns "dep_delay") null-count) 1
> (~> (read-csv "flights.tsv" #:separator #\tab #:infer-schema-length #f) (ref #:columns "dep_delay") dtype) 'string
> (~> (read-csv "flights.tsv" #:separator #\tab #:infer-schema-length 0) (ref #:columns "year") dtype) 'string
> (~> (read-csv "flights.tsv" #:separator #\tab #:null-values "NA" #:try-parse-dates #t #:schema-overrides '(("dep_delay" . f64) ("flight" . int32))) (select "dep_delay" "flight" "time_hour") (tail 3))
shape: (3, 3)
┌───────────┬────────┬─────────────────────┐
│ dep_delay ┆ flight ┆ time_hour │
│ --- ┆ --- ┆ --- │
│ f64 ┆ i32 ┆ datetime[μs] │
╞═══════════╪════════╪═════════════════════╡
│ -7.0 ┆ 1733 ┆ 2013-01-01 07:00:00 │
│ -5.0 ┆ 4525 ┆ 2013-01-01 15:00:00 │
│ null ┆ 4308 ┆ 2013-01-01 16:00:00 │
└───────────┴────────┴─────────────────────┘
> (read-csv "flights.tsv" #:separator #\tab #:schema-overrides '(("dep_dealy" . f64))) read-csv: failed to read csv from flights.tsv: schema
overrides name columns not in the file: "dep_dealy"
Layout. "notes.csv" has a comment line, ; between fields and ' around a field that holds one; "latin1.csv" is not UTF-8:
> (read-csv "notes.csv" #:separator #\; #:comment-prefix "#" #:quote-char #\')
shape: (2, 2)
┌─────────┬───────────────┐
│ carrier ┆ note │
│ --- ┆ --- │
│ str ┆ str │
╞═════════╪═══════════════╡
│ UA ┆ late; weather │
│ AA ┆ on time │
└─────────┴───────────────┘
> (read-csv "notes.csv" #:separator #\; #:comment-prefix "#") read-csv: failed to read csv from notes.csv: found more
fields than defined in 'Schema'
Consider setting #:truncate-ragged-lines #t.
> (read-csv "parts/part-1.csv" #:has-header #f #:skip-rows 1)
shape: (2, 3)
┌──────────┬──────────┬──────────┐
│ column_1 ┆ column_2 ┆ column_3 │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞══════════╪══════════╪══════════╡
│ EWR ┆ IAH ┆ 2 │
│ LGA ┆ IAH ┆ 4 │
└──────────┴──────────┴──────────┘
> (~> (read-csv "flights.tsv" #:separator #\tab #:null-values "NA" #:n-rows 2) (select "carrier" "flight" "dep_delay"))
shape: (2, 3)
┌─────────┬────────┬───────────┐
│ carrier ┆ flight ┆ dep_delay │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════════╪════════╪═══════════╡
│ UA ┆ 1545 ┆ 2 │
│ UA ┆ 1714 ┆ 4 │
└─────────┴────────┴───────────┘
> (read-csv "latin1.csv") read-csv: failed to read csv from latin1.csv: invalid utf-8
sequence
> (read-csv "latin1.csv" #:encoding 'utf8-lossy)
shape: (2, 2)
┌─────────┬──────────┐
│ carrier ┆ name │
│ --- ┆ --- │
│ str ┆ str │
╞═════════╪══════════╡
│ B6 ┆ JetBlue │
│ ZZ ┆ Caf� Air │
└─────────┴──────────┘
Files. A pattern reads every match, in filename order; one that matches nothing, a literal path with #:glob #f, and a directory are errors:
> (read-csv "parts/*.csv")
shape: (6, 3)
┌────────┬──────┬───────────┐
│ origin ┆ dest ┆ dep_delay │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞════════╪══════╪═══════════╡
│ EWR ┆ IAH ┆ 2 │
│ LGA ┆ IAH ┆ 4 │
│ JFK ┆ MIA ┆ 2 │
│ JFK ┆ BQN ┆ -1 │
│ LGA ┆ ATL ┆ -6 │
│ EWR ┆ ORD ┆ -4 │
└────────┴──────┴───────────┘
> (read-csv "parts/*.tsv") read-csv: failed to read csv from parts/*.tsv: no files
match the pattern
> (read-csv "parts/part-?.csv" #:glob #f) read-csv: failed to read csv from parts/part-?.csv: cannot
open file: No such file or directory (os error 2)
> (read-csv "parts") read-csv: failed to read csv from parts: cannot open file:
it is a directory; pass a glob pattern such as dir/*.csv
Columns. #:columns picks columns by name or position, in the file’s order; #:new-columns renames the first ones, and #:row-index-name numbers the rows:
> (read-csv "flights.tsv" #:separator #\tab #:null-values "NA" #:columns '("carrier" "flight" "dep_delay") #:row-index-name "row" #:row-index-offset 1 #:n-rows 3)
shape: (3, 4)
┌─────┬───────────┬─────────┬────────┐
│ row ┆ dep_delay ┆ carrier ┆ flight │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ str ┆ i64 │
╞═════╪═══════════╪═════════╪════════╡
│ 1 ┆ 2 ┆ UA ┆ 1545 │
│ 2 ┆ 4 ┆ UA ┆ 1714 │
│ 3 ┆ 2 ┆ AA ┆ 1141 │
└─────┴───────────┴─────────┴────────┘
> (read-csv "flights.tsv" #:separator #\tab #:columns '(9 10) #:n-rows 2)
shape: (2, 2)
┌─────────┬────────┐
│ carrier ┆ flight │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪════════╡
│ UA ┆ 1545 │
│ UA ┆ 1714 │
└─────────┴────────┘
> (read-csv "flights.tsv" #:separator #\tab #:columns '(42)) read-csv: failed to read csv from flights.tsv: projection
index: 42 is out of bounds for csv schema with length: 19
> (read-csv "parts/part-1.csv" #:has-header #f #:skip-rows 1 #:new-columns '("from" "to" "delay"))
shape: (2, 3)
┌──────┬─────┬───────┐
│ from ┆ to ┆ delay │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞══════╪═════╪═══════╡
│ EWR ┆ IAH ┆ 2 │
│ LGA ┆ IAH ┆ 4 │
└──────┴─────┴───────┘
> (read-csv "parts/part-1.csv" #:new-columns '("a" "b" "c" "d")) read-csv: failed to read csv from parts/part-1.csv: 4 new
column names for a file of 3 columns
> (read-csv "parts/part-1.csv" #:row-index-name "origin") read-csv: failed to read csv from parts/part-1.csv:
duplicate: cannot add row_index with name 'origin': column
already exists in file.
Messy files. "stations.csv" opens with a line that is not data and holds a quote, separates fields with ;, writes decimals with a comma, marks a missing temperature - and missing rain n/a, and has a line with a field too many. The note - is text, so the markers are per column:
> (read-csv "stations.csv" #:skip-lines 1 #:separator #\;) read-csv: failed to read csv from stations.csv: found more
fields than defined in 'Schema'
Consider setting #:truncate-ragged-lines #t.
> (define stations (read-csv "stations.csv" #:skip-lines 1 #:separator #\; #:decimal-comma #t #:null-values '(("temp" . "-") ("rain" . "n/a")) #:truncate-ragged-lines #t #:try-parse-dates #t)) > stations
shape: (4, 6)
┌─────────┬────────────┬──────────┬──────┬──────┬───────┐
│ station ┆ day ┆ at ┆ temp ┆ rain ┆ note │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ date ┆ time ┆ f64 ┆ f64 ┆ str │
╞═════════╪════════════╪══════════╪══════╪══════╪═══════╡
│ Oslo ┆ 2024-05-01 ┆ 06:00:00 ┆ 12.5 ┆ 0.0 ┆ - │
│ Bergen ┆ 2024-05-01 ┆ 06:30:00 ┆ null ┆ 3.2 ┆ null │
│ Tromso ┆ 2024-05-02 ┆ 07:15:00 ┆ 4.0 ┆ null ┆ cold │
│ Narvik ┆ 2024-05-02 ┆ 08:00:00 ┆ 3.5 ┆ 1.0 ┆ windy │
└─────────┴────────────┴──────────┴──────┴──────┴───────┘
> (~> (read-csv "stations.csv" #:skip-lines 1 #:separator #\; #:decimal-comma #t #:null-values '(("temp" . "-") ("rain" . "n/a")) #:truncate-ragged-lines #t #:missing-utf8-is-empty-string #t) (select "station" "note"))
shape: (4, 2)
┌─────────┬───────┐
│ station ┆ note │
│ --- ┆ --- │
│ str ┆ str │
╞═════════╪═══════╡
│ Oslo ┆ - │
│ Bergen ┆ │
│ Tromso ┆ cold │
│ Narvik ┆ windy │
└─────────┴───────┘
> (read-csv "stations.csv" #:skip-lines 1 #:separator #\; #:null-values '(("temperature" . "-"))) read-csv: failed to read csv from stations.csv: not found:
unable to find column "temperature"; valid columns:
["station", "day", "at", "temp", "rain", "note"]
> (read-csv "parts/part-1.csv" #:skip-rows-after-header 1)
shape: (1, 3)
┌────────┬──────┬───────────┐
│ origin ┆ dest ┆ dep_delay │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞════════╪══════╪═══════════╡
│ LGA ┆ IAH ┆ 4 │
└────────┴──────┴───────────┘
> (~> (read-csv "flights.tsv" #:separator #\tab #:infer-schema #f) (ref #:columns "year") dtype) 'string
Line ends and empty files. "eol.csv" ends each line with ;, and nothing is an empty file in the temporary directory:
> (read-csv "eol.csv" #:eol-char #\;)
shape: (2, 2)
┌─────────┬────────┐
│ carrier ┆ flight │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪════════╡
│ UA ┆ 1545 │
│ AA ┆ 1141 │
└─────────┴────────┘
> (collect (scan-csv "eol.csv" #:eol-char #\; #:row-index-name "row" #:row-index-offset 1))
shape: (2, 3)
┌─────┬─────────┬────────┐
│ row ┆ carrier ┆ flight │
│ --- ┆ --- ┆ --- │
│ u32 ┆ str ┆ i64 │
╞═════╪═════════╪════════╡
│ 1 ┆ UA ┆ 1545 │
│ 2 ┆ AA ┆ 1141 │
└─────┴─────────┴────────┘
> (read-csv nothing) read-csv: failed to read csv from
/var/tmp/polars-doc-nothing.csv: no data: empty CSV
> (read-csv nothing #:raise-if-empty #f)
shape: (0, 0)
┌┐
╞╡
└┘
procedure
(scan-csv path [ #:has-header has-header #:new-columns new-columns #:separator separator #:comment-prefix comment-prefix #:quote-char quote-char #:skip-rows skip-rows #:skip-lines skip-lines #:schema-overrides schema-overrides #:null-values null-values #:missing-utf8-is-empty-string missing-utf8-is-empty-string #:ignore-errors ignore-errors #:try-parse-dates try-parse-dates #:infer-schema infer-schema #:infer-schema-length infer-schema-length #:n-rows n-rows #:encoding encoding #:skip-rows-after-header skip-rows-after-header #:row-index-name row-index-name #:row-index-offset row-index-offset #:eol-char eol-char #:raise-if-empty raise-if-empty #:truncate-ragged-lines truncate-ragged-lines #:decimal-comma decimal-comma #:glob glob]) → lazyframe? path : path-string? has-header : boolean? = #t new-columns : (and/c (listof column-name/c) distinct?) = '() separator : (or/c csv-char/c #f) = #f
comment-prefix : (or/c (and/c non-empty-string? c-string/c) #f) = #f quote-char : (or/c csv-char/c #f) = #\" skip-rows : size/c = 0 skip-lines : size/c = 0
schema-overrides : (and/c (listof (cons/c column-name/c csv-dtype/c)) distinct-names?) = '()
null-values :
(or/c c-string/c (listof c-string/c) (and/c (listof (cons/c column-name/c c-string/c)) distinct-names?) #f) = #f missing-utf8-is-empty-string : boolean? = #f ignore-errors : boolean? = #f try-parse-dates : boolean? = #f infer-schema : boolean? = #t infer-schema-length : (or/c size/c #f) = 100 n-rows : (or/c size/c #f) = #f encoding : (or/c 'utf8 'utf8-lossy) = 'utf8 skip-rows-after-header : size/c = 0 row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 eol-char : eol-char/c = #\newline raise-if-empty : boolean? = #t truncate-ragged-lines : boolean? = #f decimal-comma : boolean? = #f glob : boolean? = #t
> (~> (scan-csv "flights.tsv" #:separator #\tab #:null-values "NA") (filter (> (col "dep_delay") 30)) (select "carrier" "dep_delay") collect)
shape: (2, 2)
┌─────────┬───────────┐
│ carrier ┆ dep_delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════════╡
│ UA ┆ 47 │
│ MQ ┆ 39 │
└─────────┴───────────┘
> (~> (scan-csv "parts/*.csv") (group-by "origin") (agg (sum "dep_delay")) collect)
shape: (3, 2)
┌────────┬───────────┐
│ origin ┆ dep_delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞════════╪═══════════╡
│ JFK ┆ 1 │
│ EWR ┆ -2 │
│ LGA ┆ -2 │
└────────┴───────────┘
> (shape (collect (scan-csv "flights.tsv"))) '(102 1)
> (define plan (scan-csv "/no/such/file.csv")) > (collect plan) collect: No such file or directory (os error 2):
/no/such/file.csv
> (define no-match (scan-csv "parts/*.tsv")) > (collect no-match) collect: no files match the pattern
> (scan-csv "flights.tsv" #:separator #\tab #:schema-overrides '(("dep_dealy" . f64))) scan-csv: failed to scan flights.tsv: schema overrides name
columns not in the file: "dep_dealy"
> (~> (scan-csv "parts/part-1.csv" #:new-columns '("from" "to") #:row-index-name "row") (filter (> (col "dep_delay") 3)) collect)
shape: (1, 4)
┌─────┬──────┬─────┬───────────┐
│ row ┆ from ┆ to ┆ dep_delay │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ str ┆ str ┆ i64 │
╞═════╪══════╪═════╪═══════════╡
│ 1 ┆ LGA ┆ IAH ┆ 4 │
└─────┴──────┴─────┴───────────┘
> (scan-csv "/no/such/file.csv" #:new-columns '("from" "to")) scan-csv: failed to scan /no/such/file.csv: cannot open
file: No such file or directory (os error 2)
procedure
(write-csv frame path [ #:include-bom include-bom #:include-header include-header #:separator separator #:line-terminator line-terminator #:quote-char quote-char #:batch-size batch-size #:datetime-format datetime-format #:date-format date-format #:time-format time-format #:float-scientific float-scientific #:float-precision float-precision #:decimal-comma decimal-comma #:null-value null-value #:quote-style quote-style]) → void? frame : dataframe? path : path-string? include-bom : boolean? = #f include-header : boolean? = #t separator : csv-char/c = #\, line-terminator : c-string/c = "\n" quote-char : csv-char/c = #\" batch-size : positive-size/c = 1024 datetime-format : (or/c c-string/c #f) = #f date-format : (or/c c-string/c #f) = #f time-format : (or/c c-string/c #f) = #f float-scientific : (or/c 'auto boolean?) = 'auto float-precision : (or/c float-precision/c #f) = #f decimal-comma : boolean? = #f null-value : c-string/c = ""
quote-style : (or/c 'necessary 'always 'non-numeric 'never) = 'necessary
datetime-format, date-format and time-format are chrono format strings, by default ISO 8601 at the column’s precision; a zoned datetime is written as its local time and UTC offset, 2021-03-27T12:00:00.000000+0100, as Python does. float-precision fixes the digits after the decimal point; a float-precision/c is at most 65535, the most Rust formats; float-scientific writes every float in scientific notation (#t), none (#f), or as Polars does by default ('auto): the shortest digits that read back exactly, in scientific notation for very small or large magnitudes. #:decimal-comma #t writes 1,5, quoting it when the separator is a comma.
An existing file is replaced atomically, keeping its permission bits but not its owner, ACLs, extended attributes or hard links; the directory must be writable. A failure names the path and the cause, the operating system’s for a path that cannot be created and Polars’ for a value it cannot format, and leaves an existing file as it was. API gap: no compressed output (compression).
Here out is a path in the temporary directory, and stations the frame read above:
> (write-csv stations out #:separator #\; #:decimal-comma #t #:null-value "-" #:date-format "%d.%m.%Y" #:time-format "%H:%M") > (display (file->string out))
station;day;at;temp;rain;note
Oslo;01.05.2024;06:00;12,5;0,0;-
Bergen;01.05.2024;06:30;-;3,2;-
Tromso;02.05.2024;07:15;4,0;-;cold
Narvik;02.05.2024;08:00;3,5;1,0;windy
> (write-csv (select stations "station" "note") out #:include-header #f #:quote-style 'always #:line-terminator "\r\n") > (file->string out) "\"Oslo\",\"-\"\r\n\"Bergen\",\"\"\r\n\"Tromso\",\"cold\"\r\n\"Narvik\",\"windy\"\r\n"
> (write-csv (select stations "station" "temp") out #:float-precision 2) > (display (file->string out))
station,temp
Oslo,12.50
Bergen,
Tromso,4.00
Narvik,3.50
> (write-csv (select stations "station" "temp") out #:float-scientific #t) > (display (file->string out))
station,temp
Oslo,1.25e1
Bergen,
Tromso,4e0
Narvik,3.5e0
> (write-csv (~> (read-csv "flights.tsv" #:separator #\tab #:null-values "NA" #:try-parse-dates #t #:n-rows 2) (select "carrier" "flight" "time_hour")) out #:datetime-format "%Y-%m-%dT%H:%M" #:quote-style 'non-numeric #:include-bom #t #:batch-size 1) > (file->string out) "\uFEFF\"carrier\",\"flight\",\"time_hour\"\n\"UA\",1545,\"2013-01-01T05:00\"\n\"UA\",1714,\"2013-01-01T05:00\"\n"
> (write-csv stations out #:date-format "%H") write-csv: failed to write csv to
/var/tmp/polars-doc-out.csv: cannot format NaiveDate with
format '%H'
> (write-csv stations "/no/such/dir/out.csv") write-csv: failed to write csv to /no/such/dir/out.csv:
cannot create file: No such file or directory (os error 2)
procedure
(read-parquet path [ #:columns columns #:n-rows n-rows #:row-index-name row-index-name #:row-index-offset row-index-offset #:parallel parallel #:use-statistics use-statistics #:glob glob #:rechunk rechunk #:low-memory low-memory #:include-file-paths include-file-paths #:missing-columns missing-columns]) → dataframe? path : path-string?
columns : (or/c (listof column-name/c) (listof column-index/c) #f) = #f n-rows : (or/c size/c #f) = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 parallel : parquet-parallel/c = 'auto use-statistics : boolean? = #t glob : boolean? = #t rechunk : boolean? = #f low-memory : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f missing-columns : (or/c 'raise 'insert) = 'raise
columns picks columns by name, or by position from 0 (negative counts from the end, a column-index/c being a signed 64-bit integer), in the order given; a name the file lacks, a position out of range or a column picked twice is an error. A name is matched literally. n-rows caps the rows read. row-index-name adds a first column numbering the rows from row-index-offset; a row number past 4294967295 is an error. include-file-paths adds a column of that name holding each row’s file as a complete path. When the files a pattern matches differ, a file that lacks a column of the first is an error unless #:missing-columns 'insert, which fills it with nulls. parallel, a parquet-parallel/c ('auto, 'columns, 'row-groups, 'prefiltered or 'none), sets how the reader divides the work (over columns, row groups, both after evaluating a pushed-down filter, or not at all), #:use-statistics #f stops it skipping row groups by their statistics, low-memory trades speed for memory and rechunk makes each column contiguous; none of the four changes the result.
Categorical, Enum and Decimal columns keep their dtypes (Categorical, Enum and Decimal), and a datetime column its unit and time zone. A file Polars cannot read raises exn:fail with Polars’ reason, even where Polars itself panics. API gaps: no schema, extra_columns, use_pyarrow or hive options.
> (read-parquet "produce.parquet")
shape: (4, 3)
┌───────┬───────┬───────────────┐
│ item ┆ grade ┆ price │
│ --- ┆ --- ┆ --- │
│ cat ┆ enum ┆ decimal[10,2] │
╞═══════╪═══════╪═══════════════╡
│ apple ┆ high ┆ 1.25 │
│ pear ┆ low ┆ 0.80 │
│ apple ┆ null ┆ null │
│ fig ┆ mid ┆ 12.00 │
└───────┴───────┴───────────────┘
> (read-parquet (build-path parquet-dir "*.parquet"))
shape: (6, 3)
┌────────┬──────┬───────────┐
│ origin ┆ dest ┆ dep_delay │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞════════╪══════╪═══════════╡
│ EWR ┆ IAH ┆ 2 │
│ LGA ┆ IAH ┆ 4 │
│ JFK ┆ MIA ┆ 2 │
│ JFK ┆ BQN ┆ -1 │
│ LGA ┆ ATL ┆ -6 │
│ EWR ┆ ORD ┆ -4 │
└────────┴──────┴───────────┘
> (read-parquet "flights.parquet" #:columns '("carrier" "dep_delay") #:n-rows 3)
shape: (3, 2)
┌─────────┬───────────┐
│ carrier ┆ dep_delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════════╡
│ UA ┆ 2 │
│ UA ┆ 4 │
│ AA ┆ 2 │
└─────────┴───────────┘
> (read-parquet "flights.parquet" #:columns '(0 1 -1) #:n-rows 2 #:row-index-name "row" #:row-index-offset 1)
shape: (2, 3)
┌─────┬──────┬─────────────────────┐
│ row ┆ year ┆ time_hour │
│ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ datetime[μs] │
╞═════╪══════╪═════════════════════╡
│ 1 ┆ 2013 ┆ 2013-01-01 05:00:00 │
│ 2 ┆ 2013 ┆ 2013-01-01 05:00:00 │
└─────┴──────┴─────────────────────┘
> (column-names (read-parquet "produce.parquet" #:include-file-paths "file")) '("item" "grade" "price" "file")
> (shape (read-parquet "flights.parquet" #:parallel 'row-groups #:use-statistics #f #:low-memory #t #:rechunk #t)) '(102 19)
> (shape (read-parquet "produce.parque?")) '(4 3)
> (read-parquet "produce.parque?" #:glob #f) read-parquet: failed to read parquet from produce.parque?:
cannot open file: No such file or directory (os error 2)
> (read-parquet "produce.parquet" #:columns '("item" "colour")) read-parquet: failed to read parquet from produce.parquet:
columns not in the file: "colour"
> (parameterize ([current-directory parquet-dir]) (read-parquet "mixed/*.parquet")) read-parquet: failed to read parquet from mixed/*.parquet:
not found: did not find column grade, consider passing
#:missing-columns 'insert
> (parameterize ([current-directory parquet-dir]) (read-parquet "mixed/*.parquet" #:missing-columns 'insert))
shape: (8, 3)
┌───────┬───────┬───────────────┐
│ item ┆ grade ┆ price │
│ --- ┆ --- ┆ --- │
│ cat ┆ enum ┆ decimal[10,2] │
╞═══════╪═══════╪═══════════════╡
│ apple ┆ high ┆ 1.25 │
│ pear ┆ low ┆ 0.80 │
│ apple ┆ null ┆ null │
│ fig ┆ mid ┆ 12.00 │
│ apple ┆ null ┆ null │
│ pear ┆ null ┆ null │
│ apple ┆ null ┆ null │
│ fig ┆ null ┆ null │
└───────┴───────┴───────────────┘
procedure
(scan-parquet path [ #:n-rows n-rows #:row-index-name row-index-name #:row-index-offset row-index-offset #:parallel parallel #:use-statistics use-statistics #:glob glob #:rechunk rechunk #:low-memory low-memory #:include-file-paths include-file-paths #:missing-columns missing-columns #:cache cache]) → lazyframe? path : path-string? n-rows : (or/c size/c #f) = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 parallel : parquet-parallel/c = 'auto use-statistics : boolean? = #t glob : boolean? = #t rechunk : boolean? = #f low-memory : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f missing-columns : (or/c 'raise 'insert) = 'raise cache : boolean? = #t
> (~> (scan-parquet "flights.parquet") (filter (> (col "dep_delay") 20)) (select "carrier" "dep_delay") collect)
shape: (3, 2)
┌─────────┬───────────┐
│ carrier ┆ dep_delay │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═══════════╡
│ EV ┆ 24 │
│ UA ┆ 47 │
│ MQ ┆ 39 │
└─────────┴───────────┘
> (collect (scan-parquet (build-path parquet-dir "part-*.parquet") #:n-rows 3))
shape: (3, 3)
┌────────┬──────┬───────────┐
│ origin ┆ dest ┆ dep_delay │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞════════╪══════╪═══════════╡
│ EWR ┆ IAH ┆ 2 │
│ LGA ┆ IAH ┆ 4 │
│ JFK ┆ MIA ┆ 2 │
└────────┴──────┴───────────┘
> (collect (scan-parquet "produce.parquet" #:cache #f #:row-index-name "row"))
shape: (4, 4)
┌─────┬───────┬───────┬───────────────┐
│ row ┆ item ┆ grade ┆ price │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ cat ┆ enum ┆ decimal[10,2] │
╞═════╪═══════╪═══════╪═══════════════╡
│ 0 ┆ apple ┆ high ┆ 1.25 │
│ 1 ┆ pear ┆ low ┆ 0.80 │
│ 2 ┆ apple ┆ null ┆ null │
│ 3 ┆ fig ┆ mid ┆ 12.00 │
└─────┴───────┴───────┴───────────────┘
> (scan-parquet "flights.parquet" #:parallel 'row_groups) scan-parquet: contract violation
expected: parquet-parallel/c
given: 'row_groups
in: the parallel argument of
(->i
((path path-string?))
(#:n-rows
(n-rows (or/c size/c #f))
#:row-index-name
(row-index-name (or/c column-name/c #f))
#:row-index-offset
(row-index-offset row-index/c)
#:parallel
(parallel parquet-parallel/c)
#:use-statistics
(use-statistics boolean?)
#:glob
(glob boolean?)
#:rechunk
(rechunk boolean?)
#:low-memory
(low-memory boolean?)
#:include-file-paths
(include-file-paths
(or/c column-name/c #f))
#:missing-columns
(missing-columns (or/c 'raise 'insert))
#:cache
(cache boolean?))
(result lazyframe?))
contract from:
<pkgs>/polars/private/generic/parquet.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/parquet.rkt:12:11
procedure
(write-parquet frame path [ #:compression compression #:compression-level compression-level #:statistics statistics #:row-group-size row-group-size #:data-page-size data-page-size]) → void? frame : dataframe? path : path-string? compression : parquet-compression/c = 'zstd compression-level : (or/c exact-integer? #f) = #f statistics : parquet-statistics/c = #t row-group-size : (or/c size/c #f) = #f data-page-size : (or/c size/c #f) = #f
> (define out (build-path parquet-dir "produce.parquet")) > (write-parquet produce out #:compression 'gzip #:compression-level 9 #:statistics #f) > (read-parquet out)
shape: (4, 3)
┌───────┬───────┬───────────────┐
│ item ┆ grade ┆ price │
│ --- ┆ --- ┆ --- │
│ cat ┆ enum ┆ decimal[10,2] │
╞═══════╪═══════╪═══════════════╡
│ apple ┆ high ┆ 1.25 │
│ pear ┆ low ┆ 0.80 │
│ apple ┆ null ┆ null │
│ fig ┆ mid ┆ 12.00 │
└───────┴───────┴───────────────┘
> (write-parquet produce out #:compression 'lz4 #:row-group-size 2 #:statistics '(min max null-count distinct-count) #:data-page-size 4096) > (read-parquet out)
shape: (4, 3)
┌───────┬───────┬───────────────┐
│ item ┆ grade ┆ price │
│ --- ┆ --- ┆ --- │
│ cat ┆ enum ┆ decimal[10,2] │
╞═══════╪═══════╪═══════════════╡
│ apple ┆ high ┆ 1.25 │
│ pear ┆ low ┆ 0.80 │
│ apple ┆ null ┆ null │
│ fig ┆ mid ┆ 12.00 │
└───────┴───────┴───────────────┘
> (write-parquet produce out #:compression-level 30) write-parquet: contract violation
#:compression-level 30 is outside zstd's levels, 1 to 22
in: (->i
((frame dataframe?) (path path-string?))
(#:compression
(compression parquet-compression/c)
#:compression-level
(compression-level
(or/c exact-integer? #f))
#:statistics
(statistics parquet-statistics/c)
#:row-group-size
(row-group-size (or/c size/c #f))
#:data-page-size
(data-page-size (or/c size/c #f)))
#:pre/desc
(compression compression-level)
(level-problem
compression
compression-level)
(result void?))
contract from:
<pkgs>/polars/private/generic/parquet.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/parquet.rkt:13:11
> (write-parquet produce out #:statistics '(null-count)) write-parquet: contract violation
expected: parquet-statistics/c
given: '(null-count)
in: the statistics argument of
(->i
((frame dataframe?) (path path-string?))
(#:compression
(compression parquet-compression/c)
#:compression-level
(compression-level
(or/c exact-integer? #f))
#:statistics
(statistics parquet-statistics/c)
#:row-group-size
(row-group-size (or/c size/c #f))
#:data-page-size
(data-page-size (or/c size/c #f)))
#:pre/desc
(compression compression-level)
(level-problem
compression
compression-level)
(result void?))
contract from:
<pkgs>/polars/private/generic/parquet.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/parquet.rkt:13:11
> (write-parquet produce "/no/such/dir/out.parquet") write-parquet: failed to write parquet to
/no/such/dir/out.parquet: cannot create file: No such file
or directory (os error 2)
procedure
(read-json path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length]) → dataframe? path : path-string? schema : (or/c json-schema/c #f) = #f schema-overrides : json-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100
procedure
(write-json d path) → void?
d : dataframe? path : path-string?
Column types are inferred from the first infer-schema-length objects, or from all of them for #f; a key first seen later is an error. A later value of another JSON type converts where Polars can (a float truncates to an integer, a boolean reads as 0 or 1, a number out of the type’s range reads as null) and is an error otherwise. schema gives the columns in order: other keys are skipped and a column the file lacks reads as nulls. schema-overrides retypes columns of the file or of schema; naming any other column is an error. A json-schema/c is a list of (cons name dtype) pairs with distinct names, a name a string without a NUL (column-name/c) and a dtype any series #:dtype spelling but an Enum. A string reads as the date, datetime or time the schema asks for, and as null where it does not parse.
write-json writes temporal values as strings, Categorical and Enum values as their names, and NaN and the infinities as null, so reading them back needs a schema; a binary column is an error. It replaces a file as write-csv does.
json provides a read-json and write-json too, so (require polars json) fails with “identifier already required”. Rename or leave out json’s pair:
(require polars (prefix-in js: json)) (require polars (except-in json read-json write-json))
> (define df (dataframe (list (series '(1 2 3) #:name "foo") (series (list polars-null "bak" "baz") #:name "bar")))) > (write-json df path) > (file->string path) "[{\"foo\":1,\"bar\":null},{\"foo\":2,\"bar\":\"bak\"},{\"foo\":3,\"bar\":\"baz\"}]"
> (read-json path)
shape: (3, 2)
┌─────┬──────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪══════╡
│ 1 ┆ null │
│ 2 ┆ bak │
│ 3 ┆ baz │
└─────┴──────┘
> (read-json "stations.json")
shape: (4, 5)
┌─────────┬────────────┬─────────┬──────┬──────┐
│ station ┆ day ┆ reading ┆ flag ┆ note │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 ┆ str ┆ str │
╞═════════╪════════════╪═════════╪══════╪══════╡
│ north ┆ 2024-01-01 ┆ 3.0 ┆ ok ┆ null │
│ south ┆ 2024-01-01 ┆ 4.5 ┆ low ┆ null │
│ north ┆ 2024-01-02 ┆ 2.0 ┆ ok ┆ null │
│ south ┆ 2024-01-02 ┆ 5.25 ┆ high ┆ late │
└─────────┴────────────┴─────────┴──────┴──────┘
> (read-json "stations.json" #:schema '(("day" . date) ("station" . categorical) ("reading" . f32)))
shape: (4, 3)
┌────────────┬─────────┬─────────┐
│ day ┆ station ┆ reading │
│ --- ┆ --- ┆ --- │
│ date ┆ cat ┆ f32 │
╞════════════╪═════════╪═════════╡
│ 2024-01-01 ┆ north ┆ 3.0 │
│ 2024-01-01 ┆ south ┆ 4.5 │
│ 2024-01-02 ┆ north ┆ 2.0 │
│ 2024-01-02 ┆ south ┆ 5.25 │
└────────────┴─────────┴─────────┘
> (~> (read-json "stations.json" #:schema-overrides '(("day" . date))) (select "day" "reading"))
shape: (4, 2)
┌────────────┬─────────┐
│ day ┆ reading │
│ --- ┆ --- │
│ date ┆ f64 │
╞════════════╪═════════╡
│ 2024-01-01 ┆ 3.0 │
│ 2024-01-01 ┆ 4.5 │
│ 2024-01-02 ┆ 2.0 │
│ 2024-01-02 ┆ 5.25 │
└────────────┴─────────┘
> (~> (read-json "stations.json" #:schema '(("day" . datetime))) (head 1))
shape: (1, 1)
┌──────────────┐
│ day │
│ --- │
│ datetime[μs] │
╞══════════════╡
│ null │
└──────────────┘
> (~> (read-json "stations.json" #:schema-overrides '(("reading" . i64))) (select "station" "reading"))
shape: (4, 2)
┌─────────┬─────────┐
│ station ┆ reading │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═════════╡
│ north ┆ 3 │
│ south ┆ 4 │
│ north ┆ 2 │
│ south ┆ 5 │
└─────────┴─────────┘
> (read-json "stations.json" #:infer-schema-length 1) read-json: failed to read json from stations.json: extra
field in struct data: note, consider increasing
#:infer-schema-length, or passing the full #:schema to
ignore extra fields
> (read-json "stations.json" #:schema-overrides '(("dya" . date))) read-json: failed to read json from stations.json: schema
overrides name a column not in the file: "dya"
procedure
(read-ndjson path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length #:batch-size batch-size #:n-rows n-rows #:low-memory low-memory #:rechunk rechunk #:row-index-name row-index-name #:row-index-offset row-index-offset #:ignore-errors ignore-errors #:include-file-paths include-file-paths]) → dataframe? path : path-string? schema : (or/c ndjson-schema/c #f) = #f schema-overrides : ndjson-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100 batch-size : (or/c positive-size/c #f) = 1024 n-rows : (or/c size/c #f) = #f low-memory : boolean? = #f rechunk : boolean? = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 ignore-errors : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f
procedure
(scan-ndjson path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length #:batch-size batch-size #:n-rows n-rows #:low-memory low-memory #:rechunk rechunk #:row-index-name row-index-name #:row-index-offset row-index-offset #:ignore-errors ignore-errors #:include-file-paths include-file-paths]) → lazyframe? path : path-string? schema : (or/c ndjson-schema/c #f) = #f schema-overrides : ndjson-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100 batch-size : (or/c positive-size/c #f) = 1024 n-rows : (or/c size/c #f) = #f low-memory : boolean? = #f rechunk : boolean? = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 ignore-errors : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f
Column types are inferred from the first infer-schema-length lines, or from all of them for #f. A key first seen later is left out, and a value that does not parse as its column’s type is an error, or null with #:ignore-errors #t. schema and schema-overrides work as for read-json, an override applying even with a schema; scan-ndjson reports a bad override at once. An ndjson-schema/c is a json-schema/c whose dtypes are a boolean, a 32- or 64-bit integer, a float, a string, a date, a datetime, a categorical or 'null; a 'null column holds no value, and a value in the file for it is an error, as in Python.
n-rows caps the rows read. row-index-name adds a first 'uint32 column of row numbers from row-index-offset; include-file-paths adds a last column holding each row’s file; the two name different columns. rechunk gathers the result into one chunk. Polars ignores batch-size and low-memory. A gzip, zlib or zstd file is decompressed as it is read, recognised by its first bytes rather than its name.
> (read-ndjson "stations.ndjson")
shape: (4, 5)
┌─────────┬────────────┬─────────┬──────┬──────┐
│ station ┆ day ┆ reading ┆ flag ┆ note │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 ┆ str ┆ str │
╞═════════╪════════════╪═════════╪══════╪══════╡
│ north ┆ 2024-01-01 ┆ 3.0 ┆ ok ┆ null │
│ south ┆ 2024-01-01 ┆ 4.5 ┆ low ┆ null │
│ north ┆ 2024-01-02 ┆ 2.0 ┆ ok ┆ null │
│ south ┆ 2024-01-02 ┆ 5.25 ┆ high ┆ late │
└─────────┴────────────┴─────────┴──────┴──────┘
> (~> (scan-ndjson "stations.ndjson") (filter (> (col "reading") 3)) (select "station" "reading") collect)
shape: (2, 2)
┌─────────┬─────────┐
│ station ┆ reading │
│ --- ┆ --- │
│ str ┆ f64 │
╞═════════╪═════════╡
│ south ┆ 4.5 │
│ south ┆ 5.25 │
└─────────┴─────────┘
> (read-ndjson (build-path nd-dir "part-*.ndjson") #:n-rows 4 #:row-index-name "row")
shape: (4, 4)
┌─────┬────────┬──────┬───────────┐
│ row ┆ origin ┆ dest ┆ dep_delay │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ str ┆ str ┆ i64 │
╞═════╪════════╪══════╪═══════════╡
│ 0 ┆ EWR ┆ IAH ┆ 2 │
│ 1 ┆ LGA ┆ IAH ┆ 4 │
│ 2 ┆ JFK ┆ MIA ┆ 2 │
│ 3 ┆ JFK ┆ BQN ┆ -1 │
└─────┴────────┴──────┴───────────┘
> (column-names (read-ndjson "stations.ndjson" #:include-file-paths "file")) '("station" "day" "reading" "flag" "note" "file")
> (read-ndjson "stations.ndjson" #:schema '(("station" . str) ("extra" . null)))
shape: (4, 2)
┌─────────┬───────┐
│ station ┆ extra │
│ --- ┆ --- │
│ str ┆ null │
╞═════════╪═══════╡
│ north ┆ null │
│ south ┆ null │
│ north ┆ null │
│ south ┆ null │
└─────────┴───────┘
> (read-ndjson "stations.ndjson" #:schema '(("station" . str) ("note" . null))) read-ndjson: failed to read ndjson from stations.ndjson: got
non-null value for NULL-typed column: late
> (read-ndjson "stations.ndjson" #:infer-schema-length 1) read-ndjson: failed to read ndjson from stations.ndjson:
cannot parse '4.5' (f64) as Int64
> (~> (read-ndjson "stations.ndjson" #:infer-schema-length 1 #:ignore-errors #t) (select "station" "reading"))
shape: (4, 2)
┌─────────┬─────────┐
│ station ┆ reading │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════════╪═════════╡
│ north ┆ 3 │
│ south ┆ null │
│ north ┆ 2 │
│ south ┆ null │
└─────────┴─────────┘
> (scan-ndjson "stations.ndjson" #:schema-overrides '(("dya" . date))) scan-ndjson: failed to scan stations.ndjson: schema
overrides name a column not in the file: "dya"
> (read-ndjson (build-path nd-dir "*.json")) read-ndjson: failed to read ndjson from
/var/tmp/polars-doc-17925471161792547116202/*.json: no files
match the pattern
procedure
(write-ndjson d path [ #:compression compression #:compression-level compression-level #:check-extension check-extension]) → void? d : dataframe? path : path-string? compression : (or/c 'uncompressed 'gzip 'zstd) = 'uncompressed compression-level : (or/c exact-nonnegative-integer? #f) = #f check-extension : boolean? = #t
> (define small (dataframe (list (series '(1 2) #:name "a")))) > (write-ndjson small (build-path nd-dir "small.ndjson")) > (file->string (build-path nd-dir "small.ndjson")) "{\"a\":1}\n{\"a\":2}\n"
> (write-ndjson small (build-path nd-dir "small.ndjson.gz") #:compression 'gzip) > (read-ndjson (build-path nd-dir "small.ndjson.gz"))
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
└─────┘
> (write-ndjson small (build-path nd-dir "small.ndjson.zst")) write-ndjson: failed to write ndjson to /var/tmp/polars-doc-
17925471161792547116232/small.ndjson.zst: pass
#:check-extension #f to give an uncompressed file a
compressed file's suffix
> (write-ndjson small (build-path nd-dir "small.ndjson.zst") #:compression 'zstd #:compression-level 30) write-ndjson: contract violation
#:compression-level needs #:compression 'gzip (a level
from 0 to 9) or 'zstd (1 to 22)
compression: 'zstd
compression-level: 30
in: (->i
((frame dataframe?) (path path-string?))
(#:compression
(compression
(or/c 'uncompressed 'gzip 'zstd))
#:compression-level
(compression-level (or/c natural? #f))
#:check-extension
(check-extension boolean?))
#:pre/name
(compression compression-level)
"#:compression-level needs #:compression 'gzip (a
level from 0 to 9) or 'zstd (1 to 22)"
(level-fits? compression compression-level)
(result void?))
contract from:
<pkgs>/polars/private/generic/json.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/json.rkt:15:11
procedure
(lazy d) → lazyframe?
d : dataframe?
procedure
(collect lf) → dataframe?
lf : lazyframe?
> (define four (dataframe (list (series '(1 2 3 4) #:name "v")))) > (~> four lazy (filter (> (col "v") 2)) collect)
shape: (2, 1)
┌─────┐
│ v │
│ --- │
│ i64 │
╞═════╡
│ 3 │
│ 4 │
└─────┘
> (~> four lazy (filter (> (col "nope") 2)) collect) collect: not found: unable to find column "nope"; valid
columns: ["v"]
procedure
(explain lf [ #:optimized optimized #:format format]) → string? lf : lazyframe? optimized : boolean? = #t format : (or/c 'plain 'tree) = 'plain
> (define late (~> (scan-parquet "flights.parquet") (filter (> (col "dep_delay") 20)) (select "carrier" "dep_delay"))) > (displayln (explain late #:optimized #f))
SELECT [col("carrier"), col("dep_delay")]
FILTER col("dep_delay") > 20
FROM
Parquet SCAN [/home/root/user/.local/share/racket/9.3/pkgs/polars/scribblings/data/flights.parquet]
PROJECT */19 COLUMNS
ESTIMATED ROWS: 102
> (displayln (explain late))
Parquet SCAN [/home/root/user/.local/share/racket/9.3/pkgs/polars/scribblings/data/flights.parquet]
PROJECT 2/19 COLUMNS
SELECTION: col("dep_delay") > 20
ESTIMATED ROWS: 102
> (displayln (explain (select (lazy produce) "item") #:format 'tree))
0
┌────────────────────────────────────
│
│ ╭────────────────────────────────╮
│ │ DF ["item", "grade", "price"] │
0 │ │ PROJECT: ["item"]; 1/3 COLUMNS │
│ ╰────────────────────────────────╯
> (explain (select (lazy produce) "colour")) explain: failed to explain the query: not found: unable to
find column "colour"; valid columns: ["item", "grade",
"price"]
Resolved plan until failure:
---> FAILED HERE RESOLVING 'select' <---
DF ["item", "grade", "price"]; PROJECT */3 COLUMNS: 'select'
> (explain late #:format 'dot) explain: contract violation
expected: explain-format/c
given: 'dot
in: the #:format argument of
(->*
(lazyframe?)
(#:format
explain-format/c
#:optimized
boolean?)
string?)
contract from:
<pkgs>/polars/private/generic/reshape.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/reshape.rkt:45:11
procedure
d : (or/c dataframe? lazyframe?) key : (or/c string? Expr-ptr?)
procedure
(agg g agg-expr ...) → (or/c dataframe? lazyframe?)
g : grouped? agg-expr : Expr-ptr?
procedure
v : any/c
> (~> (dataframe (list (series '("x" "y" "x") #:name "k") (series '(1 2 3) #:name "v"))) (group-by "k") (agg (alias (sum "v") "total")))
shape: (2, 2)
┌─────┬───────┐
│ k ┆ total │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═══════╡
│ y ┆ 2 │
│ x ┆ 4 │
└─────┴───────┘
> (define kv (dataframe (list (series '("x" "y" "x") #:name "k") (series '(1 2 3) #:name "v")))) > (~> kv (group-by "k") (agg (alias (sum "v") "total")))
shape: (2, 2)
┌─────┬───────┐
│ k ┆ total │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═══════╡
│ x ┆ 4 │
│ y ┆ 2 │
└─────┴───────┘
> (~> kv (with-columns (~> (col "v") sum (over "k") (alias "total"))))
shape: (3, 3)
┌─────┬─────┬───────┐
│ k ┆ v ┆ total │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╡
│ x ┆ 1 ┆ 4 │
│ y ┆ 2 ┆ 2 │
│ x ┆ 3 ┆ 4 │
└─────┴─────┴───────┘
> (define khv (dataframe (list (series '("a" "a" "a" "b") #:name "k") (series '("x" "y" "x" "x") #:name "h") (series '(1 2 3 4) #:name "v"))))
> (~> khv (with-columns (~> (col "v") sum (over "k" "h") (alias "total")) (~> (col "v") mean (over (col "k")) (alias "mean")) (~> (col "v") (rank #:descending #t) (over "k") (alias "rank")) (~> (col "v") max (over (> (col "v") 1)) (alias "band_max"))))
shape: (4, 7)
┌─────┬─────┬─────┬───────┬──────┬──────┬──────────┐
│ k ┆ h ┆ v ┆ total ┆ mean ┆ rank ┆ band_max │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 ┆ i64 ┆ f64 ┆ f64 ┆ i64 │
╞═════╪═════╪═════╪═══════╪══════╪══════╪══════════╡
│ a ┆ x ┆ 1 ┆ 4 ┆ 2.0 ┆ 3.0 ┆ 1 │
│ a ┆ y ┆ 2 ┆ 2 ┆ 2.0 ┆ 2.0 ┆ 4 │
│ a ┆ x ┆ 3 ┆ 4 ┆ 2.0 ┆ 1.0 ┆ 4 │
│ b ┆ x ┆ 4 ┆ 4 ┆ 4.0 ┆ 1.0 ┆ 4 │
└─────┴─────┴─────┴───────┴──────┴──────┴──────────┘
> (over (col "v")) over: contract violation
received: 1 argument
expected: at least 2 non-keyword arguments
in: (->*
(col-expr/c col-expr/c)
#:rest
(listof col-expr/c)
Expr-ptr?)
contract from:
<pkgs>/polars/private/generic/window.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/window.rkt:8:11
procedure
(rank x [ #:method method #:descending descending #:seed seed]) → Expr-ptr? x : (or/c Expr-ptr? string?) method : (or/c 'average 'min 'max 'dense 'ordinal) = 'average descending : boolean? = #f seed : (or/c (integer-in 0 (sub1 (expt 2 64))) #f) = #f
procedure
x : (or/c Expr-ptr? string?) indices : (or/c Expr-ptr? series? (listof exact-integer?))
> (define scores (dataframe (list (series '("a" "b" "c" "d") #:name "name") (series '(30 10 30 20) #:name "score"))))
> (~> scores (with-columns (~> (col "score") (rank #:method 'dense #:descending #t) (alias "dense")) (~> (col "score") (rank #:method 'ordinal) (alias "ordinal"))))
shape: (4, 4)
┌──────┬───────┬───────┬─────────┐
│ name ┆ score ┆ dense ┆ ordinal │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ u32 ┆ u32 │
╞══════╪═══════╪═══════╪═════════╡
│ a ┆ 30 ┆ 1 ┆ 3 │
│ b ┆ 10 ┆ 3 ┆ 1 │
│ c ┆ 30 ┆ 1 ┆ 4 │
│ d ┆ 20 ┆ 2 ┆ 2 │
└──────┴───────┴───────┴─────────┘
> (~> scores (select (gather "name" '(3 0))))
shape: (2, 1)
┌──────┐
│ name │
│ --- │
│ str │
╞══════╡
│ d │
│ a │
└──────┘
> (rank "score" #:method 'first) rank: contract violation
expected: (or/c (quote average) (quote min) (quote max)
(quote dense) (quote ordinal))
given: 'first
in: the #:method argument of
(->*
(col-expr/c)
(#:descending
boolean?
#:method
(or/c 'average 'min 'max 'dense 'ordinal)
#:seed
(or/c
#f
(integer-in 0 18446744073709551615)))
Expr-ptr?)
contract from:
<pkgs>/polars/private/generic/ordering.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/ordering.rkt:19:24
A series reduces as in Python Polars: to polars-null where the reduction is undefined, and to zero for the sum of no values. A string, temporal or categorical series has no sum, but a duration does, and a small integer one sums as 'int64, so its total does not wrap.
> (define s (series '(1 2 3 4) #:name "v")) > (list (sum s) (mean s) (min s) (max s)) '(10 2.5 1 4)
> (max 1 2 3) 3
> (define nothing (series (list polars-null polars-null) #:name "n")) > (list (sum nothing) (mean nothing) (min nothing) (max nothing)) '(polars-null polars-null polars-null polars-null)
> (define no-rows (series '() #:dtype 'int64)) > (list (sum no-rows) (mean no-rows) (min no-rows) (max no-rows)) '(0 polars-null polars-null polars-null)
> (sum (series '(#t #f #t))) 2
> (sum (series '(100 100 100) #:dtype 'int8)) 300
> (mean (series '("a" "b"))) polars-null
> (mean (series (list (date 2020 1 2) (date 2020 1 1)))) #<datetime 2020-01-01T12:00:00>
> (sum (series '("a" "b"))) sum: unsupported dtype for reduction: 'string
> (~> (dataframe (list nothing)) (select (sum "n")))
shape: (1, 1)
┌──────┐
│ n │
│ --- │
│ null │
╞══════╡
│ null │
└──────┘
procedure
x : any/c
procedure
x : (or/c series? string? Expr-ptr?)
procedure
x : any/c
procedure
x : (or/c series? string? Expr-ptr?) ddof : (integer-in 0 255) = 1
procedure
x : (or/c series? string? Expr-ptr?) ddof : (integer-in 0 255) = 1
procedure
e : (or/c Expr-ptr? string?) name : string?
> (list (std nothing) (var nothing) (n-unique nothing)) '(polars-null polars-null 1)
> (list (std (series '(5))) (std (series '(#t #f))) (n-unique (series (list 1 polars-null)))) '(polars-null 0.7071067811865476 2)
> (var (series '("a" "b"))) var: unsupported dtype for reduction: 'string
> (~> (dataframe (list nothing)) (select (count "n")))
shape: (1, 1)
┌─────┐
│ n │
│ --- │
│ u32 │
╞═════╡
│ 0 │
└─────┘
Name clash. racket/list also exports first and last (along with count and group-by, which polars exports too). Requiring both modules explicitly — (require racket/list polars) — is an error (identifier already required). A plain #lang racket/base program is unaffected, because racket/base does not export these names. See Shadowed bindings for how to take control.
procedure
x : (or/c Expr-ptr? string?) exponent : (or/c Expr-ptr? real?)
procedure
x : (or/c Expr-ptr? string? number?) decimals : exact-nonnegative-integer? = 0
> (~> (dataframe (list (series '(-2.5 -1.5 0.5 1.5 2.5) #:name "x"))) (select (round "x")))
shape: (5, 1)
┌──────┐
│ x │
│ --- │
│ f64 │
╞══════╡
│ -2.0 │
│ -2.0 │
│ 0.0 │
│ 2.0 │
│ 2.0 │
└──────┘
> (round 2.5) 2.0
> (~> (dataframe (list (series '(-2 0 3) #:name "i") (series '(-2.5 0.0 3.5) #:name "f"))) (select (sign "i") (sign "f")))
shape: (3, 2)
┌─────┬──────┐
│ i ┆ f │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ -1 ┆ -1.0 │
│ 0 ┆ 0.0 │
│ 1 ┆ 1.0 │
└─────┴──────┘
procedure
(is-between x lower upper [#:closed closed]) → Expr-ptr?
x : (or/c Expr-ptr? string?) lower : any/c upper : any/c closed : (or/c 'both 'left 'right 'none) = 'both
procedure
x : (or/c Expr-ptr? string?) rhs : (or/c list? series? Expr-ptr?)
> (define gappy (dataframe (list (series (list 1 polars-null 3) #:name "x")))) > (~> gappy (select (is-in "x" (list 1 polars-null))))
shape: (3, 1)
┌───────┐
│ x │
│ --- │
│ bool │
╞═══════╡
│ true │
│ null │
│ false │
└───────┘
> (~> gappy (select (is-in "x" (list polars-null))))
shape: (3, 1)
┌───────┐
│ x │
│ --- │
│ bool │
╞═══════╡
│ false │
│ null │
│ false │
└───────┘
> (~> gappy (select (is-in "x" '())))
shape: (3, 1)
┌───────┐
│ x │
│ --- │
│ bool │
╞═══════╡
│ false │
│ null │
│ false │
└───────┘
> (~> gappy (select (is-between "x" 2 5)))
shape: (3, 1)
┌───────┐
│ x │
│ --- │
│ bool │
╞═══════╡
│ false │
│ null │
│ true │
└───────┘
procedure
x : (or/c Expr-ptr? string?)
procedure
x : (or/c Expr-ptr? string?)
procedure
x : (or/c Expr-ptr? string?)
procedure
x : (or/c Expr-ptr? string?)
procedure
x : (or/c Expr-ptr? string?)
procedure
x : (or/c Expr-ptr? string?)
procedure
(str-extract x pattern [ #:group-index group-index]) → Expr-ptr? x : (or/c Expr-ptr? string?) pattern : string? group-index : exact-nonnegative-integer? = 1
procedure
(str->date x [ #:format format #:strict strict #:exact exact #:cache cache]) → Expr-ptr? x : (or/c Expr-ptr? string?) format : (or/c string? #f) = #f strict : boolean? = #t exact : boolean? = #t cache : boolean? = #t
procedure
(str->datetime x [ #:format format #:unit unit #:time-zone time-zone #:ambiguous ambiguous #:strict strict #:exact exact #:cache cache]) → Expr-ptr? x : (or/c Expr-ptr? string?) format : (or/c string? #f) = #f
unit : (or/c 'milliseconds 'microseconds 'nanoseconds) = 'microseconds time-zone : (or/c non-empty-string? #f) = #f ambiguous : (or/c 'raise 'earliest 'latest 'null) = 'raise strict : boolean? = #t exact : boolean? = #t cache : boolean? = #t
> (define stamps (dataframe (list (series '("2021-03-27T00:00:00+0100" "2021-03-29T00:00:00+0200") #:name "at") (series '("2021-03-27 03:00" "2021-10-31 02:30") #:name "clock")))) > (select stamps (str->datetime "at" #:format "%Y-%m-%dT%H:%M:%S%z"))
shape: (2, 1)
┌─────────────────────────┐
│ at │
│ --- │
│ datetime[μs, UTC] │
╞═════════════════════════╡
│ 2021-03-26 23:00:00 UTC │
│ 2021-03-28 22:00:00 UTC │
└─────────────────────────┘
> (select stamps (str->datetime "at" #:time-zone "Asia/Tokyo") (str->datetime "clock" #:time-zone "Europe/Brussels" #:ambiguous 'latest))
shape: (2, 2)
┌──────────────────────────┬───────────────────────────────┐
│ at ┆ clock │
│ --- ┆ --- │
│ datetime[μs, Asia/Tokyo] ┆ datetime[μs, Europe/Brussels] │
╞══════════════════════════╪═══════════════════════════════╡
│ 2021-03-27 08:00:00 JST ┆ 2021-03-27 03:00:00 CET │
│ 2021-03-29 07:00:00 JST ┆ 2021-10-31 02:30:00 CET │
└──────────────────────────┴───────────────────────────────┘
> (select (dataframe (list (series '("2021-03-28 01:30" "2021-03-28 02:30") #:name "clock"))) (~> (str->datetime "clock") (dt-replace-time-zone "Europe/Brussels" #:non-existent 'null)))
shape: (2, 1)
┌───────────────────────────────┐
│ clock │
│ --- │
│ datetime[μs, Europe/Brussels] │
╞═══════════════════════════════╡
│ 2021-03-28 01:30:00 CET │
│ null │
└───────────────────────────────┘
> (str->datetime "at" #:time-zone "Asia/Tokio") str->datetime: cannot parse into the time zone "Asia/Tokio":
unable to parse time zone: 'Asia/Tokio'. Please check the
Time Zone Database for a list of available time zones.
Hint: did you mean 'Asia/Tokyo' instead?
procedure
(dt-convert-time-zone x zone) → (or/c series? Expr-ptr?)
x : (or/c series? Expr-ptr? string?) zone : non-empty-string?
procedure
(dt-replace-time-zone x zone [ #:ambiguous ambiguous #:non-existent non-existent]) → (or/c series? Expr-ptr?) x : (or/c series? Expr-ptr? string?) zone : (or/c non-empty-string? #f) ambiguous : (or/c 'raise 'earliest 'latest 'null) = 'raise non-existent : (or/c 'raise 'null) = 'raise
A wall-clock time can name two instants, when the clocks go back, or none, when they go forward. #:ambiguous picks the 'earliest or 'latest of two, or 'null; #:non-existent 'null gives null for a time that does not exist. Both default to 'raise, as Python’s do. Zones are named as the tz database names them; a zone polars does not know raises, with the nearest name it does. On an expression, an ambiguous or non-existent time is reported when the plan runs. API gap: Python’s ambiguous may also be an expression, resolving each row its own way; here it is one symbol for the whole column (#171).
> (define clocks (dataframe (list (series (list (datetime 2021 3 27 3) (datetime 2021 10 31 2 30)) #:name "clock"))))
> (select clocks (~> "clock" (dt-replace-time-zone "UTC") (alias "utc")) (~> "clock" (dt-replace-time-zone "UTC") (dt-convert-time-zone "Asia/Kathmandu") (alias "kathmandu")) (~> "clock" (dt-replace-time-zone "Europe/Brussels" #:ambiguous 'earliest) (alias "brussels")))
shape: (2, 3)
┌─────────────────────────┬──────────────────────────────┬───────────────────────────────┐
│ utc ┆ kathmandu ┆ brussels │
│ --- ┆ --- ┆ --- │
│ datetime[μs, UTC] ┆ datetime[μs, Asia/Kathmandu] ┆ datetime[μs, Europe/Brussels] │
╞═════════════════════════╪══════════════════════════════╪═══════════════════════════════╡
│ 2021-03-27 03:00:00 UTC ┆ 2021-03-27 08:45:00 +0545 ┆ 2021-03-27 03:00:00 CET │
│ 2021-10-31 02:30:00 UTC ┆ 2021-10-31 08:15:00 +0545 ┆ 2021-10-31 02:30:00 CEST │
└─────────────────────────┴──────────────────────────────┴───────────────────────────────┘
> (dt-replace-time-zone (ref clocks "clock") "Europe/Brussels" #:ambiguous 'latest)
shape: (2,)
Series: 'clock' [datetime[us, Europe/Brussels]]
[
2021-03-27 03:00:00 CET
2021-10-31 02:30:00 CET
]
> (dt-replace-time-zone (ref clocks "clock") "Europe/Brussels") dt-replace-time-zone: cannot replace the time zone with
"Europe/Brussels": datetime '2021-10-31 02:30:00' is
ambiguous in time zone 'Europe/Brussels'. Please use
#:ambiguous to tell how it should be localized.
> (define skipped (series (list (datetime 2021 3 28 1 30) (datetime 2021 3 28 2 30)) #:name "clock")) > (dt-replace-time-zone skipped "Europe/Brussels" #:non-existent 'null)
shape: (2,)
Series: 'clock' [datetime[us, Europe/Brussels]]
[
2021-03-28 01:30:00 CET
null
]
> (dt-replace-time-zone skipped "Europe/Brussels") dt-replace-time-zone: cannot replace the time zone with
"Europe/Brussels": datetime '2021-03-28 02:30:00' is
non-existent in time zone 'Europe/Brussels'. You may be able
to use dt-replace-time-zone's #:non-existent 'null to return
`null` in this case.
> (dt-convert-time-zone "clock" "Asia/Kathmando") dt-convert-time-zone: cannot convert to the time zone
"Asia/Kathmando": unable to parse time zone:
'Asia/Kathmando'. Please check the Time Zone Database for a
list of available time zones.
Hint: did you mean 'Asia/Kathmandu' instead?
2.1.1 Shadowed bindings
(require polars) re-exports a handful of generic operations whose names also live in racket/base (min, max, sort, filter, >, <, >=, <=, =) and in racket/list (first, last, count, group-by). Under #lang racket/base this is seamless — these names are either not bound (so polars simply provides them) or bound only by the module language (which an explicit require silently shadows), and the polars versions intentionally fall back to the numeric/list behaviour for non-frame arguments.
A conflict arises only when another module providing the same name is also required explicitly — most commonly racket/list. Resolve it with the usual require sub-forms:
; keep polars' first/last/count/group-by, drop racket/list's: (require (except-in racket/list first last count group-by) polars) ; keep racket/list's, reach polars' under a prefix: (require racket/list (prefix-in pl: polars)) ; then (pl:first (col "v")) for the Expr, (first '(1 2 3)) for the list ; keep polars', reach racket/list's under a prefix: (require polars (prefix-in list: racket/list))
2.2 Series
A series wraps a typed column and prints in the REPL the way Polars prints it; series? is its predicate. (The underlying foreign pointer is an implementation detail and not part of the public series API.)
> (series '(1 2 3) #:name "ints")
shape: (3,)
Series: 'ints' [i64]
[
1
2
3
]
> (series '())
shape: (0,)
Series: '' [null]
[
]
> (series (list polars-null polars-null))
shape: (2,)
Series: '' [null]
[
null
null
]
> (series '() #:name "empty" #:dtype 'f64)
shape: (0,)
Series: 'empty' [f64]
[
]
> (series '(1 2) #:dtype 'null) series: a 'null series holds only polars-null
elements: '(1 2)
> (series '(1.5 2.5) #:name "floats" #:dtype 'f32)
shape: (2,)
Series: 'floats' [f32]
[
1.5
2.5
]
> (series (list 1 polars-null 3) #:name "with-null")
shape: (3,)
Series: 'with-null' [i64]
[
1
null
3
]
> (series '(IAH ATL IAH) #:name "dest")
shape: (3,)
Series: 'dest' [cat]
[
"IAH"
"ATL"
"IAH"
]
> (series (list (date 1997 1 10) (date 1985 2 15)) #:name "birthdate")
shape: (2,)
Series: 'birthdate' [date]
[
1997-01-10
1985-02-15
]
> (define-enum severity debug info error) > (series '("debug" "error") #:dtype severity)
shape: (2,)
Series: '' [enum]
[
"debug"
"error"
]
> (series '(debug fatal) #:dtype severity) series: 1 of 2 values is not in '(enum debug info error):
fatal
procedure
(series->string s) → string?
s : series?
value
procedure
(polars-null? v) → boolean?
v : any/c
procedure
s : has-dtype?
procedure
x : sized?
procedure
s : has-null-count?
procedure
(series-name s) → string?
s : series?
> (series-name (series '(1 2) #:name "ints")) "ints"
procedure
s : series? new-name : string?
procedure
(rename d old new) → dataframe?
d : dataframe? old : string? new : string?
procedure
s : series? new-name : string?
procedure
s : series?
procedure
(series-clone s) → series?
s : series?
> (series-name (rename (series '(1) #:name "a") "b")) "b"
> (define named (dataframe (list (series '(1) #:name "a") (series '(2) #:name "b")))) > (rename named "a" "z")
shape: (1, 2)
┌─────┬─────┐
│ z ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 2 │
└─────┴─────┘
> (rename named "a" "b") rename: column 'b' is duplicate
2.2.1 Converting to Racket values
These copy a column out of Polars in one foreign call rather than one per element: Racket allocates a buffer of the column’s native type, Rust copies the values into it, and Racket builds its values from the buffer and frees it. Nothing crosses the boundary to be freed later. At its peak a conversion holds that buffer (one native value per row, plus a byte per row when the column has nulls) beside the result it builds; in-series instead converts 4096 rows at a time. Each element comes out as ref returns it:
dtype |
| element |
integer dtypes |
| |
'float32, 'float64 |
| |
'boolean |
| |
'string |
| |
'categorical, '(enum cat ...) |
| |
'(decimal precision scale) |
| an exact rational, as exact? |
'date |
| a gregor date |
'(datetime unit tz) |
| a gregor datetime, to the column’s unit |
'(duration unit) |
| a gregor period in that unit |
'time |
| a gregor time |
'null |
| the null value |
A null entry becomes the #:null value. A series of any other dtype raises exn:fail:contract naming the dtype, even when every entry is null. The Interoperability chapter of the guide walks through all of them.
> (series->list (series (list 1.5 polars-null))) '(1.5 polars-null)
> (series->list (series (list "a" polars-null ""))) '("a" polars-null "")
> (series->list (series (list #t #f polars-null))) '(#t #f polars-null)
> (series->list (series (list (datetime 2024 1 2 3 4 5 678000000) polars-null))) '(#<datetime 2024-01-02T03:04:05.678> polars-null)
> (series->list (cast (series '(19724) #:dtype 'i32) 'date)) '(#<date 2024-01-02>)
> (series->list (cast (series '(11045000000000) #:dtype 'i64) 'time)) '(#<time 03:04:05>)
> (series->list (cast (series '(1500) #:dtype 'i64) '(duration milliseconds))) '(#<period of 1500 milliseconds>)
> (series->list (cast (series (list "UA" polars-null "UA")) 'categorical)) '(UA polars-null UA)
> (series->list (ref (read-parquet "produce.parquet") "price")) '(5/4 4/5 polars-null 12)
> (series->list (cast (series '("a") #:name "b") 'binary)) series->list: unsupported dtype
series: "b"
dtype: 'binary
procedure
(series->list s [#:null null-value]) → list?
s : series? null-value : any/c = polars-null
procedure
(series->vector s [#:null null-value]) → vector?
s : series? null-value : any/c = polars-null
> (define s (series (list 3 polars-null 1) #:name "x")) > (series->list s) '(3 polars-null 1)
> (series->list s #:null 'missing) '(3 missing 1)
> (series->vector s) '#(3 polars-null 1)
> (series->vector s #:null 0) '#(3 0 1)
procedure
(series->f64vector s [#:null null-value]) → f64vector?
s : series? null-value : (or/c real? 'error) = +nan.0
The result is ordinary garbage-collected memory, which Racket CS may move: pass it to a foreign call that is not #:blocking?, and do not let foreign code keep the pointer past the call.
> (define xs (series (list 1 polars-null 3) #:name "x")) > (f64vector->list (series->f64vector xs)) '(1.0 +nan.0 3.0)
> (f64vector->list (series->f64vector xs #:null 0)) '(1.0 0.0 3.0)
> (f64vector->list (series->f64vector (series (list #t #f)))) '(1.0 0.0)
> (f64vector->list (series->f64vector (series '(1 2)) #:null 'error)) '(1.0 2.0)
> (series->f64vector xs #:null 'error) series->f64vector: null value
series: "x"
row: 1
> (series->f64vector (series '("a") #:name "s")) series->f64vector: not a numeric series
series: "s"
dtype: 'string
procedure
s : series? null-value : any/c = polars-null
> (for/list ([x (in-series xs)]) x) '(1 polars-null 3)
> (for/sum ([x (in-series xs #:null 0)]) x) 4
> (for/list ([x (series '("a" "b"))]) (string-upcase x)) '("A" "B")
> (for/first ([x (in-series (series (build-list 100000 values)))] #:when (> x 41)) x) 42
2.2.2 Dates, times, datetimes and durations
The temporal dtypes cross the boundary as gregor values, both ways: a 'date column holds gregor dates, a 'time column gregor times, a '(datetime unit #f) column gregor datetimes, a zoned '(datetime unit "Zone/Name") column gregor moments in its zone and a '(duration unit) column gregor periods. series infers the dtype from the values, as Python’s pl.Series does from date, time, datetime and timedelta, and lit makes the literal of the same dtype:
gregor value |
| inferred dtype |
date |
| 'date |
time |
| 'time |
datetime |
| '(datetime microseconds #f), or '(datetime nanoseconds #f) |
moment |
| '(datetime microseconds "Zone/Name"), or nanoseconds |
period without years or months |
| '(duration microseconds), or '(duration nanoseconds) |
Python’s datetime and timedelta stop at the microsecond, so Python’s unit is always microseconds; gregor’s go to the nanosecond. A column (or a literal) is nanoseconds, so nothing is lost, when a value has a nonzero sub-microsecond part and a nanosecond column holds every value: datetimes from 1677-09-21 00:12:43.145224192 to 2262-04-11 23:47:16.854775807, durations shorter than 263 nanoseconds (about 292 years) either way. Otherwise it is microseconds, which hold every datetime Polars can print, and the sub-microsecond part is dropped as #:dtype drops it (below). gregor’s now reads a flonum clock and nearly always has a sub-microsecond part: a column of nows alone is then nanoseconds, and one that also holds a datetime from 1600 is microseconds.
A period’s weeks, days, hours and smaller fields add up to one fixed length (a day is 24 hours, as in Polars); years and months have no fixed length and are refused. With #:dtype, a value is converted to the dtype’s unit and a finer part dropped as Polars’ own casts drop it: a datetime is floored, a duration truncated toward zero. A value the dtype cannot hold raises, and so does a date or datetime outside the years -262143 to 262142, which Polars cannot print.
A gregor moment is an instant with a time zone, and makes a zoned column, as Python’s aware datetime does. A column has one zone, so a list settles it as Python settles a list: the first value that is not null decides. A moment in a named zone gives that zone, "Europe/Brussels" or "UTC"; a moment at a fixed offset (gregor’s #:tz 3600) gives "UTC"; and each other moment keeps its instant in that zone. A naive gregor datetime in a zoned column is read as UTC, and a list whose first value is naive makes a naive column of each moment’s UTC clock time. #:dtype names the column’s dtype outright, under the same reading: a zoned dtype converts every moment to its zone, and a naive one takes each moment’s UTC clock time, where Python’s pl.Datetime("us") keeps the values’ zone.
Polars reads zones from chrono-tz’s compiled copy of the tz database, and
every reading here is Polars’: a moment read back carries the offset Polars
gives its instant, and a printed series or dataframe and describe
show what Python shows. gregor’s own functions —
Reading back, ref and every conversion in Converting to Racket values return each value with its column’s full precision: a millisecond, microsecond or nanosecond datetime as the gregor datetime (or, for a zoned column, moment in the column’s zone) of exactly that instant, where Python’s to_list stops at the microsecond. A date column’s years may be negative, as gregor’s and Polars’ may.
> (series (list (date 2013 1 1) polars-null (date -1300 5 23)) #:name "d")
shape: (3,)
Series: 'd' [date]
[
2013-01-01
null
-1300-05-23
]
> (series (list (time 5 6 7 123456789)) #:name "t")
shape: (1,)
Series: 't' [time]
[
05:06:07.123456789
]
> (series (list (datetime 2013 1 1 5 6 7 123456000)) #:name "dt")
shape: (1,)
Series: 'dt' [datetime[us]]
[
2013-01-01 05:06:07.123456
]
> (series (list (datetime 2013 1 1 5 6 7 123456789)) #:name "dt")
shape: (1,)
Series: 'dt' [datetime[ns]]
[
2013-01-01 05:06:07.123456789
]
> (series (list (datetime 2013 1 1 5 6 7 123456789) (datetime 1600 1 1)) #:name "dt")
shape: (2,)
Series: 'dt' [datetime[us]]
[
2013-01-01 05:06:07.123456
1600-01-01 00:00:00
]
> (series (list (hours 36) (minutes 90)) #:name "wait")
shape: (2,)
Series: 'wait' [duration[us]]
[
1d 12h
1h 30m
]
> (series (list (datetime 2013 1 1 5 6 7 123456789)) #:name "dt" #:dtype '(datetime milliseconds))
shape: (1,)
Series: 'dt' [datetime[ms]]
[
2013-01-01 05:06:07.123
]
> (series->list (cast (series '(1500 -1) #:dtype 'i64) '(datetime milliseconds))) '(#<datetime 1970-01-01T00:00:01.5> #<datetime 1969-12-31T23:59:59.999>)
> (define calendar (dataframe (list (series (list (date 2013 5 31) (date 2013 6 1) (date 2013 6 2)) #:name "d") (series '(1 2 3) #:name "v")))) > (filter calendar (> (col "d") (date 2013 6 1)))
shape: (1, 2)
┌────────────┬─────┐
│ d ┆ v │
│ --- ┆ --- │
│ date ┆ i64 │
╞════════════╪═════╡
│ 2013-06-02 ┆ 3 │
└────────────┴─────┘
> (filter calendar (is-between "d" (date 2013 5 31) (datetime 2013 6 1 12)))
shape: (2, 2)
┌────────────┬─────┐
│ d ┆ v │
│ --- ┆ --- │
│ date ┆ i64 │
╞════════════╪═════╡
│ 2013-05-31 ┆ 1 │
│ 2013-06-01 ┆ 2 │
└────────────┴─────┘
> (filter calendar (is-in "d" (list (date 2013 5 31) (date 2013 6 2))))
shape: (2, 2)
┌────────────┬─────┐
│ d ┆ v │
│ --- ┆ --- │
│ date ┆ i64 │
╞════════════╪═════╡
│ 2013-05-31 ┆ 1 │
│ 2013-06-02 ┆ 3 │
└────────────┴─────┘
> (is-in "d" (list (datetime 2013 5 31) (moment 2013 6 2 #:tz "UTC"))) is-in: the values have no common dtype: their time zones
differ
dtype: '(datetime microseconds #f)
other dtype: '(datetime microseconds "UTC")
> (series (list (moment 2021 3 27 #:tz "Europe/Brussels") (moment 2021 3 28 5 #:tz "Europe/Brussels") (moment 2021 3 27 #:tz "Asia/Kathmandu")) #:name "zoned")
shape: (3,)
Series: 'zoned' [datetime[us, Europe/Brussels]]
[
2021-03-27 00:00:00 CET
2021-03-28 05:00:00 CEST
2021-03-26 19:15:00 CET
]
> (series->list (series (list (moment 2021 3 27 #:tz 3600)) #:name "offset")) '(#<moment 2021-03-26T23:00:00Z[UTC]>)
> (series (list (datetime 2021 3 27 12)) #:dtype '(datetime milliseconds "Europe/Brussels"))
shape: (1,)
Series: '' [datetime[ms, Europe/Brussels]]
[
2021-03-27 13:00:00 CET
]
> (series (list (datetime 2021 3 27)) #:dtype '(datetime microseconds "Mars/Base")) series: cannot convert to '(datetime microseconds
"Mars/Base"): unable to parse time zone: 'Mars/Base'. Please
check the Time Zone Database for a list of available time
zones.
Hint: did you mean 'Asia/Baku' instead?
> (series (list (months 1))) series: cannot infer a dtype from elements; pass #:dtype
explicitly
> (series (list (datetime 1500 1 1)) #:dtype '(datetime nanoseconds)) series: value out of range for this dtype
dtype: '(datetime nanoseconds)
value: #<datetime 1500-01-01T00:00:00>
2.2.3 Categorical, Enum and Decimal
A 'categorical column stores each distinct string once and a code per row, as Polars’ Categorical; an Enum column (dtype '(enum cat ...), defined with define-enum) does the same over categories declared up front, in order, as pl.Enum. Both read back as symbols, which Racket interns: a symbol is already the dictionary encoding. The codes stay inside Polars. Every categorical column in the process shares them, and they restart once the last one is dropped, so each conversion fetches the strings afresh.
Build one with series (a list of symbols infers 'categorical), cast, or read-csv’s #:schema-overrides ('categorical only).
A categorical sorts and compares by its strings; an Enum by the declared order of its categories.
A value outside an Enum’s categories raises, whether it is built, cast or compared; a categorical takes any string.
API gap: a categorical or Enum value that series builds, and an Enum category, cannot hold a NUL character (#\nul): it raises. A 'string value can hold one.
Two categorical columns share one encoding, so they join, stack and compare without re-encoding; two Enums with the same categories are the same dtype.
describe gives a categorical or Enum column count and null_count only, as Python does.
syntax
(define-enum id category ...+)
category = id | string
> (define-enum log-levels debug info warning error) > log-levels '(enum debug info warning error)
> (define levels (series '(debug info debug error) #:name "level" #:dtype log-levels)) > (equal? (dtype levels) log-levels) #t
> (series->list (cast (series '("warning" "info")) log-levels)) '(warning info)
> (define-enum sizes small "Very High") > sizes '(enum small |Very High|)
> (define-enum twice debug info debug) eval:430:0: define-enum: duplicate enum category
at: debug
in: (define-enum twice debug info debug)
> (series '(info fatal) #:dtype log-levels) series: 1 of 2 values is not in '(enum debug info warning
error): fatal
A '(decimal precision scale) column holds exact decimals, which ref and the conversions read as exact rationals. Decimal columns come from Parquet; cast reads them into other dtypes. API gap: no #:dtype or cast to a Decimal.
> (define logs (dataframe (list (series '(debug info debug error) #:name "level" #:dtype log-levels) (series '(api db api db) #:name "source")))) > (for/list ([name (column-names logs)]) (dtype (ref logs name))) '((enum debug info warning error) categorical)
> (filter logs (> (col "level") 'info))
shape: (1, 2)
┌───────┬────────┐
│ level ┆ source │
│ --- ┆ --- │
│ enum ┆ cat │
╞═══════╪════════╡
│ error ┆ db │
└───────┴────────┘
> (sort logs "level")
shape: (4, 2)
┌───────┬────────┐
│ level ┆ source │
│ --- ┆ --- │
│ enum ┆ cat │
╞═══════╪════════╡
│ debug ┆ api │
│ debug ┆ api │
│ info ┆ db │
│ error ┆ db │
└───────┴────────┘
> (sort logs "source")
shape: (4, 2)
┌───────┬────────┐
│ level ┆ source │
│ --- ┆ --- │
│ enum ┆ cat │
╞═══════╪════════╡
│ debug ┆ api │
│ debug ┆ api │
│ info ┆ db │
│ error ┆ db │
└───────┴────────┘
> (ref (ref logs "source") 1) 'db
> (select logs (col 'categorical))
shape: (4, 1)
┌────────┐
│ source │
│ --- │
│ cat │
╞════════╡
│ api │
│ db │
│ api │
│ db │
└────────┘
> (select logs (> (col "level") 'fatal)) select: conversion from `str` to `enum` failed for value
"fatal"
> (series (list "api" "d\u0000b") #:dtype 'categorical) series: contract violation
expected: a categorical or enum value without a NUL
character
given: "d\u0000b"
> (define produce (read-parquet "produce.parquet")) > (dtype (ref produce "price")) '(decimal 10 2)
> (for/sum ([price (ref produce "price")] #:unless (polars-null? price)) price) 281/20
2.2.4 dtype promotion
Reductions follow a simple, predictable rule. The widening order, narrow to wide, is
'int8 < 'int16 < 'int32 < 'int64
'uint8 < 'uint16 < 'uint32 < 'uint64
any integer < 'float32 < 'float64
sum, min and max preserve the input dtype. mean promotes to 'float64. Use series-cast to change a series’ dtype explicitly.
A plain value beside a series in an operator is a constant of the series’ dtype when it fits; a gregor value keeps its own dtype, and a non-integer real beside an integer series is 'float64. Otherwise the operator runs its expression form, (op (col name) value), with that form’s dtypes and values: beside a 'null series (where Python’s eager operators raise) or with polars-null, beside an Enum series a string or symbol, beside a Decimal series a real, and beside an integer series an integer past the dtype’s range. Any other value raises exn:fail:contract naming it and the series’ dtype: a string, symbol or boolean beside a numeric series, a number beside a string series, a negative integer beside an unsigned series (as in Python’s eager operators), or a value no literal takes.
> (+ (series '(1 2) #:dtype 'u8) 300)
shape: (2,)
Series: '' [u16]
[
301
302
]
> (= (series (list "low" "mid" polars-null "high") #:dtype '(enum low mid high)) "bogus")
shape: (4,)
Series: '' [bool]
[
#f
#f
null
#f
]
> (+ (series '(1 2) #:dtype 'u8) -1) +: the value cannot be an element of a series of this dtype
value: -1
series dtype: 'uint8
> (= (series '("x")) 1) =: the value cannot be an element of a series of this dtype
value: 1
series dtype: 'string
2.2.5 Low-level Series API
The generic layer is built on monomorphic, dtype-suffixed bindings that operate directly on the foreign series. They remain exported. A series wrapper is accepted anywhere one of them expects a series (the wrapper marshals transparently, and satisfies Series-ptr?), but what they return is the raw foreign pointer, not a wrapper — so the results do not print in Polars’ format and do not answer to series?. Prefer series and the generic operations above; reach for these when you need a specific dtype or a specific typed result.
procedure
(Series-ptr? v) → boolean?
v : any/c
procedure
(series-drop s) → void?
s : Series-ptr?
A drop must not run while another thread uses the same series: the check is made as the call converts its arguments, so a drop between that and the native call frees memory the call goes on to read, which can crash Racket or return wrong values. Nothing guards against it; a program that shares a value between threads synchronises the drop with them, or leaves the release to the collector, which never runs it while a thread can still reach the value.
> (define temps (series '(21.5 23.0 19.8) #:name "temp")) > (series-drop temps) > temps #<series: dropped>
> (len temps) series: series has been dropped
> (series-drop temps) series-drop: series has been dropped
procedure
(series-new-i8 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-i16 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-i32 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-i64 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-u8 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-u16 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-u32 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-u64 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-f32 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-f64 name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-bool name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-str name values) → Series-ptr?
name : string? values : list?
procedure
(series-new-null name length) → Series-ptr?
name : string? length : exact-nonnegative-integer?
> (display (series->string (series-new-null "nothing" 3)))
shape: (3,)
Series: 'nothing' [null]
[
null
null
null
]
procedure
(series-sum-i32 s) → (or/c exact-integer? #f)
s : Series-ptr?
procedure
(series-min-i32 s) → (or/c exact-integer? #f)
s : Series-ptr?
procedure
(series-max-i32 s) → (or/c exact-integer? #f)
s : Series-ptr?
procedure
(series-mean-i32 s) → (or/c flonum? #f)
s : Series-ptr?
procedure
(series-sum-f64 s) → (or/c flonum? #f)
s : Series-ptr?
procedure
(series-min-f64 s) → (or/c flonum? #f)
s : Series-ptr?
procedure
(series-max-f64 s) → (or/c flonum? #f)
s : Series-ptr?
procedure
(series-mean-f64 s) → (or/c flonum? #f)
s : Series-ptr?
> (define gaps (series (list polars-null polars-null) #:dtype 'float64)) > (list (series-min-f64 gaps) (series-sum-f64 gaps) (min gaps)) '(#f 0.0 polars-null)
procedure
(series-cast s dtype) → Series-ptr?
s : Series-ptr? dtype : (or/c symbol? pair?)
procedure
(series-sort s [ #:descending descending #:nulls-last nulls-last]) → Series-ptr? s : Series-ptr? descending : boolean? = #f nulls-last : boolean? = #f
2.3 DataFrames
A dataframe is a collection of equal-length named series. Like a series it is a wrapper value (dataframe?) carrying the column data; it prints as a Polars table, so display (or ~a, or the REPL) renders it with no separate display call.
procedure
(dataframe? v) → boolean?
v : any/c
procedure
(dataframe columns) → dataframe?
columns : (listof series?)
> (dataframe (list (series '("a" "b") #:name "k") (series '(1 2) #:name "v")))
shape: (2, 2)
┌─────┬─────┐
│ k ┆ v │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ a ┆ 1 │
│ b ┆ 2 │
└─────┴─────┘
> (dataframe (list (series '(1) #:name "k") (series '(2) #:name "k"))) dataframe: could not create a new DataFrame: column with
name 'k' has more than one occurrence
procedure
(shape x) → (listof exact-nonnegative-integer?)
x : has-shape?
procedure
(shape/values x) →
exact-nonnegative-integer? ... x : has-shape?
procedure
d : dataframe?
procedure
d : dataframe?
procedure
(column-names d) → (listof string?)
d : dataframe?
procedure
(column-name d i) → string?
d : dataframe? i : exact-nonnegative-integer?
procedure
x : has-ref? key : (or/c exact-nonnegative-integer? string?) = absent
columns :
(or/c exact-nonnegative-integer? string? (listof (or/c exact-nonnegative-integer? string?))) = absent rows : any/c = absent
> (define starred (dataframe (list (series '(1) #:name "*") (series '(2) #:name "b")))) > (ref starred '("*"))
shape: (1, 1)
┌─────┐
│ * │
│ --- │
│ i64 │
╞═════╡
│ 1 │
└─────┘
> (ref starred '("b" 1)) ref: "b" named more than once
procedure
(describe x) → dataframe?
x : (or/c series? dataframe?)
Numeric, boolean, null and nested columns summarise as 'float64, every other column as strings; temporal values are written as Python prints them, a zoned datetime as its local time and UTC offset. Quantiles use nearest interpolation. Every statistic of every column comes from one query, so Polars computes the columns in parallel.
API gap: a binary column gets no "min" or "max".
Numeric, string and boolean columns, with nulls:
> (define flights (dataframe (list (series (list "UA" "AA" "UA" polars-null) #:name "carrier") (series (list 1400 733 polars-null 1089) #:name "distance") (series (list #t #f #t #t) #:name "on_time")))) > (describe flights)
shape: (9, 4)
┌────────────┬─────────┬────────────┬─────────┐
│ statistic ┆ carrier ┆ distance ┆ on_time │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 ┆ f64 │
╞════════════╪═════════╪════════════╪═════════╡
│ count ┆ 3 ┆ 3.0 ┆ 4.0 │
│ null_count ┆ 1 ┆ 1.0 ┆ 0.0 │
│ mean ┆ null ┆ 1074.0 ┆ 0.75 │
│ std ┆ null ┆ 333.752903 ┆ null │
│ min ┆ AA ┆ 733.0 ┆ 0.0 │
│ 25% ┆ null ┆ 1089.0 ┆ null │
│ 50% ┆ null ┆ 1089.0 ┆ null │
│ 75% ┆ null ┆ 1400.0 ┆ null │
│ max ┆ UA ┆ 1400.0 ┆ 1.0 │
└────────────┴─────────┴────────────┴─────────┘
Datetime, duration and date columns get a mean and quartiles; a date column’s mean is a datetime:
> (define times (~> (dataframe (list (series (list (datetime 2013 1 1 5) (datetime 2013 1 1 6) (datetime 2013 1 2 7) (datetime 2013 1 3 8)) #:name "scheduled") (series (list (datetime 2013 1 1 5 12) (datetime 2013 1 1 5 57) polars-null (datetime 2013 1 3 9 30)) #:name "departed"))) (with-columns (alias (- (col "departed") (col "scheduled")) "delay") (alias (cast "scheduled" 'date) "day")))) > (describe times)
shape: (9, 5)
┌────────────┬─────────────────────┬─────────────────────┬──────────────────┬─────────────────────┐
│ statistic ┆ scheduled ┆ departed ┆ delay ┆ day │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ str │
╞════════════╪═════════════════════╪═════════════════════╪══════════════════╪═════════════════════╡
│ count ┆ 4 ┆ 3 ┆ 3 ┆ 4 │
│ null_count ┆ 0 ┆ 1 ┆ 1 ┆ 0 │
│ mean ┆ 2013-01-02 00:30:00 ┆ 2013-01-01 22:53:00 ┆ 0:33:00 ┆ 2013-01-01 18:00:00 │
│ std ┆ null ┆ null ┆ null ┆ null │
│ min ┆ 2013-01-01 05:00:00 ┆ 2013-01-01 05:12:00 ┆ -1 day, 23:57:00 ┆ 2013-01-01 │
│ 25% ┆ 2013-01-01 06:00:00 ┆ 2013-01-01 05:57:00 ┆ 0:12:00 ┆ 2013-01-01 │
│ 50% ┆ 2013-01-02 07:00:00 ┆ 2013-01-01 05:57:00 ┆ 0:12:00 ┆ 2013-01-02 │
│ 75% ┆ 2013-01-02 07:00:00 ┆ 2013-01-03 09:30:00 ┆ 1:30:00 ┆ 2013-01-02 │
│ max ┆ 2013-01-03 08:00:00 ┆ 2013-01-03 09:30:00 ┆ 1:30:00 ┆ 2013-01-03 │
└────────────┴─────────────────────┴─────────────────────┴──────────────────┴─────────────────────┘
A zoned datetime column’s statistics carry the offset in force at each:
> (describe (series (list (moment 2021 3 27 12 #:tz "Europe/Brussels") (moment 2021 7 1 #:tz "Europe/Brussels")) #:name "local"))
shape: (8, 2)
┌────────────┬───────────────────────────┐
│ statistic ┆ value │
│ --- ┆ --- │
│ str ┆ str │
╞════════════╪═══════════════════════════╡
│ count ┆ 2 │
│ null_count ┆ 0 │
│ mean ┆ 2021-05-14 06:30:00+02:00 │
│ min ┆ 2021-03-27 12:00:00+01:00 │
│ 25% ┆ 2021-03-27 12:00:00+01:00 │
│ 50% ┆ 2021-07-01 00:00:00+02:00 │
│ 75% ┆ 2021-07-01 00:00:00+02:00 │
│ max ┆ 2021-07-01 00:00:00+02:00 │
└────────────┴───────────────────────────┘
A series keeps only the rows its dtype has:
> (describe (series (list 3 1 polars-null 4 1 5) #:name "n"))
shape: (9, 2)
┌────────────┬──────────┐
│ statistic ┆ value │
│ --- ┆ --- │
│ str ┆ f64 │
╞════════════╪══════════╡
│ count ┆ 5.0 │
│ null_count ┆ 1.0 │
│ mean ┆ 2.8 │
│ std ┆ 1.788854 │
│ min ┆ 1.0 │
│ 25% ┆ 1.0 │
│ 50% ┆ 3.0 │
│ 75% ┆ 4.0 │
│ max ┆ 5.0 │
└────────────┴──────────┘
> (describe (ref times "day"))
shape: (8, 2)
┌────────────┬─────────────────────┐
│ statistic ┆ value │
│ --- ┆ --- │
│ str ┆ str │
╞════════════╪═════════════════════╡
│ count ┆ 4 │
│ null_count ┆ 0 │
│ mean ┆ 2013-01-01 18:00:00 │
│ min ┆ 2013-01-01 │
│ 25% ┆ 2013-01-01 │
│ 50% ┆ 2013-01-02 │
│ 75% ┆ 2013-01-02 │
│ max ┆ 2013-01-03 │
└────────────┴─────────────────────┘
A nested column (here the lists agg collects) and a null-dtype column report only their counts, as floats; a frame with no rows reports zero counts:
> (~> flights (group-by "carrier") (agg (col "distance")) describe)
shape: (9, 3)
┌────────────┬─────────┬──────────┐
│ statistic ┆ carrier ┆ distance │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 │
╞════════════╪═════════╪══════════╡
│ count ┆ 2 ┆ 3.0 │
│ null_count ┆ 1 ┆ 0.0 │
│ mean ┆ null ┆ null │
│ std ┆ null ┆ null │
│ min ┆ AA ┆ null │
│ 25% ┆ null ┆ null │
│ 50% ┆ null ┆ null │
│ 75% ┆ null ┆ null │
│ max ┆ UA ┆ null │
└────────────┴─────────┴──────────┘
> (~> flights (select (alias (cast "carrier" 'null) "nothing")) describe)
shape: (9, 2)
┌────────────┬─────────┐
│ statistic ┆ nothing │
│ --- ┆ --- │
│ str ┆ f64 │
╞════════════╪═════════╡
│ count ┆ 0.0 │
│ null_count ┆ 4.0 │
│ mean ┆ null │
│ std ┆ null │
│ min ┆ null │
│ 25% ┆ null │
│ 50% ┆ null │
│ 75% ┆ null │
│ max ┆ null │
└────────────┴─────────┘
> (describe (head flights 0))
shape: (9, 4)
┌────────────┬─────────┬──────────┬─────────┐
│ statistic ┆ carrier ┆ distance ┆ on_time │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 ┆ f64 │
╞════════════╪═════════╪══════════╪═════════╡
│ count ┆ 0 ┆ 0.0 ┆ 0.0 │
│ null_count ┆ 0 ┆ 0.0 ┆ 0.0 │
│ mean ┆ null ┆ null ┆ null │
│ std ┆ null ┆ null ┆ null │
│ min ┆ null ┆ null ┆ null │
│ 25% ┆ null ┆ null ┆ null │
│ 50% ┆ null ┆ null ┆ null │
│ 75% ┆ null ┆ null ┆ null │
│ max ┆ null ┆ null ┆ null │
└────────────┴─────────┴──────────┴─────────┘
2.3.1 Converting to Racket values
procedure
(dataframe->columns d [ #:columns columns #:null null-value]) → (listof (cons/c string? vector?)) d : dataframe? columns : (listof string?) = (column-names d) null-value : any/c = polars-null
> (define kv (dataframe (list (series '("a" "b") #:name "k") (series (list 1 polars-null) #:name "v")))) > (dataframe->columns kv) '(("k" . #("a" "b")) ("v" . #(1 polars-null)))
> (dataframe->columns kv #:columns '("v" "k") #:null 0) '(("v" . #(1 0)) ("k" . #("a" "b")))
> (dataframe->columns kv #:columns '("k" "k")) dataframe->columns: duplicate column
column: "k"
> (dataframe->columns kv #:columns '("nope")) dataframe->columns: no such column
column: "nope"
procedure
(dataframe->hash d [ #:columns columns #:null null-value]) → (and/c (hash/c string? vector?) immutable?) d : dataframe? columns : (listof string?) = (column-names d) null-value : any/c = polars-null
> (dataframe->hash kv) '#hash(("k" . #("a" "b")) ("v" . #(1 polars-null)))
> (hash-ref (dataframe->hash kv #:null 0) "v") '#(1 0)
> (dataframe->hash kv #:columns '("k")) '#hash(("k" . #("a" "b")))
> (dataframe->hash kv #:columns '("nope")) dataframe->hash: no such column
column: "nope"
procedure
(in-dataframe-columns d [#:columns columns]) → sequence?
d : dataframe? columns : (listof string?) = (column-names d)
> (for/list ([column (in-dataframe-columns kv)]) (series-name column)) '("k" "v")
> (for/list ([column (in-dataframe-columns kv #:columns '("v"))]) (series->list column #:null 0)) '((1 0))
> (for/first ([column (in-dataframe-columns kv)]) column)
shape: (2,)
Series: 'k' [str]
[
"a"
"b"
]
> (in-dataframe-columns kv #:columns '("k" "k")) in-dataframe-columns: duplicate column
column: "k"
procedure
(in-dataframe-rows d [ #:columns columns #:named? named? #:null null-value #:buffer-size buffer-size]) → sequence? d : dataframe? columns : (listof string?) = (column-names d) named? : boolean? = #f null-value : any/c = polars-null buffer-size : exact-positive-integer? = 512
The rows are converted buffer-size at a time, as buffer_size does: each buffer is one bulk copy per column, never a foreign call per value. A loop holds one buffer’s values at a time, so memory stays bounded for any height, and one that stops early converts at most one buffer beyond the rows it reads. A larger buffer makes fewer calls and holds more values.
The columns are fetched and checked when in-dataframe-rows is called: an unknown or repeated name, or a column of an unsupported dtype, raises exn:fail:contract naming the column. The sequence holds those columns rather than d, and each iteration starts from the first row. A lazyframe is not accepted; collect it first. Python’s buffer_size=0, a row at a time, has no counterpart.
> (define trips (dataframe (list (series '(UA AA UA) #:name "carrier") (series (list 2 polars-null -3) #:name "delay") (cast (series (list (datetime 2013 1 1) (datetime 2013 1 1) (datetime 2013 1 2)) #:name "day") 'date)))) > (for/list ([row (in-dataframe-rows trips)]) row)
'(#(UA 2 #<date 2013-01-01>)
#(AA polars-null #<date 2013-01-01>)
#(UA -3 #<date 2013-01-02>))
> (for/list ([row (in-dataframe-rows trips #:columns '("delay" "carrier") #:named? #t)]) row)
'(#hash(("carrier" . UA) ("delay" . 2))
#hash(("carrier" . AA) ("delay" . polars-null))
#hash(("carrier" . UA) ("delay" . -3)))
> (for/sum ([row (in-dataframe-rows trips #:columns '("delay") #:null 0)]) (vector-ref row 0)) -1
> (for/list ([row (in-dataframe-rows trips #:buffer-size 2)]) (define-values (carrier delay day) (vector->values row)) (list carrier (date->iso8601 day))) '((UA "2013-01-01") (AA "2013-01-01") (UA "2013-01-02"))
> (in-dataframe-rows trips #:columns '("nope")) in-dataframe-rows: no such column
column: "nope"
> (in-dataframe-rows trips #:buffer-size 0) in-dataframe-rows: contract violation
expected: exact-positive-integer?
given: 0
in: the #:buffer-size argument of
(->*
(dataframe?)
(#:buffer-size
exact-positive-integer?
#:columns
(listof string?)
#:named?
boolean?
#:null
any/c)
sequence?)
contract from:
<pkgs>/polars/private/generic/convert.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/generic/convert.rkt:32:3
procedure
(dataframe->rows d [ #:columns columns #:named? named? #:null null-value]) → (listof (or/c vector? (and/c hash? immutable?))) d : dataframe? columns : (listof string?) = (column-names d) named? : boolean? = #f null-value : any/c = polars-null
> (dataframe->rows trips)
'(#(UA 2 #<date 2013-01-01>)
#(AA polars-null #<date 2013-01-01>)
#(UA -3 #<date 2013-01-02>))
> (dataframe->rows trips #:columns '("carrier") #:named? #t) '(#hash(("carrier" . UA)) #hash(("carrier" . AA)) #hash(("carrier" . UA)))
> (dataframe->rows (head trips 0)) '()
> (dataframe->rows trips #:columns '("day" "day")) dataframe->rows: duplicate column
column: "day"
procedure
(dataframe->f64vector d [ #:columns columns #:order order #:null null-value])
→
f64vector? exact-nonnegative-integer? exact-nonnegative-integer? d : dataframe? columns : (listof string?) = (column-names d) order : (or/c 'fortran 'c) = 'fortran null-value : (or/c real? 'error) = +nan.0
> (define xy (dataframe (list (series (list 1 2 polars-null) #:name "a") (series '(0.5 1.5 2.5) #:name "b")))) > (define-values (m nrows ncols) (dataframe->f64vector xy)) > (list nrows ncols) '(3 2)
> (f64vector->list m) '(1.0 2.0 +nan.0 0.5 1.5 2.5)
> (define-values (m/f rows/f cols/f) (dataframe->f64vector xy #:order 'fortran)) > (equal? (f64vector->list m/f) (f64vector->list m)) #t
> (define-values (m/c rows/c cols/c) (dataframe->f64vector xy #:order 'c)) > (f64vector->list m/c) '(1.0 0.5 2.0 1.5 +nan.0 2.5)
> (define-values (b rows/b cols/b) (dataframe->f64vector xy #:columns '("b") #:null 'error)) > (f64vector->list b) '(0.5 1.5 2.5)
> (define-values (z rows/z cols/z) (dataframe->f64vector xy #:null 0)) > (f64vector->list z) '(1.0 2.0 0.0 0.5 1.5 2.5)
> (dataframe->f64vector xy #:null 'error) dataframe->f64vector: null value
column: "a"
row: 2
> (dataframe->f64vector (dataframe (list (series '("p") #:name "s")))) dataframe->f64vector: not a numeric column
column: "s"
dtype: 'string
2.3.2 Low-level DataFrame API
The generic layer above is built on a set of monomorphic dataframe-* bindings that operate directly on the foreign dataframe. They remain exported and accept the dataframe wrapper (it marshals transparently); the generic operations are simply the preferred surface.
procedure
(dataframe-new columns) → dataframe?
columns : (listof series?)
procedure
(dataframe-shape d) →
exact-nonnegative-integer? exact-nonnegative-integer? d : dataframe?
procedure
d : dataframe?
procedure
d : dataframe?
procedure
(dataframe-column d name) → series?
d : dataframe? name : string?
procedure
(dataframe-column-name d i) → string?
d : dataframe? i : exact-nonnegative-integer?
procedure
(dataframe-column-names d) → (listof string?)
d : dataframe?
procedure
(dataframe-select d names) → dataframe?
d : dataframe? names : (listof string?)
procedure
(display-dataframe d [out]) → void?
d : dataframe? out : output-port? = (current-output-port)
> (define scores (dataframe (list (series '(10 25 18) #:name "score" #:dtype 'i32)))) > (series-sum-i32 (dataframe-column scores "score")) 53
> (dataframe-column scores "points") dataframe-column: no column named "points"
> (dataframe (list far-date)) #<dataframe: cannot print: polars panicked: out-of-range date>
> (display-dataframe (dataframe (list far-date))) display-dataframe: cannot print the dataframe: polars
panicked: out-of-range date
procedure
(DataFrame-ptr? v) → boolean?
v : any/c
procedure
(dataframe-drop d) → void?
d : DataFrame-ptr?
> (define readings (dataframe (list (series '(3 1 2) #:name "v")))) > (dataframe-drop readings) > readings #<dataframe: dropped>
> (height readings) dataframe: dataframe has been dropped
> (dataframe-drop readings) dataframe-drop: dataframe has been dropped
procedure
(dataframe-vstack top bottom) → DataFrame-ptr?
top : DataFrame-ptr? bottom : DataFrame-ptr?
procedure
(dataframe-sort d names [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → DataFrame-ptr? d : DataFrame-ptr? names : (non-empty-listof string?) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
2.3.3 Reading & writing
procedure
(dataframe-read-parquet path [ #:columns columns #:n-rows n-rows #:row-index-name row-index-name #:row-index-offset row-index-offset #:parallel parallel #:use-statistics use-statistics #:glob glob #:rechunk rechunk #:low-memory low-memory #:include-file-paths include-file-paths #:missing-columns missing-columns]) → DataFrame-ptr? path : path-string?
columns : (or/c (listof column-name/c) (listof column-index/c) #f) = #f n-rows : (or/c size/c #f) = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 parallel : parquet-parallel/c = 'auto use-statistics : boolean? = #t glob : boolean? = #t rechunk : boolean? = #f low-memory : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f missing-columns : (or/c 'raise 'insert) = 'raise
procedure
(lazyframe-scan-parquet path [ #:n-rows n-rows #:row-index-name row-index-name #:row-index-offset row-index-offset #:parallel parallel #:use-statistics use-statistics #:glob glob #:rechunk rechunk #:low-memory low-memory #:include-file-paths include-file-paths #:missing-columns missing-columns #:cache cache]) → LazyFrame-ptr? path : path-string? n-rows : (or/c size/c #f) = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 parallel : parquet-parallel/c = 'auto use-statistics : boolean? = #t glob : boolean? = #t rechunk : boolean? = #f low-memory : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f missing-columns : (or/c 'raise 'insert) = 'raise cache : boolean? = #t
procedure
(dataframe-write-parquet frame path [ #:compression compression #:compression-level compression-level #:statistics statistics #:row-group-size row-group-size #:data-page-size data-page-size]) → void? frame : DataFrame-ptr? path : path-string? compression : parquet-compression/c = 'zstd compression-level : (or/c exact-integer? #f) = #f statistics : parquet-statistics/c = #t row-group-size : (or/c size/c #f) = #f data-page-size : (or/c size/c #f) = #f
> (dataframe-height (dataframe-read-parquet "flights.parquet" #:n-rows 5)) 5
> (dataframe-width (lazyframe-collect (lazyframe-scan-parquet "produce.parquet"))) 3
> (dataframe-write-parquet (dataframe-read-parquet "produce.parquet") (build-path parquet-dir "raw.parquet") #:compression 'snappy) > (dataframe-height (dataframe-read-parquet (build-path parquet-dir "raw.parquet"))) 4
> (dataframe-read-parquet "produce.parquet" #:columns '(3)) dataframe-read-parquet: failed to read parquet from
produce.parquet: column index 3 is out of range for 3
columns
> (lazyframe-scan-parquet "produce.parquet" #:missing-columns 'ignore) lazyframe-scan-parquet: contract violation
expected: (or/c (quote raise) (quote insert))
given: 'ignore
in: the missing-columns argument of
(->i
((path path-string?))
(#:n-rows
(n-rows (or/c size/c #f))
#:row-index-name
(row-index-name (or/c column-name/c #f))
#:row-index-offset
(row-index-offset row-index/c)
#:parallel
(parallel parquet-parallel/c)
#:use-statistics
(use-statistics boolean?)
#:glob
(glob boolean?)
#:rechunk
(rechunk boolean?)
#:low-memory
(low-memory boolean?)
#:include-file-paths
(include-file-paths
(or/c column-name/c #f))
#:missing-columns
(missing-columns (or/c 'raise 'insert))
#:cache
(cache boolean?))
(result LazyFrame-ptr?))
contract from:
<pkgs>/polars/private/parquet.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/parquet.rkt:23:11
> (dataframe-write-parquet (dataframe-read-parquet "produce.parquet") (build-path parquet-dir "raw.parquet") #:compression 'lzo) dataframe-write-parquet: contract violation
expected: parquet-compression/c
given: 'lzo
in: the compression argument of
(->i
((frame DataFrame-ptr?)
(path path-string?))
(#:compression
(compression parquet-compression/c)
#:compression-level
(compression-level
(or/c exact-integer? #f))
#:statistics
(statistics parquet-statistics/c)
#:row-group-size
(row-group-size (or/c size/c #f))
#:data-page-size
(data-page-size (or/c size/c #f)))
#:pre/desc
(compression compression-level)
(level-problem
compression
compression-level)
(result void?))
contract from:
<pkgs>/polars/private/parquet.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/parquet.rkt:24:11
procedure
(dataframe-read-csv path [ #:has-header has-header #:columns columns #:new-columns new-columns #:separator separator #:comment-prefix comment-prefix #:quote-char quote-char #:skip-rows skip-rows #:skip-lines skip-lines #:schema-overrides schema-overrides #:null-values null-values #:missing-utf8-is-empty-string missing-utf8-is-empty-string #:ignore-errors ignore-errors #:try-parse-dates try-parse-dates #:infer-schema infer-schema #:infer-schema-length infer-schema-length #:n-rows n-rows #:encoding encoding #:skip-rows-after-header skip-rows-after-header #:row-index-name row-index-name #:row-index-offset row-index-offset #:eol-char eol-char #:raise-if-empty raise-if-empty #:truncate-ragged-lines truncate-ragged-lines #:decimal-comma decimal-comma #:glob glob]) → DataFrame-ptr? path : path-string? has-header : boolean? = #t
columns : (or/c (and/c (listof column-name/c) distinct?) (and/c (listof size/c) distinct?)) = '() new-columns : (and/c (listof column-name/c) distinct?) = '() separator : (or/c csv-char/c #f) = #f
comment-prefix : (or/c (and/c non-empty-string? c-string/c) #f) = #f quote-char : (or/c csv-char/c #f) = #\" skip-rows : size/c = 0 skip-lines : size/c = 0
schema-overrides : (and/c (listof (cons/c column-name/c csv-dtype/c)) distinct-names?) = '()
null-values :
(or/c c-string/c (listof c-string/c) (and/c (listof (cons/c column-name/c c-string/c)) distinct-names?) #f) = #f missing-utf8-is-empty-string : boolean? = #f ignore-errors : boolean? = #f try-parse-dates : boolean? = #f infer-schema : boolean? = #t infer-schema-length : (or/c size/c #f) = 100 n-rows : (or/c size/c #f) = #f encoding : (or/c 'utf8 'utf8-lossy) = 'utf8 skip-rows-after-header : size/c = 0 row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 eol-char : eol-char/c = #\newline raise-if-empty : boolean? = #t truncate-ragged-lines : boolean? = #f decimal-comma : boolean? = #f glob : boolean? = #t
procedure
(lazyframe-scan-csv path [ #:has-header has-header #:new-columns new-columns #:separator separator #:comment-prefix comment-prefix #:quote-char quote-char #:skip-rows skip-rows #:skip-lines skip-lines #:schema-overrides schema-overrides #:null-values null-values #:missing-utf8-is-empty-string missing-utf8-is-empty-string #:ignore-errors ignore-errors #:try-parse-dates try-parse-dates #:infer-schema infer-schema #:infer-schema-length infer-schema-length #:n-rows n-rows #:encoding encoding #:skip-rows-after-header skip-rows-after-header #:row-index-name row-index-name #:row-index-offset row-index-offset #:eol-char eol-char #:raise-if-empty raise-if-empty #:truncate-ragged-lines truncate-ragged-lines #:decimal-comma decimal-comma #:glob glob]) → LazyFrame-ptr? path : path-string? has-header : boolean? = #t new-columns : (and/c (listof column-name/c) distinct?) = '() separator : (or/c csv-char/c #f) = #f
comment-prefix : (or/c (and/c non-empty-string? c-string/c) #f) = #f quote-char : (or/c csv-char/c #f) = #\" skip-rows : size/c = 0 skip-lines : size/c = 0
schema-overrides : (and/c (listof (cons/c column-name/c csv-dtype/c)) distinct-names?) = '()
null-values :
(or/c c-string/c (listof c-string/c) (and/c (listof (cons/c column-name/c c-string/c)) distinct-names?) #f) = #f missing-utf8-is-empty-string : boolean? = #f ignore-errors : boolean? = #f try-parse-dates : boolean? = #f infer-schema : boolean? = #t infer-schema-length : (or/c size/c #f) = 100 n-rows : (or/c size/c #f) = #f encoding : (or/c 'utf8 'utf8-lossy) = 'utf8 skip-rows-after-header : size/c = 0 row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 eol-char : eol-char/c = #\newline raise-if-empty : boolean? = #t truncate-ragged-lines : boolean? = #f decimal-comma : boolean? = #f glob : boolean? = #t
procedure
(dataframe-write-csv frame path [ #:include-bom include-bom #:include-header include-header #:separator separator #:line-terminator line-terminator #:quote-char quote-char #:batch-size batch-size #:datetime-format datetime-format #:date-format date-format #:time-format time-format #:float-scientific float-scientific #:float-precision float-precision #:decimal-comma decimal-comma #:null-value null-value #:quote-style quote-style]) → void? frame : DataFrame-ptr? path : path-string? include-bom : boolean? = #f include-header : boolean? = #t separator : csv-char/c = #\, line-terminator : c-string/c = "\n" quote-char : csv-char/c = #\" batch-size : positive-size/c = 1024 datetime-format : (or/c c-string/c #f) = #f date-format : (or/c c-string/c #f) = #f time-format : (or/c c-string/c #f) = #f float-scientific : (or/c 'auto boolean?) = 'auto float-precision : (or/c float-precision/c #f) = #f decimal-comma : boolean? = #f null-value : c-string/c = ""
quote-style : (or/c 'necessary 'always 'non-numeric 'never) = 'necessary
> (dataframe-height (dataframe-read-csv "flights.tsv" #:separator #\tab #:null-values "NA")) 102
> (dataframe-height (lazyframe-collect (lazyframe-scan-csv "parts/*.csv"))) 6
> (dataframe-read-csv "parts/part-1.csv" #:columns '("origin" "gate")) dataframe-read-csv: failed to read csv from
parts/part-1.csv: not found: unable to find column "gate";
valid columns: ["origin", "dest", "dep_delay"]
> (dataframe-write-csv (dataframe-read-csv "parts/part-1.csv") "/no/such/dir/out.csv" #:separator #\;) dataframe-write-csv: failed to write csv to
/no/such/dir/out.csv: cannot create file: No such file or
directory (os error 2)
procedure
(dataframe-read-json path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length]) → DataFrame-ptr? path : path-string? schema : (or/c json-schema/c #f) = #f schema-overrides : json-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100
procedure
(dataframe-write-json d path) → void?
d : DataFrame-ptr? path : path-string?
> (dataframe-width (dataframe-read-json "stations.json" #:schema '(("station" . str)))) 1
> (dataframe-read-json "stations.json" #:infer-schema-length 0) dataframe-read-json: contract violation
expected: (or/c #f positive-size/c)
given: 0
in: the #:infer-schema-length argument of
(->*
(path-string?)
(#:infer-schema-length
(or/c #f positive-size/c)
#:schema
(or/c #f json-schema/c)
#:schema-overrides
json-schema/c)
DataFrame-ptr?)
contract from:
<pkgs>/polars/private/json.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/json.rkt:26:11
procedure
(dataframe-read-json-lines path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length #:batch-size batch-size #:n-rows n-rows #:low-memory low-memory #:rechunk rechunk #:row-index-name row-index-name #:row-index-offset row-index-offset #:ignore-errors ignore-errors #:include-file-paths include-file-paths]) → DataFrame-ptr? path : path-string? schema : (or/c ndjson-schema/c #f) = #f schema-overrides : ndjson-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100 batch-size : (or/c positive-size/c #f) = 1024 n-rows : (or/c size/c #f) = #f low-memory : boolean? = #f rechunk : boolean? = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 ignore-errors : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f
procedure
(lazyframe-scan-json-lines path [ #:schema schema #:schema-overrides schema-overrides #:infer-schema-length infer-schema-length #:batch-size batch-size #:n-rows n-rows #:low-memory low-memory #:rechunk rechunk #:row-index-name row-index-name #:row-index-offset row-index-offset #:ignore-errors ignore-errors #:include-file-paths include-file-paths]) → LazyFrame-ptr? path : path-string? schema : (or/c ndjson-schema/c #f) = #f schema-overrides : ndjson-schema/c = '() infer-schema-length : (or/c positive-size/c #f) = 100 batch-size : (or/c positive-size/c #f) = 1024 n-rows : (or/c size/c #f) = #f low-memory : boolean? = #f rechunk : boolean? = #f row-index-name : (or/c column-name/c #f) = #f row-index-offset : row-index/c = 0 ignore-errors : boolean? = #f include-file-paths : (or/c column-name/c #f) = #f
procedure
(dataframe-write-json-lines d path [ #:compression compression #:compression-level compression-level #:check-extension check-extension]) → void? d : DataFrame-ptr? path : path-string? compression : (or/c 'uncompressed 'gzip 'zstd) = 'uncompressed compression-level : (or/c exact-nonnegative-integer? #f) = #f check-extension : boolean? = #t
> (dataframe-height (dataframe-read-json-lines "stations.ndjson" #:n-rows 3)) 3
> (dataframe-width (lazyframe-collect (lazyframe-scan-json-lines "stations.ndjson" #:infer-schema-length 2))) 4
> (dataframe-read-json-lines "stations.ndjson" #:schema '(("reading" . i8))) dataframe-read-json-lines: contract violation
expected: (or/c ndjson-schema/c #f)
given: '(("reading" . i8))
in: the schema argument of
(->i
((path path-string?))
(#:schema
(schema (or/c ndjson-schema/c #f))
#:schema-overrides
(schema-overrides ndjson-schema/c)
#:infer-schema-length
(infer-schema-length
(or/c positive-size/c #f))
#:batch-size
(batch-size (or/c positive-size/c #f))
#:n-rows
(n-rows (or/c size/c #f))
#:low-memory
(low-memory boolean?)
#:rechunk
(rechunk boolean?)
#:row-index-name
(row-index-name (or/c column-name/c #f))
#:row-index-offset
(row-index-offset row-index/c)
#:ignore-errors
(ignore-errors boolean?)
#:include-file-paths
(include-file-paths
(or/c column-name/c #f)))
#:pre/name
(row-index-name include-file-paths)
"#:row-index-name and #:include-file-paths must name
different columns"
(distinct-columns?
row-index-name
include-file-paths)
(result DataFrame-ptr?))
contract from:
<pkgs>/polars/private/json.rkt
blaming: top-level
(assuming the contract is correct)
at: <pkgs>/polars/private/json.rkt:28:11
2.4 Lazy frames
A lazyframe is a query plan: a sequence of operations over a frame that Polars optimises as a whole and runs only when asked to collect. The low-level surface mirrors the eager dataframe-* bindings and, like them, returns raw foreign pointers; the fluent lazy and collect are the wrapper-returning equivalents.
procedure
(LazyFrame-ptr? v) → boolean?
v : any/c
procedure
(lazyframe? v) → boolean?
v : any/c
procedure
(lazyframe-drop lf) → void?
lf : LazyFrame-ptr?
> (define query (lazy (dataframe (list (series '(3 1 2) #:name "v"))))) > (lazyframe-drop query) > query #<lazyframe: dropped>
> (collect query) lazyframe: lazyframe has been dropped
> (lazyframe-drop query) lazyframe-drop: lazyframe has been dropped
procedure
(dataframe-lazy df) → LazyFrame-ptr?
df : DataFrame-ptr?
procedure
(lazyframe-collect lf) → DataFrame-ptr?
lf : LazyFrame-ptr?
procedure
(lazyframe-explain lf [ #:optimized optimized #:format format]) → string? lf : LazyFrame-ptr? optimized : boolean? = #t format : (or/c 'plain 'tree) = 'plain
> (displayln (lazyframe-explain (dataframe-lazy (dataframe-read-parquet "produce.parquet")))) DF ["item", "grade", "price"]; PROJECT */3 COLUMNS
> (lazyframe-explain (lazyframe-select (dataframe-lazy (dataframe-read-parquet "produce.parquet")) (list (col "colour")))) lazyframe-explain: failed to explain the query: not found:
unable to find column "colour"; valid columns: ["item",
"grade", "price"]
Resolved plan until failure:
---> FAILED HERE RESOLVING 'select' <---
DF ["item", "grade", "price"]; PROJECT */3 COLUMNS: 'select'
procedure
(lazyframe-select lf exprs) → LazyFrame-ptr?
lf : LazyFrame-ptr? exprs : (listof Expr-ptr?)
procedure
(lazyframe-with-columns lf exprs) → LazyFrame-ptr?
lf : LazyFrame-ptr? exprs : (listof Expr-ptr?)
procedure
(lazyframe-filter lf predicate) → LazyFrame-ptr?
lf : LazyFrame-ptr? predicate : Expr-ptr?
procedure
(lazyframe-group-by-agg lf keys aggs) → LazyFrame-ptr?
lf : LazyFrame-ptr? keys : (listof (or/c string? Expr-ptr?)) aggs : (listof Expr-ptr?)
procedure
(lazyframe-sort lf names [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → LazyFrame-ptr? lf : LazyFrame-ptr? names : (non-empty-listof string?) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
procedure
(lazyframe-join left right [ #:on on #:left-on left-on #:right-on right-on #:how how]) → LazyFrame-ptr? left : LazyFrame-ptr? right : LazyFrame-ptr? on : (or/c #f (non-empty-listof string?)) = #f left-on : (or/c #f (non-empty-listof string?)) = #f right-on : (or/c #f (non-empty-listof string?)) = #f
how : (or/c 'inner 'left 'outer 'full 'cross 'semi 'anti) = 'inner
(lazyframe-collect (lazyframe-join (dataframe-lazy users) (dataframe-lazy orders) #:on '("uid") #:how 'inner))
2.5 Low-level expression API
The monomorphic expr-* layer that the operators in
Operators and pipelines are built from. You rarely need these names directly:
expr-gt underlies >, expr-add underlies
+, expr-sum underlies the expression arm of sum.
Reach for them when a generic name is shadowed in your module, or when you
want to be explicit that an expression —
procedure
(expr-alias e name) → Expr-ptr?
e : Expr-ptr? name : string?
procedure
procedure
(expr-exclude e names) → Expr-ptr?
e : multi-column-expr? names : (non-empty-listof (or/c string? regexp?))
procedure
(expr-dtype-col dtype) → Expr-ptr?
dtype : dtype-spec?
> (expr-all) cs.all()
> (expr-exclude (expr-all) (list "id" #rx"^w")) [cs.all() - [cs.matches("^(?s).*(?:^w).*$") | cs.by_name('id', require_all=false)]]
> (expr-dtype-col 'f64) cs.by_dtype([Float64])
> (select people (expr-exclude (expr-dtype-col 'float64) (list "height")))
shape: (3, 1)
┌────────┐
│ weight │
│ --- │
│ f64 │
╞════════╡
│ 57.9 │
│ 72.5 │
│ 53.6 │
└────────┘
procedure
(expr-meta-output-name e) → string?
e : Expr-ptr?
procedure
(expr-meta-root-names e) → (listof string?)
e : Expr-ptr?
procedure
(expr-meta-eq? a b) → boolean?
a : Expr-ptr? b : Expr-ptr?
> (~> (col "a") (expr-alias "b") expr-meta-output-name) "b"
> (~> (col "a") (expr-add (col "b")) expr-meta-root-names) '("a" "b")
> (expr-meta-eq? (col "a") (expr-col "a")) #t
> (~> (col "v") expr-sum (expr-over (list "k" (col "h")))) col("v").sum().over([col("k"), col("h")])
> (~> khv (with-columns (~> (col "v") expr-sum (expr-over (list "k")) (expr-alias "total"))))
shape: (4, 4)
┌─────┬─────┬─────┬───────┐
│ k ┆ h ┆ v ┆ total │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═══════╡
│ a ┆ x ┆ 1 ┆ 6 │
│ a ┆ y ┆ 2 ┆ 6 │
│ a ┆ x ┆ 3 ┆ 6 │
│ b ┆ x ┆ 4 ┆ 4 │
└─────┴─────┴─────┴───────┘
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
> (~> (dataframe (list (series '(1 -2 3) #:name "x"))) (select (expr-neg (col "x"))))
shape: (3, 1)
┌─────┐
│ x │
│ --- │
│ i64 │
╞═════╡
│ -1 │
│ 2 │
│ -3 │
└─────┘
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
a : any/c b : any/c
procedure
e : Expr-ptr?
procedure
e : Expr-ptr?
procedure
e : Expr-ptr?
procedure
e : Expr-ptr?
procedure
e : Expr-ptr?
procedure
(expr-median e) → Expr-ptr?
e : Expr-ptr?
procedure
(expr-count e) → Expr-ptr?
e : Expr-ptr?
procedure
(expr-n-unique e) → Expr-ptr?
e : Expr-ptr?
procedure
(expr-first e) → Expr-ptr?
e : Expr-ptr?
procedure
e : Expr-ptr?
procedure
e : Expr-ptr? ddof : exact-nonnegative-integer? = 1
procedure
e : Expr-ptr? ddof : exact-nonnegative-integer? = 1
procedure
(expr-sort e [ #:descending descending #:nulls-last nulls-last]) → Expr-ptr? e : Expr-ptr? descending : boolean? = #f nulls-last : boolean? = #f
procedure
(expr-sort-by e #:by by [ #:descending descending #:nulls-last nulls-last #:maintain-order maintain-order]) → Expr-ptr? e : Expr-ptr?
by :
(or/c string? Expr-ptr? (non-empty-listof (or/c string? Expr-ptr?))) descending : (or/c boolean? (listof boolean?)) = #f nulls-last : (or/c boolean? (listof boolean?)) = #f maintain-order : boolean? = #f
2.5.1 Eager expression contexts
These run expressions against a dataframe and return a new frame in one step. Each is the eager convenience over the corresponding lazy operation in Lazy frames: it converts with dataframe-lazy, applies the operation, and lazyframe-collects. Like the rest of the low-level layer they accept a dataframe wrapper but return a raw DataFrame-ptr?; the fluent select, with-columns, filter and group-by/agg are the wrapper-returning equivalents.
procedure
(dataframe-select-exprs df exprs) → DataFrame-ptr?
df : DataFrame-ptr? exprs : (listof Expr-ptr?)
procedure
(dataframe-with-columns df exprs) → DataFrame-ptr?
df : DataFrame-ptr? exprs : (listof Expr-ptr?)
procedure
(dataframe-filter-expr df predicate) → DataFrame-ptr?
df : DataFrame-ptr? predicate : Expr-ptr?
procedure
(dataframe-group-by-agg df keys aggs) → DataFrame-ptr?
df : DataFrame-ptr? keys : (listof (or/c string? Expr-ptr?)) aggs : (listof Expr-ptr?)
2.6 Generic interfaces
The high-level operations are small, purpose-named racket/generic interfaces. A wrapper implements the interface for each capability it has — a series and a dataframe both have a len and a shape, so both implement gen:sized and gen:has-shape; only a series has a dtype. Each interface exports its method(s) and a predicate that recognises values implementing it.
syntax
procedure
v : any/c
syntax
procedure
(has-shape? v) → boolean?
v : any/c
syntax
procedure
(has-dtype? v) → boolean?
v : any/c
syntax
procedure
(has-null-count? v) → boolean?
v : any/c
A series is also a Racket sequence (through prop:sequence): a for clause, sequence? and the racket/sequence operations see its elements, converted a block of rows at a time as by in-series, with polars-null for a null entry. A dataframe is not a sequence; iterate over its rows with in-dataframe-rows or its columns with in-dataframe-columns, or convert them with dataframe->rows or dataframe->columns.
> (define ages (series (list 34 polars-null 51) #:name "age")) > (sequence? ages) #t
> (for/list ([age ages]) age) '(34 polars-null 51)
> (for/sum ([age ages] #:unless (polars-null? age)) age) 85
> (sequence? (dataframe (list ages))) #f