1 Guide
This guide starts with a dataframe and then shows how to select columns, choose another representation, and interpret missing values and metadata. If you already know which procedure you need, turn to the Reference.
1.1 Installation and a first dataset
Use Racket 9.3 or later. Install from the Racket catalog:
raco pkg install --auto datasets
This also installs datasets-core, the data packages, and the adapter dependencies; to install from a checkout, follow the repository’s README.
The top-level directories are independently installable multi-collection packages. datasets-core provides datasets/core; datasets-r adds R’s datasets package (see Datasets from R) and datasets from CRAN packages, and datasets-uci datasets from the UCI Machine Learning Repository, to the core catalog; datasets adds the convenient loaders and adapters to the same collection. The Polars package supplies its native library. With Nix, nix develop prepares an isolated environment containing the packages and that library. datasets-islr adds the datasets of ISLR2 and four of ESL; see Datasets from ISLR2 and Datasets from ESL. datasets-fetch adds sets such as covtype, mnist and cifar-10 that are downloaded and converted on first load.
> (require datasets (prefix-in pl: polars)) > (define iris (load-iris)) > (pl:dataframe-height iris) 150
> (pl:dataframe-column-names iris) '("sepal-length" "sepal-width" "petal-length" "petal-width" "species")
The prefix keeps Polars operations distinct from Racket’s arithmetic and other similarly named procedures. load-iris returns 150 rows: four flower measurements and a species label. Each call creates an independent dataframe. Iris is bundled with the package, so loading it does not download anything.
1.2 Discovering datasets and selecting columns
dataset-names lists the available identifiers. A named loader, such as load-mtcars, is shorthand for (load-dataset 'mtcars). Both accept #:columns and #:format.
> (dataset-names)
'(abalone
ability-cov
adult
air-passengers
airmiles
airquality
anscombe
attenu
attitude
austres
auto
auto-mpg
bank-marketing
banknote-authentication
beaver1
beaver2
bike-sharing-daily
bike-sharing-hourly
bikeshare
bj-sales
bod
boston
brain-cancer
breast-cancer
california-housing
car-evaluation
caravan
cars
carseats
chick-weight
chickwts
cifar-10
cifar-100
co2
co2-uptake
college
concrete-strength
covtype
credit
crimtab
datasaurus-dozen
default
diabetes
diamonds
digits
discoveries
dnase
economics
economics-long
emnist-balanced
emnist-digits
emnist-letters
energy-efficiency
esoph
eu-stock-markets
euro
eurodist
faithful
faithfuld
fashion-mnist
formaldehyde
freeny
fund
gapminder
hair-eye-color
harman23-cor
harman74-cor
heart-disease
hitters
indometh
infert
insect-sprays
iris
islands
johnson-johnson
khan
kmnist
lake-huron
lh
life-cycle-savings
linnerud
loblolly
longley
luv-colours
lynx
midwest
mnist
morley
mpg
msleep
mtcars
mushroom
nci60
nhtemp
nile
nottem
npk
nycflights13-airlines
nycflights13-airports
nycflights13-flights
nycflights13-planes
nycflights13-weather
nyse
occupational-status
oj
optdigits
orange
orchard-sprays
ozone
penguins
penguins-raw
plant-growth
portfolio
precip
presidential
presidents
pressure
prostate
publication
puromycin
quakes
randu
rivers
rock
sa-heart
seals
seatbelts
seeds
sleep
smarket
spambase
stackloss
state
student-performance
sunspot-month
sunspot-year
sunspots
swiss
theoph
titanic
titanic-passengers
tooth-growth
treering
trees
txhousing
ucb-admissions
uk-driver-deaths
uk-gas
uk-lung-deaths
us-acc-deaths
us-arrests
us-cities-d
us-judge-ratings
us-personal-expenditure
uspop
va-deaths
volcano
vowel
wage
warpbreaks
weekly
wine
wine-quality
women
world-phones
www-usage
zip)
> (define cars (load-dataset 'mtcars #:columns '("model" "mpg"))) > (pl:dataframe-column-names cars) '("model" "mpg")
> (hash-ref (dataset-info 'mtcars) "identifiers") '("model")
Column names are strings. A selection preserves the order you give and retains all source rows. Identifiers such as "model" are real columns, so they can be kept for display or left out of a numeric matrix. Duplicate names, empty selections, and unknown columns are reported as errors.
1.3 A first look at the Palmer penguins
Datasets from datasets-r have no named loader; load them by name. The 'penguins set records 344 penguins of three species near Palmer Station, Antarctica:
> (define penguins (load-dataset 'penguins)) > (pl:head penguins 4)
shape: (4, 8)
┌─────────┬───────────┬───────────────┬───────────────┬──────────────┬─────────────┬────────┬──────┐
│ species ┆ island ┆ bill-length-m ┆ bill-depth-mm ┆ flipper-leng ┆ body-mass-g ┆ sex ┆ year │
│ --- ┆ --- ┆ m ┆ --- ┆ th-mm ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ --- ┆ f64 ┆ --- ┆ i64 ┆ str ┆ i64 │
│ ┆ ┆ f64 ┆ ┆ i64 ┆ ┆ ┆ │
╞═════════╪═══════════╪═══════════════╪═══════════════╪══════════════╪═════════════╪════════╪══════╡
│ Adelie ┆ Torgersen ┆ 39.1 ┆ 18.7 ┆ 181 ┆ 3750 ┆ male ┆ 2007 │
│ Adelie ┆ Torgersen ┆ 39.5 ┆ 17.4 ┆ 186 ┆ 3800 ┆ female ┆ 2007 │
│ Adelie ┆ Torgersen ┆ 40.3 ┆ 18.0 ┆ 195 ┆ 3250 ┆ female ┆ 2007 │
│ Adelie ┆ Torgersen ┆ null ┆ null ┆ null ┆ null ┆ null ┆ 2007 │
└─────────┴───────────┴───────────────┴───────────────┴──────────────┴─────────────┴────────┴──────┘
> (hash-ref (dataset-info 'penguins) "suggested-targets") '("species")
Missing measurements are null, and each species’ mean body mass skips them:
> (define by-species (pl:~> penguins (pl:group-by "species") (pl:agg (pl:mean (pl:col "body-mass-g"))) (pl:sort '("species"))))
> (for/list ([row 3]) (list (pl:series-ref (pl:dataframe-column by-species "species") row) (pl:series-ref (pl:dataframe-column by-species "body-mass-g") row)))
'(("Adelie" 3700.662251655629)
("Chinstrap" 3733.0882352941176)
("Gentoo" 5076.016260162602))
Datasets from datasets-uci also load by name. The 'digits set is scikit-learn’s handwritten digits: per image, 64 pixel counts and the digit:
> (pl:shape (load-dataset 'digits)) '(1797 65)
> (hash-ref (dataset-info 'digits) "suggested-targets") '("digit")
1.4 Grouping categorical data
Category labels remain strings in Polars. For example, Iris species labels can be used directly in a group operation:
> (define by-species (pl:~> iris (pl:group-by "species") (pl:agg (pl:mean (pl:col "sepal-length"))))) > (pl:dataframe-height by-species) 3
> (pl:dataframe-column-names by-species) '("species" "sepal-length")
The result has one row for each species. Grouped output need not follow category order; the intended level order is recorded separately in metadata:
> (define species-info (list-ref (hash-ref (dataset-info 'iris) "columns") 4)) > (hash-ref species-info "levels") '("Iris-setosa" "Iris-versicolor" "Iris-virginica")
Keeping labels avoids introducing arbitrary numeric codes. See Value representations for the representation of each value type.
1.5 Handling missing observations
Airquality has missing measurements. The table format uses dataset-missing; the Polars adapter translates that singleton to pl:polars-null.
> (define air (load-airquality)) > (pl:polars-null? (pl:series-ref (pl:dataframe-column air "ozone") 4)) #t
> (define air-table (load-airquality #:format 'table)) > (define ozone (dataset-table-column air-table "ozone"))
> (for/sum ([value (in-vector ozone)]) (if (dataset-missing? value) 1 0)) 37
Missing values are distinct from zero and #f. Decide whether to exclude, replace, or otherwise account for them in your analysis. The loaders preserve them; matrix conversion reports a missing value instead of silently removing it.
1.6 Datasets from ISLR2
datasets-islr bundles the 20 datasets of the R package ISLR2, the data of An Introduction to Statistical Learning, Second Edition. They load by name. Player names, R row names in the book, are the first column of 'hitters:
> (define hitters (load-dataset 'hitters #:format 'table)) > (dataset-table-row-count hitters) 322
> (vector-ref (dataset-table-column hitters "player") 1) "-Alan Ashby"
> (hash-ref (dataset-info 'hitters) "suggested-targets") '("salary")
> (for/sum ([salary (in-vector (dataset-table-column hitters "salary"))]) (if (dataset-missing? salary) 1 0)) 59
The islr2-chapters key of each ISLR2 dataset’s metadata lists the chapters whose text and labs, and whose exercises, use it:
> (hash-ref (dataset-info 'auto) "islr2-chapters") '#hash(("exercises" . (2 3 4 7 9)) ("text" . (2 3 5)))
The table below is built from those keys. Chapter 12 also uses 'us-arrests, from datasets-core. The ISLR2 and ESL Datasets Catalog describes the columns and provenance of each dataset.
Chapter |
| Text and labs |
| Exercises |
1 Introduction |
| ’nci60, ’smarket, ’wage |
| |
2 Statistical Learning |
| ’auto |
| ’auto, ’boston, ’college |
3 Linear Regression |
| ’auto, ’boston, ’carseats, ’credit |
| ’auto, ’boston, ’carseats |
4 Classification |
| ’bikeshare, ’caravan, ’default, ’smarket |
| ’auto, ’boston, ’weekly |
5 Resampling Methods |
| ’auto, ’portfolio |
| ’boston, ’default, ’weekly |
6 Linear Model Selection and Regularization |
| ’credit, ’hitters |
| ’boston, ’college |
7 Moving Beyond Linearity |
| ’wage |
| ’auto, ’boston, ’college, ’wage |
8 Tree-Based Methods |
| ’boston, ’carseats, ’hitters |
| ’boston, ’caravan, ’carseats, ’hitters, ’oj |
9 Support Vector Machines |
| ’khan |
| ’auto, ’oj |
10 Deep Learning |
| ’hitters, ’nyse |
| ’default, ’nyse |
11 Survival Analysis |
| ’brain-cancer, ’publication |
| ’brain-cancer |
12 Unsupervised Learning |
| ’nci60 |
| ’boston |
13 Multiple Testing |
| ’fund, ’khan |
| ’carseats |
1.7 Datasets from ESL
datasets-islr also bundles 'prostate, 'sa-heart, 'vowel and 'ozone from The Elements of Statistical Learning. The ZIP code digits, 'zip, are in datasets-fetch and are downloaded on first load. A train/test partition is a final split column:
> (define vowel (load-dataset 'vowel #:format 'table)) > (dataset-table-names vowel) '("y" "x1" "x2" "x3" "x4" "x5" "x6" "x7" "x8" "x9" "x10" "split")
> (hash-ref (dataset-info 'zip) "tier") "fetch"
ESL’s spam data is 'spambase in datasets-uci.
1.8 Image datasets
datasets-fetch lists 'cifar-10, 'cifar-100, 'kmnist, 'emnist-balanced, 'emnist-letters and 'emnist-digits beside 'mnist and 'fashion-mnist. They are fetch tier: each is downloaded and converted on first load. A row is one image: its pixel columns, then its label column or columns, then split.
An image entry’s image-shape is (channels height width); the pixel columns in order reshape row-major to it, as torch’s per-image CHW.
> (hash-ref (dataset-info 'cifar-10) "image-shape") '(3 32 32)
> (take (map (lambda (c) (hash-ref c "name")) (hash-ref (dataset-info 'cifar-10) "columns")) 2) '("red1" "red2")
> (hash-ref (dataset-info 'cifar-10) "tier") "fetch"
1.9 Preparing a numeric matrix
Choose 'matrix for a math/matrix matrix. Rows are observations, and the requested column order determines the matrix’s columns. For Iris, select the four measurements explicitly to leave out the species label:
> (require math/matrix)
> (define measurements (load-iris #:format 'matrix #:columns '("sepal-length" "sepal-width" "petal-length" "petal-width"))) > (matrix-num-rows measurements) 150
> (matrix-num-cols measurements) 4
> (matrix-ref measurements 0 0) 5.1
Trying to include a label or a missing observation produces an error naming the column and the zero-based row:
> (load-iris #:format 'matrix #:columns '("species")) table->matrix: requires numeric, nonmissing values
column: "species"
row (zero-based): 0
value: "Iris-setosa"
> (load-airquality #:format 'matrix #:columns '("ozone")) table->matrix: requires numeric, nonmissing values
column: "ozone"
row (zero-based): 4
value: #<missing-value>
Diabetes already has ten standardized predictors in the lars representation. See its entry in the Dataset Catalog and Provenance before comparing it with another library’s dataset; raw measurements and exact scikit-learn parity are not promised.
1.10 Using mutable data-frame objects
The 'data-frame format works with the Racket data-frame library. Its NA value is explicitly set to dataset-missing.
> (require (prefix-in df: data-frame) datasets/data-frame)
> (define frame (load-airquality #:format 'data-frame #:columns '("ozone" "temp"))) > (data-frame-column-names frame) '("ozone" "temp")
> (df:df-is-na? frame "ozone" (df:df-ref frame 4 "ozone")) #t
The underlying library stores columns by name and does not promise an order from df:df-series-names. data-frame-column-names retrieves the initial selection order recorded by this adapter.
Changes to one frame do not affect a later load:
1.11 Interpreting counts and time series
Titanic’s rows are contingency cells, not individual passengers. Sum the "frequency" column to count people, and retain zero-frequency rows when you need the complete table of combinations:
> (define counts (load-titanic #:format 'table)) > (dataset-table-row-count counts) 32
> (for/sum ([count (in-vector (dataset-table-column counts "frequency"))]) count) 2201
AirPassengers contains monthly international airline passenger counts, measured in thousands. Its year and month columns make the chronology explicit:
> (define monthly (load-air-passengers #:format 'matrix)) > (for/list ([column (in-range 3)]) (matrix-ref monthly 0 column)) '(1949 1 112)
> (for/list ([column (in-range 3)]) (matrix-ref monthly 143 column)) '(1960 12 432)
> (hash-ref (dataset-info 'air-passengers) "time-series")
'#hash(("end" . (1960 12))
("frequency" . 12)
("index" . ("year" "month"))
("start" . (1949 1)))
The same metadata interface records source citations, licenses, units, identifiers, and file checksums for every dataset. The catalog is generated from that registry rather than maintained as a second independent list.
1.12 Generating synthetic data
The scikit-learn generators are available under the same names with a #:format keyword. A fixed #:seed makes the result reproducible. Two interleaving moons, plotted by label:
> (require plot/pict) > (define moons (make-moons #:n-samples 200 #:noise 0.1 #:seed 0 #:format 'table))
> (define (moon label) (for/list ([x (in-vector (dataset-table-column moons "x0"))] [y (in-vector (dataset-table-column moons "x1"))] [l (in-vector (dataset-table-column moons "y"))] #:when (= l label)) (vector x y)))
> (plot (list (points (moon 0) #:color 1 #:label "y = 0") (points (moon 1) #:color 2 #:label "y = 1")) #:x-label "x0" #:y-label "x1")
> (pl:dataframe-column-names (make-moons #:seed 0)) '("x0" "x1" "y")
Each table’s metadata records the arguments and seed that produced it. See Synthetic Data for the generators and their seeds.
1.13 Using core data in another package
A package that constructs its own dataframes can depend on datasets-core and use datasets/core directly. This is particularly useful for Polars documentation: depending only on the core avoids a dependency cycle through datasets.
The following example uses core data with Polars’ constructors:
> (require datasets/core) > (define table (load-dataset-table 'iris #:columns '("sepal-length")))
> (define own-frame (pl:dataframe (list (pl:series (dataset-table-column table "sepal-length") #:name "sepal-length")))) > (pl:dataframe-height own-frame) 150
The repository’s examples/core-with-polars.rkt demonstrates the same pattern and is tested before the adapter package is installed. For downstream Nix, fetch this repository with flake = false and install datasets-core/ from that source. This keeps the complete flake and its native dependencies out of the downstream dependency graph.