2 Reference

9.3

Racket

2 Reference🔗ℹ

This section specifies the core API. For a step-by-step introduction, see the Guide. Contracts below describe accepted arguments and results using the notation of The Racket Reference.

2.1 Registry and loading🔗ℹ

Procedures in this module that discover registries or take a dataset name (dataset-names, dataset-packages, dataset-info, load-dataset-table, fetch-dataset, dataset-cached? and clear-dataset-cache) are not thread-safe: concurrent calls can silently omit the datasets of other packages. Callers using several threads must serialize these calls.

procedure
(dataset-names [#:package package]) → (listof symbol?)
package : (or/c #f string?) = #f

Returns the identifiers of every dataset declared by an installed registry, sorted alphabetically, including fetch-tier datasets. If package is a string, returns only that package’s datasets; a package that declares no installed dataset raises exn:fail:contract listing the available packages. Conflicting or malformed registries raise exn:fail; see Registries.

Example:

> (dataset-names #:package "datasets-core")
'(air-passengers
  airquality
  anscombe
  breast-cancer
  diabetes
  faithful
  iris
  mtcars
  plant-growth
  titanic
  tooth-growth
  us-arrests
  wine)

procedure
(dataset-packages) → (listof string?)

Returns the names of the packages that declare at least one installed dataset, sorted by string<?. Every identifier in (dataset-names) belongs to exactly one of them, recorded in its metadata as package.

Examples:

> (dataset-packages)
'("datasets-core")
> (hash-ref (dataset-info 'iris) "package")
"datasets-core"

procedure
(dataset-info name) → (and/c hash? immutable?)
name : symbol?

Returns deeply immutable metadata for name. An unknown identifier raises exn:fail:contract. Metadata describes the complete source dataset; it is independent of any column selection. See Metadata fields for its fields.

procedure
(load-dataset-table name
[ #:columns columns
#:download? download?]) → dataset-table?
  name : symbol?
  columns : (or/c #f (non-empty-listof string?)) = #f
  download? : boolean? = #t

Reads the observations for name and returns a table. Bundled data files are resolved relative to the installed module and read when this procedure is called; the current directory does not affect loading.

If columns is #f, all columns are returned in source order. Otherwise, it must be a nonempty list of distinct known column names. Its order determines the table’s column order. Row order and row count are preserved. Unknown datasets, unknown columns, empty selections, duplicate columns, and invalid argument types raise exn:fail:contract.

The result contains immutable column vectors. Strings inside those vectors are also immutable. Missing observations are dataset-missing. Repeated loads are equal; callers should not rely on object identity.

Bundled data is stored gzip-compressed and decompressed on each call; a fetch-tier dataset is read from (dataset-cache-directory) in the same form, after downloading it as by fetch-dataset if it is not cached. Column arguments are checked before any download. If the dataset is not cached and download? is #f, raises exn:fail naming the dataset and how to fetch it (see Disabling downloads). Every load checks the data’s SHA-256 against the sha256 metadata before parsing, for bundled and cached data alike; a mismatch or an unreadable file raises exn:fail naming the file. For a cached file that cannot be read or is not the pinned download, the error gives clear-dataset-cache as the remedy. For a pinned, readable one, and for any data that matches its SHA-256 but does not parse, it names the dataset and package instead: the registry entry is wrong.

Examples:

> (dataset-table-names
   (load-dataset-table 'mtcars #:columns '("mpg" "model")))
'("mpg" "model")
> (load-dataset-table 'iris #:columns '("unknown"))
load-dataset-table: unknown column
  column: "unknown"
  available: '("sepal-length" "sepal-width" "petal-length"
"petal-width" "species")

2.1.1 Registries🔗ℹ

A registry is a file mapping dataset names to metadata hashes. Any installed package can declare registries with the datasets-registries key in a collection-level info.rkt: a list of paths relative to that file’s directory. Each dataset’s data is data/name.rktd.gz beside its registry. The registries of all installed packages are merged when first used. Each later call checks the registry files and declaring info.rkt files of the packages already found, and merges again when one changes. The installed package index is read again when the collection search path changes, as with current-library-collection-paths, when raco setup runs in the same process, or after a call whose discovery failed or was interrupted, so a package installed after the first call is usually not seen until the process restarts. Package datasets-core is reserved for the registries that datasets/core-data/info.rkt lists. Another installation’s datasets/core-data collection is skipped unread; any other registry declaring datasets-core is an error naming its file. A name declared twice is an error naming both packages. A malformed registry is an error naming its file. Either error makes every registry operation raise until the package at fault is fixed or removed. A fetch-tier entry has no data file; see Downloads and the cache.

When reading the installed package index fails with a permission error, or a security guard denies it as in a default racket/sandbox evaluator, only the datasets-core registries are used: a warning is logged to the datasets topic, an unknown dataset or package error gives the reason, and the next call tries discovery again. Any other failure to read the index raises.

2.2 Downloads and the cache🔗ℹ

Fetch-tier datasets are not stored in any package. Their data is downloaded once into the cache directory and read from there. A cached file is named by the SHA-256 its registry entry pins, so a registry update that changes the data downloads the new file rather than reusing the old one.

procedure
(dataset-cache-directory) → complete-path?

Returns the directory holding downloaded datasets: the value of DATASETS_HOME when it is set and nonempty, otherwise (build-path (find-system-path 'cache-dir) "racket-datasets"). The variable is read on every call. A relative value makes this procedure, fetch-dataset, dataset-cached?, clear-dataset-cache, and loading a fetch-tier dataset raise exn:fail; loading a bundled dataset is unaffected. The directory is created by the first download, not by this procedure.

Example:

> (parameterize ([current-environment-variables
(make-environment-variables #"DATASETS_HOME" #"/srv/racket-datasets")])
(dataset-cache-directory))
#<path:/srv/racket-datasets>

procedure
(dataset-cached? name) → boolean?
name : symbol?

Returns #t if loading name needs no download: always for a bundled dataset, and for a fetch-tier dataset when its file is in (dataset-cache-directory). Never contacts a server and does not read the file; load-dataset-table checks its SHA-256. An unknown dataset raises exn:fail:contract.

Example:

> (dataset-cached? 'iris)
#t

procedure
(fetch-dataset name
[ #:idle-timeout idle-timeout]) → complete-path?
name : symbol?
idle-timeout : (and/c real? positive?) = 60

Makes name’s data available locally and returns the path of the gzip-compressed .rktd file load-dataset-table reads: the file in its package for a bundled dataset, the cached file for a fetch-tier one. A fetch-tier dataset that is not cached is downloaded from the urls in its fetch metadata, in order, until one succeeds. Each download goes to a temporary file beside its destination and is renamed into place only after its SHA-256 matches the metadata, so an interrupted or failed download never leaves a partial file under the cached name. A URL whose body delivers no data for idle-timeout seconds fails. The idle timeout bounds waits for body data only; connecting, the TLS handshake and waiting for response headers, including redirects, have no timeout. If every URL fails, raises exn:fail listing each URL and its failure, with the expected and actual SHA-256 for a mismatch. Raises exn:fail without contacting a server when the format is not supported; an unknown dataset raises exn:fail:contract.

Example:

> (file-exists? (fetch-dataset 'iris))
#t

procedure
(clear-dataset-cache [name]) → void?
name : (or/c #f symbol?) = #f

Deletes downloaded files from (dataset-cache-directory): those of name, or, when name is #f, those of every dataset, including datasets whose package is no longer installed. Only files the cache created (verified downloads and leftover temporary files) are removed; other files and the directory itself are left alone. A bundled dataset has nothing to delete. An unknown name raises exn:fail:contract. Clearing a dataset while another program downloads it makes that download fail.

2.2.1 Disabling downloads🔗ℹ

A download happens only when a fetch-tier dataset is not cached. #:download? #f disables it for one call to load-dataset-table; a refused download raises exn:fail naming the dataset, the expected cache file and how to fetch it. Bundled and cached datasets load as usual. Every load checks the data’s SHA-256, for both tiers; a corrupt cached file raises exn:fail naming clear-dataset-cache as the remedy.

Downloads are logged at level 'info with topic 'datasets; set PLTSTDERR to info@datasets to see them.

2.3 Table inspection🔗ℹ

procedure
(dataset-table? value) → boolean?
value : any/c

Returns #t if value is a dataset table, #f otherwise. There is no public table constructor or mutator.

procedure
(dataset-table-names table) → (listof string?)
table : dataset-table?

Returns the selected column names in order. Each string is immutable.

procedure
(dataset-table-columns table) → (and/c vector? immutable?)
table : dataset-table?

Returns an immutable vector of immutable column vectors, in the same order as dataset-table-names. Each column has (dataset-table-row-count table) elements.

procedure
(dataset-table-column table name) → (and/c vector? immutable?)
table : dataset-table?
name : string?

Returns the column named name from table. A name that is not among the selected columns raises exn:fail:contract.

procedure
(dataset-table-row-count table) → exact-nonnegative-integer?
table : dataset-table?

Returns the number of source observations, including rows with missing values.

procedure
(dataset-table-info table) → (and/c hash? immutable?)
table : dataset-table?

Returns the full source metadata. Its "column-count" may exceed the number of selected columns in table.

2.4 Missing values🔗ℹ

value
dataset-missing : any/c
procedure
(dataset-missing? value) → boolean?
value : any/c

dataset-missing is the missing-value singleton. The predicate recognizes that value by identity. It returns #f for #f, zero, NaN, strings, and symbols, including 'missing. The singleton is not a number and must be handled explicitly before arithmetic.

2.5 Metadata fields🔗ℹ

All metadata hashes, lists, and strings are immutable. Dataset hashes have these string keys:

Key		Meaning
name		Dataset identifier as a string.
rows		Number of source rows.
column-count		Number of source columns, including identifiers.
columns		Ordered list of column-description hashes.
identifiers		Column names identifying observations; an empty list if none.
suggested-targets		Optional response or label column names; an empty list if unspecified.
source-url		URL of the pinned upstream archive.
source-version		Upstream version or hash-pinned snapshot description.
source-sha256		Hexadecimal SHA-256 of the upstream archive.
sha256		Hexadecimal SHA-256 of the normalized .rktd content's UTF-8 bytes, before compression.
license		Accepted upstream redistribution terms.
citation		Source attribution and bibliographic citation.
notes		Transformations, representation choices, and historical quirks.
time-series		Time-series description hash, or #f.
tier		bundled (shipped with its package) or fetch (downloaded on first use).
package		Name of the package whose registry declares the dataset.
fetch		#f for bundled data. For fetch-tier data, a hash with urls (mirrors, tried in order), the sha256 of the download, and its format. The only format is rktd.gz: the download is the dataset’s gzip-compressed .rktd file, the form bundled data is stored in. Other keys are ignored.

Each column-description hash has name, original-name, type, levels, unit, description, and missing-count keys. Semantic types are strings: integer, real, categorical, identifier, or string. Category levels are ordered strings; a noncategory has an empty level list. A unit of #f means unspecified. The missing count refers to that column in the complete source dataset.

For AirPassengers, the time-series hash has frequency (12 observations per year), start and end (year/month lists), and index (the year and month column names). Other v0.1 datasets have #f for time-series.

1	Guide
2	Reference
3	Synthetic Data
4	Dataset Catalog and Provenance
	Index

2.1	Registry and loading
2.2	Downloads and the cache
2.3	Table inspection
2.4	Missing values
2.5	Metadata fields