1.7.12 Describing strings and datetimes🔗ℹ

The post finds df-describe “pretty useless” on carrier, dest and time_hour, which data-frame keeps as strings. describe gives a string column its count, null count, minimum and maximum; parsed as a datetime, time_hour also gets a mean and quartiles. The summary’s columns are strings, as in Python; describe leaves the input’s dtypes alone:

> (~> flights       (with-columns (str->datetime "time_hour"))       (select "carrier" "dest" "time_hour")       describe)

shape: (9, 4) ┌────────────┬─────────┬──────┬────────────────────────────┐ │ statistic  ┆ carrier ┆ dest ┆ time_hour                  │ │ ---        ┆ ---     ┆ ---  ┆ ---                        │ │ str        ┆ str     ┆ str  ┆ str                        │ ╞════════════╪═════════╪══════╪════════════════════════════╡ │ count      ┆ 5434    ┆ 5434 ┆ 5434                       │ │ null_count ┆ 0       ┆ 0    ┆ 0                          │ │ mean       ┆ null    ┆ null ┆ 2013-06-01 16:28:23.938167 │ │ std        ┆ null    ┆ null ┆ null                       │ │ min        ┆ 9E      ┆ ABQ  ┆ 2013-01-01 05:00:00        │ │ 25%        ┆ null    ┆ null ┆ 2013-03-01 14:00:00        │ │ 50%        ┆ null    ┆ null ┆ 2013-05-01 20:00:00        │ │ 75%        ┆ null    ┆ null ┆ 2013-09-01 13:00:00        │ │ max        ┆ YV      ┆ XNA  ┆ 2013-11-01 23:00:00        │ └────────────┴─────────┴──────┴────────────────────────────┘

read-csv’s #:try-parse-dates parses time_hour at load instead. A categorical column gets only the two counts, as in Python; its useful summary is a count per category:

> (~> coded       (group-by "carrier")       (agg (alias (count "carrier") "flights"))       (sort "flights" #:descending #t)       (head 5))

shape: (5, 2) ┌─────────┬─────────┐ │ carrier ┆ flights │ │ ---     ┆ ---     │ │ cat     ┆ u32     │ ╞═════════╪═════════╡ │ UA      ┆ 960     │ │ B6      ┆ 920     │ │ EV      ┆ 838     │ │ DL      ┆ 754     │ │ AA      ┆ 547     │ └─────────┴─────────┘