1.7.3 Loading the file
The post times three readers. data-frame takes no other separator, so the post converts the TSV to CSV for it, and tabular-asa reads the same copy:
; csv-reading, from the URL: cpu 4366 ms, real 9121 ms (time (define data (csv->list file)) (void)) ; data-frame: 3708 ms (time (define data (df-read/csv file #:headers? #t)) (void)) ; tabular-asa: 7142 ms (time (define data (call-with-input-file "nycflights.csv" table-read/csv)) (void))
API gap: read-csv reads a local file, not a URL as the post’s first reader does and Python’s read_csv can (#186); fetch the file first.
The sample’s first lines and its last:
$ head -n 3 nycflights-sample.tsv
year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay carrier flight tailnum origin dest air_time distance hour minute time_hour
2013 1 1 517 515 2 830 819 11 UA 1545 N14228 EWR IAH 227 1400 5 15 2013-01-01 05:00:00
2013 1 1 533 529 4 850 830 20 UA 1714 N24211 LGA IAH 227 1416 5 29 2013-01-01 05:00:00
$ tail -n 1 nycflights-sample.tsv
2013 9 1 NA 1920 NA NA 2052 NA 9E 3920 NA LGA ROC NA 254 19 20 2013-09-01 19:00:00
read-csv reads the TSV as it is: #:separator names the tab, and #:null-values the NA marker, which becomes a null in an integer column.
> (define flights (read-csv "nycflights-sample.tsv" #:separator #\tab #:null-values "NA")) > flights
shape: (5_434, 19)
┌──────┬───────┬─────┬──────────┬───┬──────────┬──────┬────────┬─────────────────────┐
│ year ┆ month ┆ day ┆ dep_time ┆ … ┆ distance ┆ hour ┆ minute ┆ time_hour │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 ┆ i64 ┆ str │
╞══════╪═══════╪═════╪══════════╪═══╪══════════╪══════╪════════╪═════════════════════╡
│ 2013 ┆ 1 ┆ 1 ┆ 517 ┆ … ┆ 1400 ┆ 5 ┆ 15 ┆ 2013-01-01 05:00:00 │
│ 2013 ┆ 1 ┆ 1 ┆ 533 ┆ … ┆ 1416 ┆ 5 ┆ 29 ┆ 2013-01-01 05:00:00 │
│ 2013 ┆ 1 ┆ 1 ┆ 542 ┆ … ┆ 1089 ┆ 5 ┆ 40 ┆ 2013-01-01 05:00:00 │
│ 2013 ┆ 1 ┆ 1 ┆ 544 ┆ … ┆ 1576 ┆ 5 ┆ 45 ┆ 2013-01-01 05:00:00 │
│ 2013 ┆ 1 ┆ 1 ┆ 554 ┆ … ┆ 762 ┆ 6 ┆ 0 ┆ 2013-01-01 06:00:00 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ 2013 ┆ 9 ┆ 1 ┆ 2302 ┆ … ┆ 187 ┆ 23 ┆ 5 ┆ 2013-09-01 23:00:00 │
│ 2013 ┆ 9 ┆ 1 ┆ 2329 ┆ … ┆ 266 ┆ 22 ┆ 45 ┆ 2013-09-01 22:00:00 │
│ 2013 ┆ 9 ┆ 1 ┆ 2351 ┆ … ┆ 1617 ┆ 23 ┆ 59 ┆ 2013-09-01 23:00:00 │
│ 2013 ┆ 9 ┆ 1 ┆ 2352 ┆ … ┆ 1598 ┆ 23 ┆ 59 ┆ 2013-09-01 23:00:00 │
│ 2013 ┆ 9 ┆ 1 ┆ null ┆ … ┆ 254 ┆ 19 ┆ 20 ┆ 2013-09-01 19:00:00 │
└──────┴───────┴─────┴──────────┴───┴──────────┴──────┴────────┴─────────────────────┘
Without #:null-values, the first rows infer integers and the first NA fails the read; the error says what to pass:
> (read-csv "nycflights-sample.tsv" #:separator #\tab) read-csv: failed to read csv from nycflights-sample.tsv:
could not parse `NA` as dtype `i64` at column 'arr_delay'
(column number 9)
The current offset in the file is 1853 bytes.
You might want to try:
- increasing #:infer-schema-length (e.g.
#:infer-schema-length 10000, or #f for every row),
- specifying correct dtype with #:schema-overrides
- setting #:ignore-errors to #t,
- adding `NA` to #:null-values.
Original error: ```invalid primitive value found during CSV
parsing```