1.2.1 Data layout🔗ℹ

A design matrix holds the predictors: one row per observation and one column per predictor. Every fit and prediction procedure accepts it in either of two forms:

• a non-empty list or vector of rows, each a list or vector of reals of the same length; or

• a design-matrix? value, which holds the matrix in the layout the Fortran reads.

> (define X '((1.0 2.0)               (2.0 1.0)               (3.0 4.0)               (4.0 3.0)               (5.0 6.0)))

> (define y '(1 4 3 6 5))

> (elnet-result-coefficients (ols X y))

'#(1.9999829127580482 -0.9999884545662485)

The Fortran reads a matrix as one array of doubles stored column by column: element (i, j) of a matrix with n rows is at index i + jn, counting from 0. A design-matrix? value holds exactly that array, together with the numbers of rows and columns and, optionally, column names. rows->design-matrix builds one from a list of rows:

> (define D (rows->design-matrix X #:column-names '(x1 x2)))

> D

#<design-matrix 5x2>

> (design-matrix-ref D 2 1)

4.0

> (design-matrix-column-names D)

'("x1" "x2")

> (require ffi/vector)

> (f64vector->list (design-matrix->f64vector D))

'(1.0 2.0 3.0 4.0 5.0 2.0 1.0 4.0 3.0 6.0)

A fit on D gives the same result as a fit on X. The list is converted on every call, while D was converted once. The solver copies its input before working on it, so one design matrix serves any number of fits:

> (equal? (ols D y) (ols X y))

#t

> (for/list ([lam (in-list '(1.0 0.1 0.01))])     (elnet-result-coefficients (lasso D y #:lambda lam)))

'(#(0.29289321881345226 0.0)   #(1.602368245039978 -0.6732060759253727)   #(1.9598955357543744 -0.9670900070860349))

A fit from a matrix carries the column names along but does not use them; to fit from named columns, see Named data. columns->design-matrix builds a design matrix from columns, and f64vector->design-matrix from an array that is already in the column-major layout. design-matrix->rows, design-matrix->columns and design-matrix->f64vector convert back:

> (define C (columns->design-matrix '((1 2 3) (4 5 6))))

> (design-matrix->rows C)

'((1.0 4.0) (2.0 5.0) (3.0 6.0))

> (define v (f64vector 1.0 2.0 3.0 4.0 5.0 6.0))

> (equal? (f64vector->design-matrix v 3 2) C)

#t

Every conversion checks its input once, before any solver runs. The matrix must have at least one row and one column, every row must have the same length, and every entry must be a real, finite number. Exact numbers become flonums. An error names the offending row and column, counting from 0:

> (design-matrix->rows (rows->design-matrix '((1 1/2) (-3 2.5))))

'((1.0 0.5) (-3.0 2.5))

> (rows->design-matrix '((1.0 2.0) (3.0)))

rows->design-matrix: the matrix has rows of different

lengths

row: 1

length: 1

length of row 0: 2

> (ols '((1.0 2.0) (2.0 +nan.0) (3.0 4.0) (4.0 3.0) (5.0 6.0)) y)

ols: X has an element that is not finite

column: 1

row: 1

element: +nan.0

Without the check, a +nan.0 or an infinity would reach the solver, which would return meaningless coefficients and no error. R’s glmnet also stops on a missing value in x, but not on every non-finite value. In every family but Cox it fits an x with an infinite entry, giving that column a zero coefficient. It fits a Poisson response with an infinite count and returns an empty model. Its predict carries non-finite new data through to the predictions. Here each of these is an error.

The response has one entry per row of the design matrix. It is a list, a vector, an flvector or an f64vector, and its entries depend on the family:

A response is checked in the same way as a design matrix, by response->f64vector: every entry must be finite, and there must be one per row of the design matrix:

> (ols X '(1.0 4.0 +inf.0 6.0 5.0))

ols: y has an element that is not finite

position: 2

element: +inf.0

> (ols X '(1.0 2.0))

ols: y does not have one entry per row of X

length of y: 2

rows of X: 5

The rows of a design matrix, and the entries of a response, can be vectors as well as lists. A design matrix comes in four nestings, whose outer sequence always holds the rows, and a response in four one-dimensional forms:

Every shape fits exactly as the same lists do, and the prediction procedures read new data in any nesting:

> (require racket/flonum)

> (define list-of-vectors (map list->vector X))

> (define vector-of-lists (list->vector X))

> (define Xv (list->vector (map list->vector X)))

> Xv

'#(#(1.0 2.0) #(2.0 1.0) #(3.0 4.0) #(4.0 3.0) #(5.0 6.0))

> (for/list ([rows (list X list-of-vectors vector-of-lists Xv)])     (equal? (ols rows y) (ols X y)))

'(#t #t #t #t)

> (for/list ([response (list (list->vector y)                              (flvector 1.0 4.0 3.0 6.0 5.0)                              (f64vector 1.0 4.0 3.0 6.0 5.0))])     (equal? (ols X response) (ols X y)))

'(#t #t #t)

> (elnet-predict (ols Xv (list->vector y)) (vector #(6.0 5.0) '(0.0 1.0)))

'(7.999969520054897 2.5861771602242456e-5)

glmnet/data/nested converts the nestings to a design-matrix? value and back (see Lists and vectors).

The Fortran reads a response as an f64vector, and every form of a response is copied into a fresh one. An flvector holds the same doubles, but it could not be passed as it is in any case: the FFI does not accept an flvector where it expects an f64vector.