1.6.2.1 The data🔗ℹ

The response is an exact linear function of two predictors, y = 1 + 2x₁ − x₂, with no error term, so least squares should recover the intercept 1.0 and the coefficients 2.0 and -1.0:

> (define X '((1.0 2.0)               (2.0 1.0)               (3.0 4.0)               (4.0 3.0)               (5.0 6.0)))

> (define y '(1.0 4.0 3.0 6.0 5.0))

1.6.2.2 Fitting🔗ℹ

ols is elnet-fit with #:lambda 0.0. The fit prints as a summary; coef lists the intercept and then the coefficients:

> (define fit (ols X y))

> fit

#<glmnet:gaussian λ=0 dev=1.0000 nz=2/2>

> (coef fit)

'#(1.0000143163378508 1.9999829127580482 -0.9999884545662485)

> (elnet-result-r-squared fit)

0.999999999936016

The intercept and coefficients match the generating equation to about five decimal places, and R² is 1 to within 1e-10. They are not exact because coordinate descent stops once a pass changes the objective by less than #:thresh. A tighter threshold costs more passes and gets closer:

> (define tight (ols X y #:thresh 1e-14))

> (coef tight)

'#(1.0000001296257732 1.9999998452853673 -0.9999998954630858)

> (list (elnet-result-num-passes fit) (elnet-result-num-passes tight))

'(29 41)

1.6.2.3 Fitted values🔗ℹ

A fitted value is the intercept plus the dot product of a row with the coefficients. predict computes it for each row, here the training rows, whose fitted values reproduce y to the same five decimal places:

> (predict fit X)

'(1.0000203199634021   3.9999916872876984   3.000009236347002   5.999980603671299   4.999998152730601)

elnet-predict is the same computation under the Gaussian family’s own name. On new rows the fit extrapolates the plane it found:

> (elnet-predict fit '((6.0 5.0) (0.0 0.0)))

'(7.999969520054897 1.0000143163378508)

1.6.2.4 When to use it🔗ℹ

Least squares is the baseline the penalized models are measured against. It has no answer to overfitting or to collinear predictors: when two predictors carry the same information, or when there are more predictors than rows, the least-squares coefficients are not unique. Ridge regression (L2, α = 0) handles that case, and Lasso (L1, α = 1) handles the case where most predictors are noise.