1.6.3.1 The data🔗ℹ

The fixture from Ordinary least squares (λ = 0), extended by one row and one irrelevant predictor, x₃ = x₁². The response is still exactly y = 1 + 2x₁ − x₂, so x₃ carries no signal, but it is strongly correlated with x₁:

> (define X '((1.0 2.0  1.0)               (2.0 1.0  4.0)               (3.0 4.0  9.0)               (4.0 3.0 16.0)               (5.0 6.0 25.0)               (6.0 5.0 36.0)))

> (define y '(1.0 4.0 3.0 6.0 5.0 8.0))

1.6.3.2 Fitting🔗ℹ

ridge is elnet-fit with #:alpha 0.0:

> (define fit (ridge X y #:lambda 0.1))

> (elnet-result-coefficients fit)

'#(1.1614345231080536 -0.7532810830027743 0.08306586531538782)

> (elnet-result-r-squared fit)

0.978302360842607

Least squares would give x₃ a coefficient of zero. Ridge gives it a small nonzero weight instead, and pulls x₁ down from 2.0: the penalty prefers to spread weight across correlated predictors rather than concentrate it. That is the characteristic ridge behaviour: shrink, never select.

1.6.3.3 The shrinkage path🔗ℹ

elnet-path with #:alpha 0.0 fits the ridge penalty over a list of λ in one call, largest first (Regularization paths), and traces how the coefficients shrink:

> (define (round3 x)     (/ (round (* 1000 x)) 1000))

> (define (rounded v)     (for/list ([b (in-vector v)])       (round3 b)))

> (define path (elnet-path X y #:alpha 0.0 #:lambda '(0.01 0.1 1.0 10.0 100.0)))

> (for ([lam (in-vector (glmnet-path-lambda path))]         [a0 (in-vector (glmnet-path-intercepts path))]         [beta (in-vector (glmnet-path-coefficients path))])     (printf "λ = ~a: β₀ = ~a, β = ~a\n" lam (round3 a0) (rounded beta)))

λ = 100.0: β₀ = 4.316, β = (0.025 0.013 0.003) λ = 10.0: β₀ = 3.301, β = (0.173 0.069 0.023) λ = 1.0: β₀ = 2.003, β = (0.553 -0.14 0.069) λ = 0.1: β₀ = 1.812, β = (1.161 -0.753 0.083) λ = 0.01: β₀ = 1.241, β = (1.788 -0.965 0.025)

Read upward, as λ grows, every coefficient heads toward zero and none reaches it, while the intercept heads toward the mean of y, which is 4.5. Along the way the coefficient on x₂ changes sign. As λ grows, each ridge coefficient approaches a scaled-down copy of its predictor’s covariance with the response, taken on its own; x₂ rises with x₁, so on its own it is positively correlated with y, even though its coefficient in the generating equation is negative.

1.6.3.4 When to use it🔗ℹ

Reach for ridge when many predictors each carry a little signal, or when predictors are strongly correlated, and you want a stable fit rather than a short list of predictors. It never drops a predictor; when you want the fit to say which predictors matter, use Lasso (L1, α = 1) or Elastic net (0 < α < 1).