1.6.5.1 Fitting🔗ℹ

The running fixture again, with the irrelevant x₃ = x₁²:

> (define X '((1.0 2.0  1.0)               (2.0 1.0  4.0)               (3.0 4.0  9.0)               (4.0 3.0 16.0)               (5.0 6.0 25.0)               (6.0 5.0 36.0)))

> (define y '(1.0 4.0 3.0 6.0 5.0 8.0))

> (define fit (elastic-net X y #:alpha 0.5 #:lambda 0.5))

> (elnet-result-coefficients fit)

'#(0.5773773035370475 0.0 0.054686809526739884)

The fit both shrinks its coefficients, as ridge does, and sets one exactly to zero, as the lasso does.

1.6.5.2 Between ridge and lasso🔗ℹ

At a shared λ, the number of coefficients the elastic net zeroes lies between ridge, which zeroes none, and the lasso, which zeroes the most:

> (define (zeros fit)     (for/sum ([b (in-vector (elnet-result-coefficients fit))])       (if (zero? b) 1 0)))

> (map zeros (list (ridge X y #:lambda 0.5)                    (elastic-net X y #:alpha 0.5 #:lambda 0.5)                    (lasso X y #:lambda 0.5)))

'(0 1 2)

Sweeping α at that λ moves between the two:

> (define (round3 x)     (/ (round (* 1000 x)) 1000))

> (define (rounded v)     (for/list ([b (in-vector v)])       (round3 b)))

> (for ([alpha (in-list '(0.0 0.25 0.5 0.75 1.0))])     (printf "α = ~a: β = ~a\n"             alpha             (rounded (elnet-result-coefficients                       (elastic-net X y #:alpha alpha #:lambda 0.5)))))

α = 0.0: β = (0.712 -0.335 0.082) α = 0.25: β = (0.649 -0.183 0.07) α = 0.5: β = (0.577 0.0 0.055) α = 0.75: β = (0.623 0.0 0.042) α = 1.0: β = (0.879 0.0 0.0)

1.6.5.3 Correlated predictors🔗ℹ

The case the elastic net was designed for is a group of predictors that carry the same signal. Take the extreme: duplicate x₁, so the first two columns are identical, and drop x₃:

> (define X-dup     (for/list ([row (in-list X)])       (list (car row) (car row) (cadr row))))

> (rounded (elnet-result-coefficients (lasso X-dup y #:lambda 0.1)))

'(1.658 0.0 -0.658)

> (rounded (elnet-result-coefficients (elastic-net X-dup y #:alpha 0.5 #:lambda 0.1)))

'(0.87 0.855 -0.726)

The lasso puts all of x₁’s weight on one copy and zeroes the other; which copy wins is an accident of the column order. The elastic net’s L2 term makes splitting the weight cheaper than concentrating it, so the two copies get almost equal coefficients. On real data the copies are merely correlated, but the effect is the same.

1.6.5.4 Choosing α and λ🔗ℹ

elnet-cv chooses λ for one α (Choosing λ by cross-validation). To choose α too, cross-validate each candidate on the same folds, so that the errors differ only because the models do. Take the 40 noisy observations of Choosing λ by cross-validation:

> (random-seed 3)

> (define (uniform a b)     (+ a (* (- b a) (random))))

> (define X40     (for/list ([i (in-range 40)])       (define x1 (uniform 0 6))       (list x1 (uniform 0 6) (* x1 x1))))

> (define y40     (for/list ([row (in-list X40)])       (+ 1.0 (* 2 (car row)) (- (cadr row)) (uniform -1 1))))

> (define folds (random-fold-ids 40))

> (for ([alpha (in-list '(0.0 0.25 0.5 0.75 1.0))])     (define cv (elnet-cv X40 y40 #:alpha alpha #:fold-ids folds))     (define best (glmnet-cv-index-min cv))     (printf "α = ~a: λ-min = ~a, error = ~a ± ~a\n"             alpha             (round3 (glmnet-cv-lambda-min cv))             (round3 (vector-ref (glmnet-cv-cvm cv) best))             (round3 (vector-ref (glmnet-cv-cvsd cv) best))))

α = 0.0: λ-min = 0.284, error = 0.512 ± 0.083 α = 0.25: λ-min = 0.024, error = 0.339 ± 0.058 α = 0.5: λ-min = 0.023, error = 0.338 ± 0.058 α = 0.75: λ-min = 0.021, error = 0.338 ± 0.057 α = 1.0: λ-min = 0.017, error = 0.338 ± 0.057

Ridge, which keeps every predictor, the irrelevant x₃ among them, predicts clearly worse. Every α above 0 reaches nearly the same error, well within one standard error of each other, so on these data the choice among them matters much less than the choice of λ.

1.6.5.5 When to use it🔗ℹ

The elastic net is a good default when you want selection but expect groups of correlated predictors, as with gene expression, sensor arrays or technical indicators. α is a second tuning parameter alongside λ; values around 0.5 are a common starting point, and cross-validation on shared folds chooses between candidates.