1.2.5 Regularization paths🔗ℹ

The right λ is rarely known in advance. A regularization path fits a whole decreasing sequence of λ in one call, starting each fit from the solution before it, which is much cheaper than fitting each value separately. It is how R’s glmnet(x, y) is normally used. Every family has a path fitter: elnet-path, logistic-path, multinomial-path, cox-path, poisson-path and mgaussian-path. They take the same keywords as the single fits, except that #:lambda is optional and, when given, a list.

Without #:lambda, glmnet chooses the sequence the way R does: #:nlambda values (default 100) from λmax, the smallest λ at which every coefficient is zero, down to #:lambda-min-ratio times λmax (default 0.01 when there are fewer observations than predictors, otherwise 0.0001):

> (vector-length (glmnet-path-lambda (elnet-path X y)))

50

> (define path (elnet-path X y #:nlambda 12))

> (vector-ref (glmnet-path-coefficients path) 0)

'#(0.0 0.0)

> (glmnet-path-df path)

'#(0 1 2 2 2 2 2)

A glmnet-path holds one entry per fitted λ in each of its lambda, intercepts, coefficients, dev-ratio and df fields. At the first λ every coefficient is zero; df counts the predictors in the model as the penalty relaxes, here x₁ first and then x₂. The default path has 50 values, not 100, because glmnet, like R, stops an automatic sequence once another λ would barely change the fit. For the Gaussian family that is when R² gains less than 1e-5 of its own value from one λ to the next, or passes 0.999; the reference lists each family’s rule (Regularization paths).

A path prints as R prints one: a numbered row for each fitted λ, with the number of nonzero coefficients (Df), the percentage of the null deviance explained (%Dev) and λ:

> path

#<glmnet-path:gaussian   Df  %Dev  Lambda 1  0  0.00 1.41400 2  1 54.91 0.61220 3  2 73.33 0.26500 4  2 95.00 0.11470 5  2 99.06 0.04966 6  2 99.82 0.02149 7  2 99.97 0.00931>

With #:lambda, the path fits those values, largest first:

> (define user-path (elnet-path X y #:lambda '(0.01 0.5 0.1)))

> (glmnet-path-lambda user-path)

'#(0.5 0.1 0.01)

> (glmnet-path-coefficients user-path)

'#(#(0.646446609406726 0.0)    #(1.602368245039978 -0.6732060759253727)    #(1.9599009650300776 -0.9670936755155641))

Each point agrees with the single fit at that λ to within the solver’s tolerance. The early-stopping rule does not apply to a user sequence, but any path can still end early, for example where glmnet fails at one of its values by not converging within #:max-iters passes. It then keeps the values before that one and logs a warning (Convergence and errors):

> (glmnet-path-lambda (elnet-path X y #:lambda '(0.5 0.1 0.01) #:max-iters 1))

'#(0.5)

predict and coef evaluate a path at any λ (Predictions and coefficients). Cross-validation chooses among the values on a path (Choosing λ by cross-validation).

To plot a path’s coefficients against λ, as R’s plot does, use plot-coefficient-path from glmnet/plot; see Coefficient paths.