1.6.6.1 The data🔗ℹ

Class 1 has high x₁ and low x₂, class 0 the reverse, and x₃ is noise, identically distributed in both classes:

> (define X '((1.0 5.0 2.0)               (2.0 6.0 1.0)               (2.0 5.0 3.0)               (1.0 4.0 1.0)               (3.0 6.0 2.0)               (2.0 4.0 2.0)               (6.0 2.0 2.0)               (5.0 1.0 1.0)               (6.0 1.0 3.0)               (5.0 2.0 1.0)               (4.0 1.0 2.0)               (6.0 3.0 2.0)))

> (define y '(0 0 0 0 0 0 1 1 1 1 1 1))

1.6.6.2 Fitting🔗ℹ

> (define fit (logistic-fit X y #:lambda 0.04))

> fit

#<glmnet:binomial λ=0.04 dev=0.9266 nz=2/3>

> (coef fit)

'#(-0.7794670198037468 0.9979652738654424 -0.8362224983591743 0.0)

coef lists the intercept and then the coefficients, all on the log-odds scale. x₁ raises the odds of class 1 and x₂ lowers them; the noise predictor is exactly 0.0. Exponentiating a coefficient gives an odds ratio, the factor by which one more unit of the predictor multiplies the odds of class 1:

> (for/list ([b (in-vector (logistic-result-coefficients fit))])     (exp b))

'(2.7127564925605556 0.43334439467246694 1.0)

logistic-result-dev-ratio is the fraction of null deviance explained, the logistic analogue of R².

1.6.6.3 Predicting🔗ℹ

logistic-predict-proba returns the class-1 probability for each row; logistic-predict thresholds it into a label. The first is predict with #:type 'response; at its default threshold, the second agrees with #:type 'class:

> (logistic-predict-proba fit X)

'(0.018658581830764638   0.02186262725172905   0.04904870960935964   0.04203165380714618   0.057167332847504866   0.10636446519086237   0.9716915927629215   0.9668863972296748   0.9875327213177286   0.9267573826785769   0.9149921263903765   0.9370064088370706)

> (equal? (logistic-predict fit X) y)

#t

> (equal? (predict fit X #:type 'class) y)

#t

Every training point is classified correctly. The threshold defaults to 0.5; for a point near the boundary, where it sits decides the label:

> (define borderline '((3.5 3.5 2.0)))

> (logistic-predict-proba fit borderline)

'(0.44685962480034513)

> (logistic-predict fit borderline)

'(0)

> (logistic-predict fit borderline #:threshold 0.4)

'(1)

Lowering the threshold trades false negatives for false positives, which is what you want when missing a class-1 case costs more than a false alarm.

1.6.6.4 Separation and the penalty🔗ℹ

These two classes are linearly separable: a line in the (x₁, x₂) plane splits them perfectly. Unpenalized logistic regression has no finite solution on such data, because making the coefficients larger always fits a little better. The penalty is what keeps the fit well defined, as logistic-path shows by fitting a list of λ in one call, largest first (Regularization paths):

> (define (round3 x)     (/ (round (* 1000 x)) 1000))

> (define (rounded v)     (for/list ([b (in-vector v)])       (round3 b)))

> (define path (logistic-path X y #:lambda '(0.0 0.001 0.01 0.04 0.1 0.3)))

> (for ([lam (in-vector (glmnet-path-lambda path))]         [beta (in-vector (glmnet-path-coefficients path))]         [dev (in-vector (glmnet-path-dev-ratio path))])     (printf "λ = ~a: β = ~a, dev-ratio = ~a\n" lam (rounded beta) (round3 dev)))

λ = 0.3: β = (0.246 -0.158 0.0), dev-ratio = 0.412 λ = 0.1: β = (0.669 -0.547 0.0), dev-ratio = 0.817 λ = 0.04: β = (0.998 -0.836 0.0), dev-ratio = 0.927 λ = 0.01: β = (1.503 -1.267 0.0), dev-ratio = 0.981 λ = 0.001: β = (2.386 -2.008 0.0), dev-ratio = 0.998 λ = 0.0: β = (7.975 -5.369 1.842), dev-ratio = 1.0

At λ = 0 the solver returns large coefficients, the noise predictor among them, with a deviance ratio of essentially 1: the model has memorized the training labels. With no finite optimum, which large coefficients it returns depends on where it starts. The path starts from its fit at λ = 0.001; a single fit starts from zero and returns different ones:

> (rounded (logistic-result-coefficients (logistic-fit X y #:lambda 0.0)))

'(6.677 -5.589 -3.542)

Even a small penalty removes the noise and keeps the coefficients finite. On harder data, glmnet can instead fail with an error saying a class probability collapsed; a larger #:lambda fixes that too.

1.6.6.5 When to use it🔗ℹ

Use the binomial family for any yes/no outcome: default or not, spam or not, malignant or benign. The lasso logistic is a standard tool for building a sparse, interpretable classifier from many candidate features. For more than two classes, see Multinomial classification (K classes).