1.6.7.1 The data🔗ℹ

Three separable clusters: class 0 at low x₁, class 1 at high x₁, class 2 at high x₂:

> (define X '((1.0 1.0) (2.0 1.0) (1.0 2.0) (2.0 2.0)               (5.0 1.0) (6.0 1.0) (5.0 2.0) (6.0 2.0)               (3.0 5.0) (4.0 5.0) (3.0 6.0) (4.0 6.0)))

> (define y '(0 0 0 0 1 1 1 1 2 2 2 2))

1.6.7.2 Fitting🔗ℹ

> (define fit (multinomial-fit X y #:lambda 0.01))

> (multinomial-result-intercepts fit)

'#(6.334774419914438 -2.453188136398622 -3.881586283515815)

> (multinomial-result-coefficients fit)

'#(#(-1.2558971734686988 0.0)    #(1.2550226289748376 0.0)    #(0.0 2.0229335782284603))

Each class keeps only the predictor that sets it apart. Classes 0 and 1 split on x₁, with opposite signs; class 2 is picked out by x₂; every other coefficient is exactly 0.0.

Only differences between the classes’ linear predictors matter: adding the same vector to every class’s coefficients leaves the softmax unchanged. glmnet resolves that ambiguity with the penalty, which prefers the smallest coefficients, so the class-0 and class-1 weights on x₁ come out nearly equal and opposite rather than, say, zero and twice as large. The penalty does not touch the intercepts, which have the same freedom; as R does, the bindings centre them to sum to zero.

1.6.7.3 Predicting🔗ℹ

multinomial-predict-proba returns one probability per class for each row, summing to 1, and multinomial-predict returns the most probable class. They are predict with #:type 'response and #:type 'class:

> (define new-points '((1.5 1.5) (5.5 1.5) (3.5 5.5) (3.5 1.5)))

> (multinomial-predict-proba fit new-points)

'((0.9885378101387693 0.006518498377706142 0.004943691483524595)   (0.006515125658992151 0.9885338196834262 0.0049510546575815845)   (0.004915825133083355 0.0049170874442380445 0.9901670874226786)   (0.4849885303770453 0.4851130682552687 0.029898401367686074))

> (multinomial-predict fit new-points)

'(0 1 2 1)

> (equal? (multinomial-predict fit X) y)

#t

The first three points sit at the cluster centres and are classified with confidence. The fourth lies between clusters 0 and 1; its probabilities show how close the call is.

1.6.7.4 Class labels🔗ℹ

The labels must be the integers 0 to K−1, with every class present. A gap is an error, not an empty class:

> (multinomial-fit X '(0 0 0 0 2 2 2 2 2 2 2 2) #:lambda 0.01)

multinomial-fit: class 1 has no observations; labels must

cover 0..2 contiguously

1.6.7.5 When to use it🔗ℹ

Use the multinomial family for a categorical outcome with more than two unordered levels: species, document topics, product categories. For two classes, use Binomial logistic regression (classification).