Practice: Linear Discriminants, and the Covariance They Assume

(a) Separation is a ratio, not a gap.

What decides whether two projected classes overlap is the gap between their centres relative to how widely each class spreads around its own centre. A direction can push the centres far apart and, if the classes are elongated along that same direction, spread each class over an even wider interval, so the intervals still overlap.

That is what happens here. The difference of means points along the direction in which both classes are internally stretched, so the gap of $20$ comes with an enormous within-class spread. The discriminant direction gives a gap of only $1.863636$, but each class projects to a narrow interval, and the intervals do not touch.

The Fisher criterion makes this precise, dividing squared gap by within-class spread: $1.863636$ for the discriminant against $0.911854$ for the difference of means, better by a factor of $2.043788$. The raw gaps are not comparable quantities, because they are measured along directions with different scales.

(b) The within-class correlation.

A correlation near $0.9$ means the two measurements move together strongly within each class: each class is a long, thin, diagonally oriented cloud rather than a round blob. The two variables are nearly redundant inside a class, so the classes are stretched along one diagonal and narrow across it.

That shape is exactly what makes the naive direction fail, because the line joining the class centres runs roughly along the elongation.

(c) When the two coincide.

They coincide when $S_w$ is a multiple of the identity, equal spread in both variables and zero within-class correlation, so each class is round. Then $S_w^{-1}$ is also a multiple of the identity, and

the same direction up to scale, which leaves the classification unchanged.

This is why the naive answer feels right: it is right, for round classes. Diagrams illustrating classification almost always draw round clouds, so the case where the correction does nothing is the case most often pictured. The correction's size is a property of the data, inspect $S_w$ and you can tell in advance whether it will matter.

(a) Barely better than guessing. With two balanced classes, assigning at random gives an expected error rate of $50\mathrm{\%}$, or $9$ of $18$. The fitted rule misclassifies $7$ of $18$, which is $38.89\mathrm{\%}$. So the classifier improves on chance by about eleven percentage points. Two observations. For a method given labelled data and a two-dimensional predictor space, that is close to no information at all, and it is the figure that should have prompted the diagnosis rather than a request for more data. (b) Two independent reasons. The class means nearly coincide. They are $(0,0)$ and $(0.7\bar{7},0.1\bar{1})$, so ${\overline{x}}_1-{\overline{x}}_0$ is a very short vector. That difference is the numerator of the whole construction, $w=S_w^{-1}({\overline{x}}_1-{\overline{x}}_0)$: if the classes have essentially the same centre, there is no direction along which their centres separate, and every projection puts the two classes on top of one another. This alone defeats any linear rule, whatever the covariances. The covariances are grossly unequal. The scatter traces are $24$ and $\frac{3496}{9}\approx 388.44$, a ratio of $16.1852$. Pooling them produces a matrix describing neither class. An average of a tight cloud and a very broad one. The linear form is optimal only when both classes share a covariance, and that premise is false by more than an order of magnitude. The two reasons are separate. Fixing the covariance disparity would not create a separation between coincident means, and separating the means would not repair the pooling. (c) More data would not help, and the per-class result shows why. Additional observations from the same process would estimate the same means and the same covariances more precisely. The means would remain nearly coincident and the covariances would remain unequal, because these are properties of the populations rather than artefacts of a small sample. The error rate would not improve. What the per-class result establishes is decisive: a rule permitting each class its own covariance misclassifies $0$ of $18$ on the identical observations. So the data contains all the information needed to separate the classes perfectly. The information was never missing. The constraint was the shape of boundary the method was permitted to draw. A linear rule can only cut the plane with a straight line, and these classes are arranged so that no straight line separates them: the configuration is one class surrounding the other, which a quadratic boundary handles and a linear one cannot. The correct recommendation is to change the method, not the sample size. And the general lesson is that the fitted output gave no sign of any of this. A direction, a threshold and a classification were produced exactly as they are when the assumptions hold.

(a) It is a training figure, and it is on twelve samples. The direction and threshold were computed from the same twelve observations the accuracy was measured on. The class means and pooled scatter that produced $w$ came from those points, so the rule was fitted to separate exactly them. A perfect score under those conditions is the weakest evidence of performance, not the strongest. The sample size compounds it. Twelve observations, six per class, in two dimensions, perfect separation is unsurprising even for classes that overlap substantially in the population, and would be achievable for many random label assignments. What the figure does support is a comparison on this data between two candidate directions, which is how the unit uses it: $0$ errors for the discriminant against $1$ for the difference of means. What it does not support is any claim about future observations. That needs held-out or resampled error, and none is reported. (b) The coefficient interpretation is wrong. A discriminant coefficient is not a per-variable effect. The direction is $S_w^{-1}({\overline{x}}_1-{\overline{x}}_0)$, and the inverse mixes the variables together. Each component depends on the scatter of both and on their correlation, not on one variable's relationship to the class. The data contradicts the reading directly. Both variables increase from class $0$ to class $1$: the means go from $(3.5,3.1\bar{6})$ to $(5.5,7.1\bar{6})$, so the difference $(2,4)$ is positive in both components. Variable 1 is positively associated with class membership, and the discriminant's first coefficient is still negative. The reason is the within-class correlation of $0.897150$. Because the variables move together strongly inside each class, the discriminant achieves separation by taking a contrast between them, roughly, how much variable 2 exceeds what variable 1 predicts. A negative weight on variable 1 is what constructs that contrast, and it says nothing about variable 1's own direction of association. Reading individual discriminant coefficients as effects is the same error as reading individual regression coefficients as effects when predictors are collinear: the combination is determined, the split between them is not interpretable. (c) The threshold is the wrong default for screening. The midpoint between projected class means is correct only when two conditions hold: the classes are equally likely in the population, and the two kinds of error cost the same. Neither is typical in screening. The condition being screened for is usually rare, so a threshold calibrated on a balanced sample of six and six misrepresents the prior. And a missed case and a false alarm almost never carry equal cost, in most screening settings a miss is far worse, which argues for moving the threshold toward catching more cases at the price of more false alarms. So the threshold should be set from the prevalence and the relative costs, both of which are decisions about the application rather than quantities computed from the fit. The report inherits a default from a balanced twelve-sample study and proposes deploying it unchanged. Before deployment, the defensible sequence is: estimate performance on data not used in fitting, establish the prevalence in the screened population, decide the relative cost of the two errors, and set the threshold from those, reporting the resulting sensitivity and specificity rather than a single accuracy number, which conceals the trade entirely.

(a) The $95\mathrm{\%}$ was a training figure on a balanced sample. The direction and threshold were computed from the same $40$ components the accuracy was measured on, so the rule was fitted to separate exactly those points. Training accuracy is optimistic by construction, and on $40$ observations in two dimensions a high figure is easy to obtain even for classes that overlap in the population. It was also measured on a sample that was $50\mathrm{\%}$ defective, against a production rate of $2\mathrm{\%}$. Accuracy on a balanced sample says nothing about accuracy on a stream where one class is rare. A rule calling everything sound would score $98\mathrm{\%}$ on production and be useless. What the figure supports: a comparison of candidate directions on that data. What it does not support: any statement about performance on future components, which needs error on observations not used in fitting. (b) A linear boundary cannot work here, for two separate reasons. The class means nearly coincide. $(10.0,10.0)$ against $(10.4,10.2)$, so ${\overline{x}}_1-{\overline{x}}_0=(0.4,0.2)$, tiny against within-class spreads with traces in the hundreds and thousands. That difference is the numerator of $w=S_w^{-1}({\overline{x}}_1-{\overline{x}}_0)$, so the construction is built on almost no signal. Projected onto any direction, the two classes will have nearly the same centre. The covariances are grossly unequal. The scatter traces are $180$ and $\mathrm{2,900}$, a ratio of

over sixteen to one. Pooling them yields a matrix describing neither class. The linear boundary is optimal only when both classes share a covariance, and that premise fails by more than an order of magnitude. The mechanism explains both. Defects arise from several unrelated causes pushing readings away from nominal in different directions, so the defective class is a diffuse cloud surrounding the tight sound class, with roughly the same centre. This is the ring-around-a-cloud configuration, and no straight line separates a surrounding shell from its own centre. The $95\mathrm{\%}$ was fitted to a particular sample of that shell; as different defect causes appeared over six months, the sampled shell filled out and the fitted line stopped corresponding to anything. (c) Refitting on $400$ components does not address it. More data estimates the same means and the same covariances more precisely. The means would remain nearly coincident and the covariance ratio would remain about sixteen, because these are properties of the populations rather than artefacts of the sample size. The linear rule would be fitted more stably to a boundary that cannot separate the classes. The proposal also repeats the sampling error: if the $400$ are again balanced, the threshold will again be calibrated for a prevalence the production line does not have. More data is worth collecting, it is needed for honest evaluation and for a per-class covariance estimate, but refitting the same model on it is the part that does not help. (d) Prevalence and costs both move the threshold, in the same direction. The midpoint between projected class means is correct only when the classes are equally likely and the two errors cost the same. Neither holds. Defects are $2\mathrm{\%}$ of production, not $50\mathrm{\%}$. On its own, a rarer positive class argues for a threshold less willing to call a component defective. But a missed defect costs about $200$ times a false alarm, which argues strongly the other way. Combining them, the relevant comparison weighs prevalence against cost: $0.02\times 200=4$ against $0.98\times 1$, so the expected cost of missing a defect outweighs that of a false alarm by roughly four to one. The threshold should be set to flag substantially more components than a midpoint rule would, accepting many false alarms to avoid misses. Neither figure is computable from the fit. Both are facts about the application, and the current rule encodes the opposite of both. (e) What to do instead. Change the shape of the boundary, or the framing. Given one tight class and one diffuse class surrounding it, the natural alternatives are a rule permitting each class its own covariance, which gives a quadratic boundary and is the direct fix for unequal covariances, or, better suited to the mechanism, treating this as anomaly detection rather than classification: model the sound class alone, which is tight and well characterised, and flag components far from it. That matches how defects actually arise, since "defective" is not one coherent class but several unrelated departures from nominal. Split the estimation from the evaluation. Fit on one portion and estimate error on another never used in fitting. Report performance on a sample reflecting the $2\mathrm{\%}$ prevalence, not a balanced one. Report sensitivity and specificity, never accuracy. At $2\mathrm{\%}$ prevalence accuracy is dominated by the common class and conceals exactly the errors that matter. Report the miss rate and the false-alarm rate separately, with the threshold that produced them. Set the threshold from prevalence and costs explicitly, stating both, so that the choice is reviewable rather than inherited from a balanced pilot study. Monitor for drift. The failure appeared over six months, which suggests the defect population changed. Whatever rule replaces this one should have its error rates tracked on continuing labelled samples, since no fitted boundary reports its own obsolescence. What I would tell the team. The original figure was measured on the data that produced the rule and on a class balance that does not exist in production, so it never supported deployment. The deeper problem is that the defective class surrounds the sound one rather than sitting beside it, visible in the near-identical means and the sixteen-to-one spread ratio, so no straight boundary was ever going to work, and the six months of apparent success were a property of the particular defects sampled rather than of the method.