Practice: Shrinkage, and the Trade It Makes

(a) The determinant and the correlation.

The correlation between the columns is

A determinant of $0.19$ against diagonal entries near $10$ means $X^{\mathsf{T}}X$ is nearly singular, and a correlation of $0.999032$ says the two predictors carry almost the same information. These are the same fact stated two ways. (b) Why it is neither an error nor an effect. It is not an arithmetic error because $(+2.605263,-0.631579)$ genuinely minimises squared error on this data, recomputation confirms it, and it would confirm it however many times the analyst repeated the calculation. The procedure did what it is defined to do. It is not evidence that $x_2$ acts negatively because the coefficient is not describing $x_2$ on its own. When two predictors are nearly identical, many coefficient pairs fit almost equally well: $3x_1-x_2$ and $x_1+x_2$ are nearly the same vector, so least squares has no substantive basis for choosing between them and settles the tie on whichever matches the noise marginally better. The negative value is the second half of a cancelling pair, and it is a property of the pair rather than of the predictor. Mechanically: solving requires inverting a matrix with determinant $0.19$, and dividing by a small number magnifies small features of the data into large coefficients. The instability is real, the arithmetic is right, and the interpretation is what fails. (c) What the data determines. The sum $+1.973684$ is close to the true total effect of about $2$, while neither individual coefficient is close to anything sensible. So the data determines the joint contribution of $x_1$ and $x_2$ well, and says almost nothing about how to divide that contribution between them. This matters for what the model can be used for. A prediction uses the coefficients only through the combination ${\beta }_1{x}_1+{\beta }_2{x}_2$, which is well determined, so predictions from this fit may be perfectly serviceable. Reading either coefficient as the effect of its predictor uses exactly the part the data cannot resolve. The instability is therefore not uniform across everything one might want to know, and whether it is a problem depends on which of the two is wanted.

(a) The displacements.

Least squares:

Ridge:

Ratio:

The unpenalised coefficients move almost ten times as far in response to the same change in one observation.

(b) What it estimates.

It estimates variance, how much the fitted coefficients depend on the particular sample rather than on the underlying relationship. A different draw from the same process would give somewhat different responses, and this comparison shows how much the answer would move in consequence.

A single fit cannot reveal it because a single fit produces one number per coefficient, with nothing to compare against. $+2.605263$ does not announce that it would have been $+2.773684$ had one observation differed slightly. Variance is a property of the procedure across datasets, so it becomes visible only when the data is varied and the procedure rerun, by perturbation, as here, or by resampling.

(c) Correcting the accuracy claim.

The comparison says nothing about accuracy. Both fits were computed from the same data and compared with each other, not with any true value.

On accuracy the evidence points the other way. The unpenalised fit minimises squared error on this data by construction, so ridge is strictly worse there. And the ridge coefficients sum to $1.892670$ where the unpenalised pair sums to $1.973684$ and the underlying signal totals about $2$, so shrinkage has pulled the estimated total further from the truth. That is bias, and it is the price rather than a side effect.

What the comparison shows is stability: the penalised estimate depends far less on which particular observations were collected. The summary is a trade: variance down by a factor of about ten, bias up, training fit worse, and whether it was worth making is not decidable from these figures, since every one of them comes from the same five observations. That question needs error measured on data the fit has not seen.

(a) It will select $\lambda =0$, whatever the data. Ordinary least squares is defined as the minimiser of the residual sum of squares on the fitting data. Every minimises a different objective, squared error plus a penalty, so its solution cannot beat the least-squares solution on squared error alone. Residual sum of squares therefore rises monotonically with $\lambda$, and the minimum is at $\lambda =0$ by construction. The answer is fixed before the data is seen. A selection procedure whose outcome is determined in advance is not selecting anything, and it returns precisely the unpenalised estimator the penalty existed to avoid. (b) The two errors. First: selecting $\lambda$ by training error. As above, this cannot work even in principle. The criterion has no interior minimum, so it cannot express a preference for any positive penalty. Second: reporting performance on the data used to fit and select. Even with a valid selection rule, quoting the error on data that participated in choosing $\lambda$ gives an optimistic figure. The parameter was chosen to make that number small, so the number no longer estimates performance on new data. These are separate mistakes: the first makes the choice impossible, and the second would remain wrong even after the first was fixed. (c) A procedure that answers both. Split the data three ways, or use nested resampling: 1. Hold out a test set and set it aside untouched.
2. On the remainder, use $k$-fold cross-validation: for each candidate $\lambda$, fit on $k-1$ folds and measure error on the held-out fold, averaging over folds. Held-out error falls and then rises as $\lambda$ grows, so it has an interior minimum and can express a genuine preference.
3. Choose the $\lambda$ minimising cross-validated error, or the largest $\lambda$ within one standard error of that minimum if a simpler model is wanted, and say which rule was used, since they give different answers.
4. Refit at the chosen $\lambda$ on all the non-test data.
5. Report performance on the test set, which took no part in fitting or selection. The structure matters: step 2 chooses, step 5 measures, and they use different data because a dataset used to choose a parameter can no longer give an unbiased estimate of the result. One caveat worth adding. Cross-validation estimates the performance of the procedure, and the selected $\lambda$ can vary across splits. Reporting the chosen value as though it were a stable property of the problem overstates what the resampling established.

(a) What each model asserts. Least squares asserts that sensor 1 has a strong positive effect of $+2.605263$ and sensor 2 a negative effect of $-0.631579$. Given that both measure the same physical quantity, this is not credible: two readings of one thing cannot act in opposite directions. The pair is a cancelling artefact of near-singularity, not a description of the sensors. Ridge asserts that each sensor carries roughly half the effect, $+1.003705$ and $+0.934853$. This is credible and is arguably the most honest of the three: the underlying quantity has one effect, both sensors measure it, and neither reading is privileged. The split reflects a genuine symmetry in what is known. Lasso asserts that sensor 1 carries the entire effect of $+1.929397$ and sensor 2 contributes nothing. As a prediction rule this is fine; as a statement about the sensors it is misleading, since sensor 2 measures the quantity just as well and its exclusion reflects the optimisation breaking a near-tie rather than any difference between the instruments. Had the sample differed slightly, the lasso might well have kept sensor 2 and dropped sensor 1. (b) Why closest-to-the-truth does not settle it. The lasso total $1.929397$ is nearer $2$ than ridge's $1.938558$, but the gap is $0.009161$, far smaller than the uncertainty in any of these estimates from five observations. Treating that as a ranking reads more into the figures than they carry. More importantly, it is a fact about this sample. The comparison was computed on the same data that produced the fits, so it measures which estimator happened to land closer here, not which is generally better. A different draw could reverse it. And the criterion is the wrong one for the stated purpose. If the deliverable is a prediction rule, both are fine and the choice can rest on held-out error. If it is a statement about the sensors, the question is which description is true, and "both sensors measure the quantity, each accounting for part of the fitted effect" is true, while "sensor 2 has no effect" is false. (c) What to report. Report the sum, about $1.93$ to $1.97$ depending on the estimator, as the effect of the underlying quantity, with the explicit statement that it is the combined contribution of the two sensors and that the data does not determine how to divide it between them. The justification is in the fits themselves: the individual coefficients swing wildly across the three models, from $-0.631579$ to $+1.929397$ for the same sensor, while the sums agree to within about $0.05$. What is stable is what the data determines, and what is unstable is what it does not. Also report that the two sensors correlate at $0.999032$, so a reader can see why the split is unavailable; state which penalty was used and at what strength; and if the penalised coefficients are shown, note that they are biased toward zero by construction and so understate the effect slightly. Presenting any single coefficient as "the effect" would report a number that the next sample would contradict.

(a) Zero training error is the warning, not the result. With $p=\mathrm{10,000}$ predictors and $n=400$ observations, $X^{\mathsf{T}}X$ is a $\mathrm{10,000}\times \mathrm{10,000}$ matrix of rank at most $400$, singular, not merely near-singular. Least squares has no unique solution: infinitely many coefficient vectors reproduce the training responses exactly, and the fitting software returned one of them arbitrarily. A perfect fit here carries no information about the data. It is guaranteed by the shape of the problem whenever $p\ge n$, and would have appeared just as readily had the predictors been random numbers. Least squares can deliver a fit; it cannot deliver an estimate. This is the extreme of the situation the unit's small design illustrates. There, two predictors correlating at $0.999032$ gave $det(X^{\mathsf{T}}X)=0.19$ and coefficients of $+2.605263$ and $-0.631579$. A cancelling pair. Here the determinant is exactly zero and the arbitrariness is total. (b) Their selection procedure will return $\lambda =0$. Residual error on the fitting data is minimised by ordinary least squares by definition, and rises monotonically as $\lambda$ grows, because every minimises a different objective. So the rule as stated selects no penalty at all. The answer is fixed before any data is examined. That this did not visibly happen suggests something else occurred: perhaps the search grid excluded zero, or the software's default cross-validation was used and the report describes it incorrectly. Either way the procedure as written cannot have produced a meaningful $37$-predictor model, and the discrepancy needs resolving before anything else in the report can be trusted. (c) The $37$ are not the true drivers. The lasso returns a set of predictors sufficient to predict well, not the set that matters. Where predictors are near-duplicates, it keeps roughly one from each redundant group and zeroes the rest, and which one survives is close to arbitrary, decided by whichever matches the sample noise marginally better. On the unit's two-predictor design this is visible in miniature: the lasso zeroes the second predictor from $\lambda \approx 0.006030$ onward, though both sensors measure the same quantity equally well. Nothing distinguishes them except the sample. So "months at address" surviving while "months since last move" is dropped is not evidence that the first drives default and the second does not. A different $400$ applicants could easily reverse it. The right check is refitting on resamples and recording how often each predictor is selected; a variable chosen in $30\mathrm{\%}$ of resamples is not a finding. The recommendation to collect only these $37$ inherits the problem. If a dropped near-duplicate is the one that remains measurable in future, discarding it on this evidence is a costly mistake. (d) The coefficients cannot be interpreted as effects. Three reasons, each sufficient. First, lasso coefficients are deliberately biased toward zero. On the unit's design, ridge at $\lambda =1$ shrinks the coefficient sum from $1.973684$ to $1.892670$ when the underlying signal is about $2$. Penalised coefficients systematically understate effects, by design. Second, a coefficient in a model with near-duplicate predictors describes the predictor's contribution given the others in the model, and with duplicates that contribution is not identifiable. The unit's design shows the point sharply: the same sensor takes coefficients from $-0.631579$ to $+1.929397$ across three fits, while the sum stays near $1.93$–$1.97$. The data determines combinations, not individual shares. Third, these are observational credit records. Even a perfectly estimated coefficient would be an association, and calling it "the effect on default risk" asserts a causal claim no regression on this data supports. A claim with consequences, given that the model will be used to decline applicants. (e) What to do instead. Structure the evidence properly. Hold out a test set and touch it once. On the remainder, use $k$-fold cross-validation to choose $\lambda$: held-out error falls and then rises, so it has an interior minimum and can genuinely express a preference, which training error cannot. Refit at the chosen $\lambda$ on all non-test data and report performance on the test set alone. If the selected $\lambda$ varies substantially across folds, say so. Choose the penalty by the deliverable. If the goal is a cheaper data-collection pipeline, the lasso's sparsity is the right tool, accompanied by a selection-stability analysis over resamples, reporting selection frequencies rather than a list of $37$ names. If the goal is the best risk prediction from data already collected, ridge is usually preferable with grouped near-duplicates, since it keeps correlated predictors and splits the signal rather than picking arbitrarily among them. The elastic net, combining both penalties, is the standard compromise and selects groups together instead of one member. Whichever is chosen, the decision should be stated with its reason. Quantify the stability gained. Refit on perturbed or resampled data and report how far the coefficients move, with and without the penalty. On the unit's design the displacement falls from $0.269605$ to $0.027302$, a factor of $9.875079$. A magnitude a reader can weigh, unlike an assertion that the penalty stabilises the fit. Report the trade, both halves. The penalised model fits the training data worse, necessarily, and its coefficients are biased toward zero. What it gains is that the answer depends far less on which $400$ applicants happened to be in the sample. State both, and state that no figure computed from the fitting data can settle whether the exchange was worthwhile. Separate the claims. "These predictors give the best held-out prediction we could obtain" is supportable. "These are the drivers of default, and their coefficients are the effects" is not, and in a lending context the difference is not academic: the second invites decisions about individual applicants that the evidence does not license.