Practice: Fitting Without a Formula

1. The fitted value. For $k$-nearest neighbours: measure the distance from the query point to every calibration reading, take the $k$ smallest, and average their responses. Weights are equal inside that neighbourhood and zero outside it, so the fitted curve jumps as the query point moves and an observation enters or leaves. For a kernel fit: weight every reading by $K_h$, a function falling with distance, and form $\sum _i{w}_i{y}_i/\sum _i{w}_i$. Weights never reach zero, so the curve is continuous. The kernel version also reports the effective number of observations behind each estimate, ${\left(\sum _i{w}_i\right)}^2/\sum _i{w}_i^2$. That is the quantity the nearest-neighbour fit conceals: it always supplies exactly $k$ neighbours, whether they are close by or scattered across the range. 2. The colleague's selection. At $k=1$ every training reading is its own nearest neighbour, so each prediction equals the observed value and the training residual sum of squares is exactly zero. No other $k$ can match that, so minimising training error selects $k=1$ every time, on any dataset. The criterion is monotone in the parameter being chosen and therefore cannot choose it. What the curve has done is record the 200 readings, including their measurement noise. Collect the 200 readings again and the entire curve moves, because each fitted value is a single noisy measurement rather than an estimate combining several. That is the point of maximum variance, reached by the criterion that reports perfection. 3. Choosing $k$ properly. Cross-validated error, and leave-one-out is natural here because excluding an observation from its own neighbourhood costs nothing. Compute the held-out error across a grid of $k$ and take the minimum. The curve should be U-shaped. At small $k$ the variance term dominates and the error is high; as $k$ grows the noise is averaged down and the error falls; past some point the neighbourhood reaches into regions of genuinely different concentration, bias rises, and the error climbs again. I would also look at how flat the minimum is: a flat one means several values are equivalent and the choice does not matter much, a sharp one means it does. 4. The ends of the range. Near 5 units every neighbour lies above the query point, and near 90 every neighbour lies below. Since the relationship is monotone, those one-sided neighbourhoods have systematically different expected responses, so the fit is pulled toward the interior at both ends, too high at the bottom and too low at the top. More readings between 5 and 90 would not fix it. The lowest query point still has no readings below it however many are collected, so the neighbourhood remains one-sided. This is bias with a fixed direction, not noise that averages away. What would help is calibration standards below 5 and above 90, which changes the range rather than the density. 5. The reading at 120 units. There are no readings above 90, so the $k$ nearest to any query beyond it are the same $k$ readings at the top of the calibrated range. The fit returns their average, the same constant for 120, for 200, for any value out there, and returns it with no indication that the query is outside the data. That value should not be reported as a concentration. The reading is out of calibrated range and should be reported as such, exactly as a laboratory reports an out-of-range result rather than converting it. This is the cost of assuming no functional form: a parametric calibration curve would at least extrapolate in a direction that could be argued about from the instrument's physics, while local averaging has nothing to extrapolate from. Extending the range needs standards at higher concentrations, not a different fit.