Fitting Without a Formula

Local averaging estimates $E[Y\mid X=x]$ from observations near $x$, without assuming a functional form. Under regularity conditions, including a smooth regression function, a bandwidth shrinking at an appropriate rate, and enough data near $x$, the estimate is consistent. The cases below differ in what that costs.

A smooth curve of unknown form. A response rising and then levelling off, with no theory fixing whether the curve is logarithmic, a saturating exponential, or something else. Averaging neighbours follows the shape without committing to a form. This is the case the method is designed for.

A relationship known to be linear. A local average can approximate a line, but it re-estimates the fit at every point from the observations in a neighbourhood, while least squares uses all $n$ observations to estimate two parameters. For comparable precision the local method needs more data, and it returns no slope coefficient to report.

A discontinuity. Where the regression function has a jump, any neighbourhood spanning the threshold averages across it, so the fitted curve shows a transition whose width is roughly the neighbourhood width rather than a step. The output contains no indication that this has happened. A regression tree can place a split at the threshold, so a step function is within the class of functions a tree can represent; whether a tree fitted to a particular sample locates the threshold accurately depends on the sample and on how the tree is grown and pruned.

Several predictors. With one predictor a neighbourhood holding $10\mathrm{\%}$ of the observations spans about $10\mathrm{\%}$ of the range. With ten predictors, holding $10\mathrm{\%}$ of the observations requires spanning about ${0.10}^{1/10}\approx 80\mathrm{\%}$ of each variable's range, so the neighbourhood is no longer local. Local averaging suits problems with few predictors.

Sparse regions. Where observations thin out, a $k$-nearest-neighbour estimate draws on points far from $x$ and reports an average whose stated precision does not reflect that distance, while a fixed-bandwidth kernel estimate is visibly unstable because it uses few points. The second makes the data shortage apparent in the output.

Local averaging is efficient where the regression function is smooth, its form is unknown, and the predictors are few. It is less efficient than a parametric fit when the form is known, and it is biased near discontinuities and at the edges of the observed range, in both cases without a signal in the output.

Training error against held-out error, on the same fits.

The two columns rank $k=1$ oppositely: best by the first, close to worst by the second. Training error is monotone in $k$ and held-out error is U-shaped, so only one of them contains the information needed to choose. The disagreement is largest exactly at the setting a reader is most tempted by.

$k$-nearest neighbours against a kernel, at matched flexibility.

At comparable flexibility the two agree closely on the estimate and differ in what they do when the data thin out. Neither behaviour is safer in general; the kernel's instability is at least legible.

Bandwidth against neighbourhood size.

Both control smoothing, and they respond to the data's density oppositely. $k$ fixes the number of observations and lets the width float; $h$ fixes the width and lets the count float. In a dense region they behave alike. In a sparse one, $k$-NN quietly averages distant points while the kernel returns a noisy estimate from two or three.

A local average against a regression tree.

Both are local and neither assumes a form. A tree partitions by searching for splits, so it represents a genuine threshold exactly and a smooth trend as a staircase. Local averaging assumes smoothness, so it represents a smooth trend well and a threshold as a ramp whose width is the neighbourhood's. The two fail on each other's easy cases.

A smoothing spline at large $\lambda$ against ordinary least squares.

These converge: as $\lambda \to \mathrm{\infty }$ the curvature penalty forces $f^{″}=0$ and the spline becomes the least-squares straight line. The nonparametric family contains the parametric fit as its most heavily smoothed member, which is the precise sense in which assuming a form is the strongest smoothing available, and why the choice between them is a matter of degree rather than of kind.

That a local fit using fewer neighbours is more accurate, because it uses only the observations closest to the query point and so is less contaminated by distant ones. Shrinking the neighbourhood lowers bias and raises variance, and the sum has a minimum at some intermediate size. At $k=1$ the fit passes through every training observation and its training error is exactly zero, which looks like perfect accuracy and is instead the point of maximum variance: the prediction at any query point is one noisy observation, so a resample of the data moves it by the full noise standard deviation. Training error cannot reveal this because it falls monotonically as the neighbourhood shrinks, which is why the parameter is chosen by held-out error.

That a nonparametric fit makes no assumptions, so it is the safe choice whenever the shape of a relationship is unknown. What is dropped is the assumption of a particular functional form; several assumptions remain, and one of them is strong. Local averaging assumes the response varies smoothly enough that observations near a query point are informative about it, which is why the method fails at a genuine discontinuity and why it degrades near the boundary of the observed range, where the available neighbours lie on one side only. It also gives up what a parametric form provides: coefficients that can be reported, and any basis for predicting outside the observed range, where there are no neighbours to average. The choice is a trade rather than a removal of assumptions.