Practice: Trees and Ensembles of Them

1. The first split. At the left group is $\{1,2,1,2\}$ with mean $1.5$, and the right group is $\{6,7,6,7\}$ with mean $6.5$. Squared deviations are $0.25\times 4=1$ on each side, so the total residual sum of squares is $2$. To know this is the selected split, every other candidate boundary must be scored the same way, the midpoints between adjacent hour values, so 15, 25, 35, 45, 55, 65 and 75, and the minimum taken. The unsplit total is $\sum (y_i-4{)}^2=58$, so this candidate removes $56$ of it. Any candidate that leaves a low and a high group mixed will score far worse; the only competitive ones would be those separating the same two groups. 2. Prediction. Both machines fall in the right leaf, so both receive $6.5$. They have the same prediction because the fitted function is piecewise constant: within a region every input gets that region's training mean, and the 23-hour difference between them carries no information the tree can use. A tree distinguishes inputs only where it has placed a split. 3. Stability. The tree establishes that these eight observations separate cleanly into a low group and a high group, and that the boundary lies somewhere between 40 and 50 hours. It does not establish 45. That value is the midpoint convention applied to the gap between the last low observation and the first high one; there are no observations between them, so every threshold in that interval fits these data identically. Reporting "faults rise sharply after 45 hours" states a precision the data do not contain. The defensible claim is that the rise occurs between 40 and 50 hours. 4. Whether to average. Bagging reduces variance and does not reduce bias. These trees have almost no variance to remove: the two groups are separated by a gap of four faults with no overlap, so nearly every bootstrap resample produces the same split and the same two leaf means. Averaging 500 near-identical trees returns approximately the single tree, at 500 times the computation. The deciding property is how much the trees disagree. Bagging pays when individual trees are unstable, deep trees on noisy data with competing candidate splits, and pays nothing when they already agree. A useful check is to refit on a few resamples and see whether the structure moves before committing to an ensemble. 5. A forest. A random forest restricts the candidate predictors at each split to a random subset, so with ten predictors it might consider three or four at each node. The strongly informative predictor is then unavailable at most splits, and different trees are forced to open on different predictors. The cost is that each individual tree fits worse: denied the best predictor, it makes a weaker split. The trade can still favour the ensemble because the quantity being minimised is the variance of the average, which is $\rho {\sigma }^2+(1-\rho ){\sigma }^2/B$. Once $B$ is moderately large the second term is small and the floor is $\rho {\sigma }^2$. Under plain bagging every tree would open on the same strong predictor, making $\rho$ high and the floor high with it. Lowering $\rho$ moves the floor itself, and that gain routinely exceeds the loss from weaker individual trees. At $\rho =0.2$ and $B=100$ the variance is $0.208{\sigma }^2$; at $\rho =0.05$ it is $0.060{\sigma }^2$, a far larger improvement than any number of additional trees could deliver at the higher correlation.