Hyperparameter Tuning

A hyperparameter sets a rule for making predictions. For a nearest-neighbor regressor, k is the number of training targets to average. Choosing k by training error alone can reward a model that merely reproduces its own examples. A separate validation set lets us compare candidates using predictions at other inputs. This module makes that comparison concrete: four candidates, seven fixed data points and a search budget you can replay.

One prediction rule, two different scores

The training pairs are (0, 0), (2, 0), (4, 8) and (6, 8), in that original order. At a query x, rank the training points by absolute distance |x−x_i|, breaking equal distances by the smaller original training index. The prediction is the uniform average of the first k targets. Validation targets never become neighbors.

For m scored examples, mean squared error is MSE = (1/m) Σ(y−prediction)². Training MSE here uses the four training inputs as explicit queries, including each point itself among its neighbors. Validation MSE uses the held-out pairs (1, 0), (3, 4) and (5, 8). This training score is resubstitution error, not leave-one-out error.

Why a higher training error can choose a better candidate

Take the validation query x = 3. The original training indices are ranked 1, 2, 0, 3, with distances 1, 1, 3, 3. For k = 1, index 1 wins the first tie, so the prediction is the target at x = 2: zero. For k = 2, the prediction averages the targets at x = 2 and x = 4: (0+8)/2 = 4. That exactly matches the validation target at x = 3.

k	Validation predictions at x = 1, 3, 5	Validation MSE	Training MSE
1	0, 0, 8	16/3	0
2	0, 4, 8	0	4
3	8/3, 8/3, 16/3	16/3	64/9
4	4, 4, 4	32/3	16

For k = 1, only the middle validation prediction is wrong: (0²+4²+0²)/3 = 16/3. Its zero training error comes from predicting each training target with that point itself. For k = 2, the training predictions are (0, 0, 4, 8), giving training MSE = 16/4 = 4. A higher training error accompanies the lower validation error here.

For k = 3, squared validation errors are 64/9, 16/9 and 64/9. Their mean is 144/27 = 16/3, exactly the same score as k = 1. The module compares those fractions before rounding and uses smaller k to settle equal scores. A decimal display of 5.333333 is only a rounded view of that exact tie.

A budget changes what is known

Grid order is 1, 2, 3, 4. With a budget of one, k = 1 is the best evaluated candidate; k = 2 is not eligible yet. With two trials, k = 2 wins. Seeded shuffle with seed 21 instead produces 3, 1, 4, 2. Its first incumbent is k = 3. After the second trial, k = 1 wins the exact tie. It stays incumbent after the third trial, and k = 2 wins only on the fourth.

The shuffle uses descending Fisher–Yates with a deterministic mulberry32 generator. It visits every candidate once, without replacement. A seed and candidate set define a complete order first; choosing a different budget only exposes a different prefix. Neither a candidate’s predictions nor an earlier trial’s score depends on how many future trials you permit.

Spend, inspect and rewind a real search

Open the Hyperparameter Tuning interactive module. On narrow screens the controls scroll above the results; the trial and neighbor tables have their own horizontal scroll. Compare the cards with the numerical tables, rather than estimating a score from bar height.

Press Reset search. Grid order is 1→2→3→4, Evaluated trials is 0, and Best k so far is None. The fixed training and validation data are still available. No untested candidate is presented as a result.
Press Next trial. Confirm best k = 1, validation MSE = 16/3 and training MSE = 0. At the default Validation query x = 3, inspect the ranked indices 1, 2, 0, 3 and prediction 0. Change the query to x = 5: the first index is 2 and prediction is 8.
Press Next trial again. Best k becomes 2, validation MSE becomes 0, and training MSE becomes 4. The three purple prediction bars now match the gray validation targets. Select Inspect candidate → k = 1: its earlier errors return in the inspector, while the best-result card stays k = 2.
Press Finish budget, then inspect k = 4. Its validation predictions are 4, 4, 4 and MSE is 32/3. The incumbent remains k = 2. Set Trial budget to 1: only the first trial remains eligible and the removed inspection falls back to Best so far. Increase the budget to 4; the evaluated count stays 1 until you request another trial.
Select Search order → Seeded shuffle and set Shuffle seed to 21. Each edit starts at zero evaluated trials. Press Next trial twice: the order is 3→1→4→2, and k = 1 wins the exact 16/3 tie. Set Evaluated trials to 3 and then 4 to see that k = 2 becomes eligible only at the final trial.
Keep shuffled order, enter seed 42, set budget to 2 and press Finish budget. Best k is 1. Raise the budget to 3: the two past scores remain unchanged. Press Next trial to admit k = 2 and obtain zero validation MSE. Press Previous trial to remove that candidate and restore the earlier incumbent.
Set Evaluated trials to 0 before trying playback. Set Playback speed to 2, press Play, then Pause before the budget is complete. Play advances one trial per half-second and stops at the budget. Manual query, inspection and speed changes pause without changing evaluated scores; changing order or seed starts a new search. Resize the page and confirm the chosen experiment remains intact.
Enter 1e1 in Shuffle seed. The text stays visible with an error and old search results disappear. Replace it with 001, a valid spelling of seed 1. The new order is 4→2→1→3 with zero trials evaluated. Reset search returns every control to its default.

What this small holdout can and cannot establish

This is an actual finite kNN search with deterministic predictions, not a simulated epoch curve or a provider training API. All candidates use the same four training examples and the same three validation examples. The feature is one-dimensional, distances are unscaled absolute differences, and every selected neighbor has equal weight. Other distance metrics, feature scaling, distance weighting or tie rules can change the predictions.

Picking a winner after repeatedly inspecting the same tiny holdout can overfit that holdout. A zero validation error on three hand-chosen examples does not establish zero error on unseen data, nor does it make k = 2 universally best. This experiment uses one fixed split, not cross-validation. Independent testing or a carefully designed outer validation procedure is needed for a separate performance estimate after selection.

Training resubstitution error is deliberately shown beside validation error. At k = 1, self-inclusion makes training error zero even though validation error is positive. When libraries are asked for neighbors without an explicit query array, they can exclude a point from its own neighbors. This module always uses explicit query values, so that alternate API convention does not apply.

Four uniformly weighted candidates do not demonstrate that random search is always superior to grid search. With seed 1, shuffle evaluates k = 2 on its second trial, as grid does. With seed 21, it takes four trials. A full budget evaluates the same set in both orders and therefore obtains the same final winner. There is no early-stopping model, changing training budget or hidden optimization run here.

The seed accepts one to three ASCII digits with value 1–100. Leading zeros stay visible; whitespace, signs, decimal points, exponents and a final newline are rejected. Invalid text is never converted into a fallback result. The playback budget is limited to four trials. Scores use integer numerators and denominators small enough for exact binary64 integer comparison; displayed decimals round to six places. The original training index determines equal-distance neighbors, and smaller k determines an exact score tie. These are explicit reproducibility conventions rather than properties that every library must share.

Official references and the original example

scikit-learn, Tuning the hyper-parameters of an estimator explains the estimator, parameter space, search method and scoring rule, and distinguishes exhaustive grid from randomized searches. The four-candidate without-replacement shuffle and the numerical data in this article are the module’s own bounded experiment.
scikit-learn, KNeighborsRegressor documents uniform neighbor averaging and warns that a boundary distance tie can make results depend on training-data order. Its kneighbors documentation also distinguishes explicit queries from the self-excluding query=None convention. This module states its index tie rule and self-including training queries explicitly.

For a different question, Cross-Validation changes which observations form a training/validation split. Here the split is fixed and k changes. Keeping those decisions separate makes it easier to explain what a search score actually measures.

Hyperparameter Tuning