Ask the people most like you
Say you want to guess whether a new student passed. The simplest honest thing you can do is find the students most similar to them — similar hours studied, similar attendance, similar past marks — and see how those students did. That is the whole idea. We call those similar past students the neighbours, and KNN reads the answer straight off them.
When the answer is a category
The neighbours vote. If most of the students most like this one passed, the guess is "passed". The most common answer among the neighbours wins.
When the answer is a number
The neighbours are averaged. To guess a new student's final score, take the scores of the most similar students and average them.
Notice there is no clever formula to learn here. KNN simply keeps the training data and looks things up when asked, a bit like flipping back through old records. That is refreshingly simple, though it does mean predictions slow down as the pile of records grows.
"Most similar" only makes sense if the columns are on a fair footing
To decide who is "similar", KNN measures distance across your feature columns. But imagine one column is hours studied (roughly 0 to 10) and another is total minutes on the study app (which could run into the thousands). The big column would drown out the small one entirely, just because its numbers happen to be larger — not because it matters more.
Scaling means rewriting each numeric column onto the same scale first, so every column gets a fair say. This lesson does that for you inside the pipeline, and — this is the careful part — it learns the scaling from the training rows only, never from the rows it will be tested on.
A quick safety check before we begin
Colab sometimes forgets everything if it has been idle or restarted. This first cell is just a friendly nudge that confirms the six Start cells are still in memory — it works the same whether you are predicting a category or a number. If it complains, no harm done: click Runtime → Run all to re-run the whole notebook, then come back and try again.
What this does: checks that the names the Start cells created are still available, and prints your task and data sizes so you know it is safe to continue.
required_names = [
"TASK", "RANDOM_STATE",
"X_train", "X_test", "y_train", "y_test",
"make_preprocessor", "make_cv", "CV_FIT_PARAMS",
"PRIMARY_SCORING", "evaluate_predictions", "results_log",
]
missing_names = [name for name in required_names if name not in globals()]
if missing_names:
raise RuntimeError(
"Run the foundation cells first. Missing: "
+ ", ".join(missing_names)
)
if TASK not in {"classification", "regression"}:
raise ValueError("KNN needs TASK to be classification or regression.")
print("Task:", TASK)
print("Training shape:", X_train.shape)
print("Testing shape: ", X_test.shape)
print("The test set stays closed while K is selected.")A short printout naming your task (classification or regression) and the number of rows in your training and test sets — for the example students that is a few hundred rows, but your own data will differ. If nothing prints and you instead see a red error asking for the Start cells, that is the safety check doing its job: run the notebook top to bottom and return. If the shapes look wrong or surprising, go back to the split cell rather than working around it here.
How many neighbours should you ask?
The K in K-Nearest Neighbours is simply how many of the most similar past cases you consult. Ask just one neighbour and you copy whatever that single closest case did. Ask twenty-one and you go with the mood of a small crowd. Neither extreme is automatically right, so it helps to picture what each end feels like.
A small K listens closely to the one or two nearest cases, so it reacts to every quirk. A large K averages over a bigger group, so it is calmer but can smooth away a genuine local pattern. There is no single right K — you will let the training data suggest one in a moment.
| Choice | Likely behaviour | Question to ask |
|---|---|---|
| Small K | Flexible and sensitive to individual rows | Is the model following noise or unusual duplicates? |
| Large K | Smoother and pulled toward the common answer | Is a real local pattern being averaged away? |
So rather than guess, you will try several values of K using only the training data and keep whichever performs best. The real test stays sealed the whole time.
Going deeper (optional): why distance gets confused with too many columns
When you have a great many feature columns, a strange thing happens: almost every row starts to look about equally far from every other row, so "nearest" stops meaning much. This is often called the curse of dimensionality. ID-like columns, categories with hundreds of possible values, and piles of weak columns can make a nearest-neighbour model both slow and unhelpful. In practice, choosing a handful of columns that genuinely matter helps far more than searching over a wider range of K.
Assemble the model on an assembly line
Think of a pipeline as a small assembly line. Your data goes in one end, gets its missing values filled and its categories tidied up (using the tools the Start cells built), then gets scaled onto a fair footing, and only then reaches the KNN model at the far end. Bundling every step this way is what keeps the scaling honest: the steps only ever learn from training rows, so the rows we test on later cannot secretly influence anything.
What this does: picks the classifier or the regressor to match your task, then wires it onto the assembly line right after the preparation steps.
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import (
KNeighborsClassifier,
KNeighborsRegressor,
)
if TASK == "classification":
knn_estimator = KNeighborsClassifier()
else:
knn_estimator = KNeighborsRegressor()
knn_pipeline = Pipeline([
("prepare", make_preprocessor(scale_numeric=True)),
("model", knn_estimator),
])
knn_pipelineA tidy diagram of the assembly line: a prepare step, then the KNN model that fits your task. Nothing has been trained yet, and that is on purpose — the next cell is the one that does the actual fitting, and only on training rows.
Let the training data suggest a good K
Here is where you find your K without ever peeking at the test set. The trick is called cross-validation.
Cross-validation means: take only your training rows, hide a slice of them, train on the rest, and see how well each K guesses the hidden slice. Rotate which slice is hidden and repeat. It is like giving each candidate K a few practice exams before the real one. You keep the K that does best across the practice exams, and only THEN do you sit the real test. The list of K values below even shrinks itself for small datasets, so you never ask for more neighbours than there are rows to offer.
What this does: quietly runs those practice exams for several candidate K values on the training rows, then prints a small table and names the K that scored best.
cv = make_cv()
# Find the smallest training fold, without touching X_test.
if "groups" in CV_FIT_PARAMS:
cv_splits = cv.split(
X_train,
y_train,
groups=CV_FIT_PARAMS["groups"],
)
else:
cv_splits = cv.split(X_train, y_train)
fold_train_sizes = [
len(fold_train_index)
for fold_train_index, _ in cv_splits
]
largest_safe_k = min(31, min(fold_train_sizes))
candidate_k = [
k for k in [1, 3, 5, 7, 9, 15, 21, 31]
if k <= largest_safe_k
]
if not candidate_k:
raise ValueError("There are too few training rows to fit KNN.")
knn_search = GridSearchCV(
estimator=knn_pipeline,
param_grid={"model__n_neighbors": candidate_k},
scoring=PRIMARY_SCORING,
cv=cv,
n_jobs=-1,
return_train_score=True,
error_score="raise",
)
knn_search.fit(X_train, y_train, **CV_FIT_PARAMS)
raw_cv = pd.DataFrame(knn_search.cv_results_)
if TASK == "classification":
cv_summary = pd.DataFrame({
"k": raw_cv["param_model__n_neighbors"].astype(int),
"mean_validation_balanced_accuracy": raw_cv["mean_test_score"],
"fold_to_fold_variation": raw_cv["std_test_score"],
"rank": raw_cv["rank_test_score"],
})
else:
cv_summary = pd.DataFrame({
"k": raw_cv["param_model__n_neighbors"].astype(int),
"mean_validation_MAE": -raw_cv["mean_test_score"],
"fold_to_fold_variation": raw_cv["std_test_score"],
"rank": raw_cv["rank_test_score"],
})
display(cv_summary.sort_values("k").reset_index(drop=True))
print("Selected K:", knn_search.best_params_["model__n_neighbors"])One row per candidate K, plus the single K that was chosen. For a category, the table shows balanced accuracy, and higher is better. For a number, it shows MAE, and lower is better. Do not worry if several K values score about the same — that just means the exact winner is a bit of a coin toss, which is useful to know. Your own numbers will differ from any example.
Good news: the real test is still sealed
All of that practice used only your training rows — the test set was never opened. Where your split respects groups (say, keeping all rows for one person together), that boundary was honoured too. At the end, the search quietly re-trained the winning setup on all your training rows, so it is primed and ready for a single, honest test.
- Does the selected K suggest a very local rule or a broad average?
- How much do validation scores vary from fold to fold?
- Would a time or group-aware cross-validation scheme be more realistic for your project than the foundation helper?
Now, and only now, open the test set
Your model and your K are settled. This is the moment you have been protecting: you make predictions on the test rows once and hand them to the same scoring helper that earlier recorded your baseline. The baseline is your sanity check: if KNN cannot beat a lazy always-guess-the-common-answer model, it is not earning its keep.
What this does: predicts on the held-out test rows once and records how KNN did, right alongside the baseline for comparison.
best_knn = knn_search.best_estimator_
knn_predictions = best_knn.predict(X_test)
knn_test_result = evaluate_predictions(
"KNN",
y_test,
knn_predictions,
show_details=(TASK == "classification"),
)Your results log now lists both the baseline and KNN side by side. On the example students, KNN lands around 76% accuracy against a baseline near 64% — or, if you are predicting a number like the final score, an R-squared around 0.53. Treat those as ballpark only; your own data will land somewhere different. The question that matters is not whether you match someone else's number — it is whether this lift is enough to change what you would actually do. For a category, glance at the report and confusion matrix too, to see which kinds of mistakes KNN tends to make.
Look at where it actually went wrong
A single score tells you how often the model is right, but not where it stumbles — and that is usually the more useful thing to know. This last cell reuses the predictions you already made; it does not train anything new or go hunting for a better setting. It just holds the mistakes up to the light.
What this does: lines up each test row's real answer against KNN's guess and surfaces the biggest misses so you can inspect them.
outcomes = pd.DataFrame(
{
"__actual__": np.asarray(y_test),
"__predicted__": np.asarray(knn_predictions),
},
index=X_test.index,
)
review = pd.concat([outcomes, X_test], axis=1)
if TASK == "classification":
mistakes = review[
review["__actual__"] != review["__predicted__"]
]
print(f"Mistakes: {len(mistakes):,} of {len(review):,}")
if mistakes.empty:
print("No test mistakes. Audit for leakage before celebrating.")
else:
mistake_types = (
mistakes
.groupby(["__actual__", "__predicted__"], dropna=False)
.size()
.sort_values(ascending=False)
.rename("rows")
)
display(mistake_types.to_frame())
display(mistakes.head(10))
else:
review["__residual__"] = (
review["__actual__"] - review["__predicted__"]
)
review["__absolute_error__"] = review["__residual__"].abs()
display(
review
.sort_values("__absolute_error__", ascending=False)
.head(10)
)
plt.figure(figsize=(7, 4))
plt.scatter(
review["__predicted__"],
review["__residual__"],
alpha=0.55,
)
plt.axhline(0, color="black", linestyle="--")
plt.xlabel("Predicted value")
plt.ylabel("Residual = actual - predicted")
plt.title("KNN residual check")
plt.show()If you are predicting a category, a small count of mistakes and a breakdown of which answers got confused for which — look for the most common direction of error and read a few of those rows. If you are predicting a number, a list of the biggest misses plus a scatter plot of the leftover errors (the residuals). Do not expect a flawless picture; you are hunting for tell-tale shapes like a curve, a widening funnel, or a cluster of rows the model handles badly. Your own data will differ.
Looking is fine; tuning to the test is not
This review is for understanding your model, not for improving it against the test. If what you see here sparks an idea for a new feature or a different setting, that is great — but take it back to training-only cross-validation to try it out. Every time you tweak things to score better on this test set, it quietly stops being an honest test. For any claim that really matters, set aside or gather a fresh batch of data you have never looked at.
- Which kind of error matters most in your real use case?
- Do the difficult rows share a feature pattern, missingness pattern, group, or time period?
- Did KNN beat the baseline by enough to change a decision?
- What evidence would make you reject this model despite a good score?
What you can now say in your own words
- KNN makes a prediction by looking at the most similar past cases: they vote for a category, or get averaged for a number.
- Because it works by distance, the feature columns have to be put on a fair footing first, or the big-numbered ones take over.
- Keeping preparation and the model together on one assembly line stops the test rows from sneaking into the learning.
- You chose K with cross-validation on the training data, never by chasing the test score.
- A single number opens the investigation; reading the actual mistakes tells you whether the model is truly useful.
Going deeper (optional): a fair way to try distance-weighted voting later
By default every neighbour gets an equal say. You could instead let closer neighbours count for more. On a future dataset or a freshly set-aside batch of data, compare the two by adding "model__weights": ["uniform", "distance"] to the training-only search. Remind yourself that this is a genuinely new question to test — so decide it with cross-validation, and keep your final set-aside data closed until the very end.
YOUR PACE, YOUR PROJECT
Save the evidence, not just the score.
Jot down the K you landed on, how steady the practice scores were, how KNN compared with the baseline on the real test, the most telling mistake pattern you noticed, and one place where judging by distance might be shaky on your particular columns.