THE IDEA
It's a flowchart the computer writes itself
Think of the game 20 Questions. You're trying to guess something, so you ask one narrow question at a time — "Is it bigger than a breadbox?" — and each answer sends you down a different path until you're confident. A decision tree works exactly like that. It asks a question ("Did the student attend more than 80% of classes?"), splits the rows into "yes" and "no", and then asks another question inside each group, and another, until it reaches a confident guess.
A decision tree is an automatic flowchart of yes-or-no questions. The clever part is that the computer figures out which questions to ask, and in what order, all by itself — you don't write a single rule.
The end of each path is called a leaf. When you're predicting a yes/no answer (like whether a student passed), the leaf votes for the most common outcome among the training rows that ended up there. When you're predicting a number (like a final score), the leaf guesses the average of those rows. Same flowchart idea, either way.
Trees don't care about the size of your numbers
Remember how the neighbours model needed every clue rescaled first? A tree doesn't. A question like "is attendance below 80%?" gives the same answer whether attendance is written as a percent, a fraction, or a count — the ordering never changes. So there's nothing to scale here, which is one less thing to worry about. (The shared setup still tidies up missing values and text categories for you — that part still matters.)
Easy to read isn't the same as trustworthy
A small tree is lovely because you can read it like a flowchart and follow its logic. But being readable doesn't prove the model is right, fair, or that it'll hold up on new data. A clear explanation of a shaky rule is still a shaky rule — so we'll keep checking it honestly.
Set up the tree
This first cell gets everything ready. It looks at whether your project is a yes/no question or a number-guessing one, and quietly picks the matching kind of tree — you don't have to choose. It also runs a quick safety check first. If it complains about missing setup, that's just a friendly reminder that the foundation cells haven't run yet: click Runtime → Run all and try again. Nothing is broken.
What this does: Checks that your setup cells ran, builds the right kind of tree for your task, and prints a tiny summary so you know it's ready.
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.tree import (
DecisionTreeClassifier,
DecisionTreeRegressor,
plot_tree,
)
required_foundation = [
"TASK", "RANDOM_STATE", "FEATURES", "X_train", "X_test",
"y_train", "y_test", "make_preprocessor", "make_cv",
"CV_FIT_PARAMS", "PRIMARY_SCORING", "evaluate_predictions",
"results_log",
]
missing_foundation = [
name for name in required_foundation if name not in globals()
]
if missing_foundation:
raise NameError(
"Run every Foundation cell first. Missing: "
+ ", ".join(missing_foundation)
)
if TASK == "classification":
tree_estimator = DecisionTreeClassifier(
random_state=RANDOM_STATE
)
elif TASK == "regression":
tree_estimator = DecisionTreeRegressor(
random_state=RANDOM_STATE
)
else:
raise ValueError("TASK must be 'classification' or 'regression'.")
tree_pipeline = Pipeline([
("prepare", make_preprocessor(scale_numeric=False)),
("model", tree_estimator),
])
print("Task:", TASK)
print("Training rows:", len(X_train))
print("Held-out test rows:", len(X_test))A few short lines: your task (something like "classification" or "regression"), how many training rows you have, and how many rows are set aside for the final test. On the example study-habits data that's a few hundred rows. Your own data will show different numbers — just glance at the task to be sure it matches what you're trying to predict.
Let the tree find its own best size
Here's the one thing to watch with trees. A tree left to grow freely will keep asking questions until it has a tiny, hyper-specific path for almost every training row. That feels impressive, but it's really just memorising — like a student who memorises the answer key instead of learning the subject, and then freezes on a slightly different exam.
The problem has a name: overfitting. The fix is simple: don't let the tree grow too deep. A shorter tree is forced to learn real patterns instead of memorising individual rows.
So how deep is the right depth? Rather than guess, the notebook tries several sizes and lets your data decide. It uses cross-validation — practising on part of your training data and checking on the part it held back, over and over — so the winning size is chosen fairly and your real test set stays sealed.
What this does: Tries trees of several depths and sizes, uses the practise-and-check method to find the one that works best on your data, and prints the winner.
tree_search = GridSearchCV(
estimator=tree_pipeline,
param_grid={
"model__max_depth": [2, 3, 5, 8, None],
"model__min_samples_leaf": [1, 5, 0.02],
},
scoring=PRIMARY_SCORING,
cv=make_cv(),
n_jobs=-1,
refit=True,
return_train_score=True,
)
tree_search.fit(X_train, y_train, **CV_FIT_PARAMS)
best_tree = tree_search.best_estimator_
tree_cv_table = pd.DataFrame(tree_search.cv_results_)[[
"param_model__max_depth",
"param_model__min_samples_leaf",
"mean_train_score",
"mean_test_score",
"std_test_score",
"rank_test_score",
]].sort_values("rank_test_score")
display(tree_cv_table.head(15))
print("Chosen settings:", tree_search.best_params_)
print("Validation metric:", PRIMARY_SCORING)A short table of the tree sizes it tried, ranked best to worst, and a line naming the settings it picked. On the example study-habits data it usually lands on a modestly-sized tree, not the deepest one — which is exactly the point. Your own data will pick different settings, and that's fine. If the top few rows are nearly tied, the ranking is just a little unsure; that's normal.
Catch memorising in the act
This cell shows you overfitting with your own eyes. For each tree size, it compares two scores: how well the tree did on the rows it practised on, and how well it did on the rows it was checked on. A tree that has memorised will ace its practice rows but slip on the check rows — and the gap between those two numbers is the tell-tale sign.
Big gap between practice and check scores = the tree is memorising rather than learning. A small gap is reassuring. This is why we let the notebook pick a sensible size instead of growing the tree as deep as it can go.
What this does: Lines up each tree size with the gap between its practice score and its check score, so the memorisers stand out at the top.
overfit_view = tree_cv_table.copy()
overfit_view["train_minus_validation"] = (
overfit_view["mean_train_score"]
- overfit_view["mean_test_score"]
)
display(
overfit_view.sort_values(
"train_minus_validation", ascending=False
).head(12)
)
best_cv_row = overfit_view.loc[
overfit_view["rank_test_score"] == 1
].iloc[0]
print(
"Chosen model train–validation gap:",
best_cv_row["train_minus_validation"],
)A table with the widest practice-versus-check gaps at the top — usually the deepest, most unrestricted trees. Then a single line showing the gap for the size the notebook actually chose, which should be much smaller. A big gap warns you a tree is unstable; a small gap is comforting, though on its own it doesn't prove the model is useful — for that we still compare to your baseline. Your numbers will differ from any example.
Going deeper (optional): why the number-guessing scores look negative
If your project predicts a number, the score in these tables is written as a negative value (it's the average miss, flipped to a minus). Don't let that throw you — the rule is still "bigger is better", so a score closer to zero is a smaller miss. The practice-minus-check gap reads exactly the same way: a wide gap still means memorising.
- Which tree sizes showed the widest practice-versus-check gaps?
- Would you accept a slightly weaker but much smaller tree? What might that buy you?
Give the tree its one and only exam
The practise-and-check step already picked the best tree size. Now — for the first and only time — we hand the tree the test rows it has never seen and see how it does. Think of it as the final exam after all the studying. You get one honest look, and that's the number you trust.
What this does: Uses the winning tree to make predictions on the untouched test rows, scores them, and reports how deep the tree ended up and how many leaves it grew.
tree_predictions = best_tree.predict(X_test)
tree_result = evaluate_predictions(
"Decision tree",
y_test,
tree_predictions,
)
fitted_tree = best_tree.named_steps["model"]
print("Fitted depth:", fitted_tree.get_depth())
print("Number of leaves:", fitted_tree.get_n_leaves())Your tree's score on the test rows, plus its final depth and number of leaves. On the example study-habits data a tree lands around 71% correct — comfortably above the roughly 64% baseline you built earlier, though usually a touch below the other models in this course. Your own number will be different; there's no single right score. The honest question is always the same: did it beat your baseline?
Don't peek and re-tune
If this score disappoints you, resist the urge to go back and try a different depth to squeeze out a nicer number. The moment you tune based on the test result, that result stops being honest — it starts flattering the model. Your test score is evidence, not a grade to chase.
Draw the flowchart and read it
This is the fun payoff. Because a tree is just a flowchart, we can actually draw it. This cell shows the top few levels — the very first questions the model asks — so you can follow its reasoning like a game of 20 Questions. (The real tree may go deeper; we only draw the top so it stays readable.)
What this does: Draws a picture of the first few questions your tree asks, with the real column names on each branch.
transformed_feature_names = (
best_tree.named_steps["prepare"].get_feature_names_out().tolist()
)
fitted_tree = best_tree.named_steps["model"]
plot_options = {}
if TASK == "classification":
plot_options["class_names"] = [
str(label) for label in fitted_tree.classes_
]
plt.figure(figsize=(18, 8))
plot_tree(
fitted_tree,
feature_names=transformed_feature_names,
max_depth=2,
filled=True,
rounded=True,
impurity=False,
fontsize=8,
**plot_options,
)
plt.title("First three levels of the fitted decision tree")
plt.show()A boxes-and-branches diagram. The box at the top is the very first question the tree found most useful; follow "yes" one way and "no" the other, and each box below asks a narrower question. Try tracing one row through it. A question built from a text category usually reads as "is this category present or not?", and a number cutoff won't always land on a tidy real-world value. Your picture will look different from any example — that's your model's own logic.
- Is the very first question reasonable for your subject — or does it hint that a clue is secretly giving away the answer?
- What important behaviour might be hiding below the levels shown here?
- Could someone unfamiliar with your data follow these box labels without your help?
DEEPER PRACTICE
Try a few gentle experiments
Only if you're curious — none of this is required. Each row is a small tweak you can try, the question it helps you answer, and the one rule to keep it honest. Take them at your own pace.
| Try this | What it tells you | Keep it honest |
|---|---|---|
| Use fewer clues | Can a simpler tree do about as well? | Judge by the practise-and-check score, never the test |
| Require bigger leaves | Do the rules get steadier and easier to read? | Compare on the practice folds |
| Re-run with a different seed | Do the chosen questions stay the same? | Leave the test set sealed |
- What did the tree learn that matched your expectations?
- What rule surprised you — and how will you look into it?
- Where in your evidence is overfitting easiest to spot?
YOUR PACE, YOUR PROJECT
Jot down what your tree learned.
Before you move on, note the size the notebook chose, the final test score next to your baseline, and one question from the flowchart you'd like to double-check. That's your evidence — no grade attached.