THE IDEA
The wisdom of a crowd
Imagine you have a hard question and you ask hundreds of people instead of just one. Each person is a little different and might get it slightly wrong, but their individual mistakes point in different directions. When you take the group's overall answer, those errors tend to cancel out. A random forest works the same way, using trees instead of people.
A single tree can be opinionated and jumpy. A forest asks many trees and goes with the crowd — usually a calmer, more reliable answer.
To make sure the trees actually disagree in useful ways, each one is shown a slightly different slice of your data and only allowed to look at some of the columns at a time. That built-in variety is the whole trick. For a yes/no question the trees take a vote; for predicting a number, their answers are averaged.
You do not have to memorize the knobs, but here is the gist: more trees usually make the crowd steadier, while settings for depth and leaf size control how bold or cautious each individual tree is allowed to be.
No need to rescale your numbers
Like a single tree, a forest just asks yes/no threshold questions ("is this value above 5?"), so the size of your units never matters. The shared preprocessor still quietly fills in missing values and turns text categories into numbers, but it leaves your numeric columns as they are.
A powerful model still has to beat your baseline
Your baseline is the simple guess from the foundation page (for the example students, always guessing "pass" scores about 64%). A fancier model is only worth it if it clearly beats that number and fits what your real project actually needs.
Set up your forest
This cell reads the TASK you chose on the foundation page and automatically picks the right kind of forest — one for yes/no questions, one for predicting a number. It also wraps the cleanup step and the forest together into a single pipeline.
A pipeline is like a set of connected pipes: your data flows through the cleaning step first and into the model second, always in that order, so the two stay in sync and nothing leaks between them.
What this does: builds a random forest that matches your task and bundles it with your data-cleaning step, then prints your task name and how many rows are for training versus the held-back test.
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.ensemble import (
RandomForestClassifier,
RandomForestRegressor,
)
from sklearn.inspection import permutation_importance
from sklearn.model_selection import RandomizedSearchCV
from sklearn.pipeline import Pipeline
required_foundation = [
"TASK", "RANDOM_STATE", "FEATURES", "X_train", "X_test",
"y_train", "y_test", "make_preprocessor", "make_cv",
"CV_FIT_PARAMS", "PRIMARY_SCORING", "evaluate_predictions",
"results_log",
]
missing_foundation = [
name for name in required_foundation if name not in globals()
]
if missing_foundation:
raise NameError(
"Run every Foundation cell first. Missing: "
+ ", ".join(missing_foundation)
)
if TASK == "classification":
forest_estimator = RandomForestClassifier(
random_state=RANDOM_STATE,
n_jobs=1,
)
elif TASK == "regression":
forest_estimator = RandomForestRegressor(
random_state=RANDOM_STATE,
n_jobs=1,
)
else:
raise ValueError("TASK must be 'classification' or 'regression'.")
forest_pipeline = Pipeline([
("prepare", make_preprocessor(scale_numeric=False)),
("model", forest_estimator),
])
print("Task:", TASK)
print("Training rows:", len(X_train))
print("Held-out test rows:", len(X_test))Three short lines: your task, your number of training rows, and your number of test rows. This is just a quick safety check that the earlier setup cells ran. If it complains that something is missing, click Runtime -> Run all and try again — nothing is broken, the notebook just needs its earlier cells run first.
Let the forest tune itself — using training data only
A forest has a few settings you can adjust (how many trees, how deep, and so on). Instead of guessing, this cell tries a handful of sensible combinations and keeps the one that performs best. These adjustable settings are called hyperparameters.
Think of it like tuning a radio: the cell nudges the dials, listens to how well each setting does, and settles on the clearest one — all without ever peeking at your final test rows.
It judges each combination using cross-validation, the same fair method from your foundation page, which respects whether your data is random, time-based, or grouped. Your test rows stay sealed.
What this does: quietly tries several forest settings, keeps the best one, and shows you a table of what it tried ranked from best to worst, plus the winning settings.
forest_search = RandomizedSearchCV(
estimator=forest_pipeline,
param_distributions={
"model__n_estimators": [150, 300],
"model__max_depth": [None, 8, 16],
"model__min_samples_leaf": [1, 4, 0.02],
"model__max_features": ["sqrt", 0.7],
},
n_iter=12,
scoring=PRIMARY_SCORING,
cv=make_cv(),
random_state=RANDOM_STATE,
n_jobs=-1,
refit=True,
return_train_score=True,
)
forest_search.fit(X_train, y_train, **CV_FIT_PARAMS)
best_forest = forest_search.best_estimator_
forest_cv_table = pd.DataFrame(forest_search.cv_results_)[[
"param_model__n_estimators",
"param_model__max_depth",
"param_model__min_samples_leaf",
"param_model__max_features",
"mean_train_score",
"mean_test_score",
"std_test_score",
"rank_test_score",
]].sort_values("rank_test_score")
display(forest_cv_table)
print("Chosen settings:", forest_search.best_params_)
print("Validation metric:", PRIMARY_SCORING)A table with one row per setting it tried, sorted best-first, and a line showing the winning settings. On the example students this lands in roughly the mid-70s; your own data will land somewhere else, and that is completely fine.
Going deeper (optional): when the "winner" is really a tie
Notice the last column of each row is a spread — a "give or take" range. If several top rows overlap within that give-or-take, they are basically tied, and the exact number-one row is not truly better than its close neighbours. In that case the difference is likely just luck of the split, so don't read too much into which row landed first.
- Did the winning forest prefer deeper trees, or gentler ones with bigger leaves?
- For the boldest settings, how far apart are the training score and the validation score? A big gap is a hint the model is memorizing.
Give the finished forest its one real test
The tuning is done and the best forest has been re-trained on all your training rows. Now — and only now — we open the sealed test rows to see how it does on data it has genuinely never seen. This is the honest exam, and you take it once.
What this does: asks the finished forest to predict on the held-back test rows, scores those predictions, adds the result to your shared comparison log, and prints how many trees the forest ended up with.
forest_predictions = best_forest.predict(X_test)
forest_result = evaluate_predictions(
"Random forest",
y_test,
forest_predictions,
)
fitted_forest = best_forest.named_steps["model"]
print("Trees in fitted forest:", fitted_forest.n_estimators)Your forest's score on the test rows, plus a line counting its trees. On the example students the forest usually lands around 76% — comfortably above the ~64% baseline, and typically a touch steadier than the single tree from the last lesson. Your own data will give a different number, and there is no "correct" score here; the point is whether it beats your baseline. Also glance at where it makes mistakes, not just the headline number: a model can win on the overall score yet be worse at the one mistake that matters most to your real project.
This score is now locked in
Once you have looked at the test result, resist the urge to go back and tweak settings or drop a column to chase a higher number. Doing that quietly turns the test into practice data, and the score stops being honest. If you do decide to change things, you would need a brand-new, untouched set of test rows to judge the result fairly.
Which clues did the forest lean on most?
A nice thing about a forest is that you can ask which of your columns it relied on. The trick, called feature importance by shuffling, is beautifully simple: scramble one column so it becomes useless noise, then see how much the score drops. If the score falls a lot, the forest was leaning on that clue. If nothing much happens, that clue barely mattered.
Imagine covering up one column on a form and seeing how much worse your guesses get. Cover the truly useful column and you struggle; cover a useless one and you barely notice.
Because this runs on the whole pipeline and your original table, you get exactly one clear result per column you started with — even columns that were text categories stay grouped as one clue, instead of being split into confusing pieces.
What this does: scrambles each of your columns in turn to measure how much the forest relied on it, then shows a ranked table and a bar chart of the most-relied-on clues.
permutation_result = permutation_importance(
best_forest,
X_test,
y_test,
scoring=PRIMARY_SCORING,
n_repeats=8,
random_state=RANDOM_STATE,
n_jobs=-1,
)
importance_table = pd.DataFrame({
"original_feature": FEATURES,
"mean_score_drop": permutation_result.importances_mean,
"variation": permutation_result.importances_std,
}).sort_values("mean_score_drop", ascending=False)
display(importance_table)
plot_table = importance_table.head(20).sort_values("mean_score_drop")
plt.figure(figsize=(9, max(4, 0.35 * len(plot_table))))
plt.barh(
plot_table["original_feature"],
plot_table["mean_score_drop"],
xerr=plot_table["variation"],
)
plt.xlabel(f"Drop in {PRIMARY_SCORING} after shuffling")
plt.title("Permutation importance on original input columns")
plt.tight_layout()
plt.show()A ranked list and a bar chart of your columns. Longer bars are the clues the forest leaned on most for these test rows. Bars near zero (or even slightly below) mean that clue barely helped — that can happen with weak columns, plain noise, or two columns that carry the same information and share the credit. On the example students, columns like previous score and attendance tend to rise to the top, but your own data will tell its own story.
"Leaned on" is not the same as "caused"
This chart tells you which clues this model found useful — nothing more. It does not prove that changing that thing in real life would change the outcome. Treat a tall bar as a question to investigate, not a conclusion.
Going deeper (optional): two ways a bar can mislead you
Two related columns can trade importance between them: if both carry the same information, the forest may lean on one and leave the other looking unimportant, even though either would do. And a column that accidentally gives away the answer (called leakage) can look important for the wrong reason — it's not helping you predict, it's quietly handing over the answer. Both are reasons to read a tall bar as a starting question, not a finding.
- Is the top clue actually available at the moment you would make a real prediction, or does it sneak in information from later?
- Are any of your columns closely related, so they might be splitting the credit between them?
- Who with real-world knowledge of your subject should sanity-check this chart before anyone acts on it?
See how confident the forest is (yes/no tasks only)
For a yes/no task, the forest can tell you not just its answer but how sure it is — for example, "about 80% of my trees voted yes." That confidence comes from counting the votes across all the trees. It is a helpful hint, but treat it as the model's opinion, not a guaranteed real-world chance. If your task is predicting a number, there is no such thing as a vote share, so this cell simply says so and moves on — nothing to do.
What this does: for a yes/no task, shows a small table of the first few test rows with the actual answer, the predicted answer, and the forest's confidence in each option. For a number-prediction task, it just prints a short note that this step does not apply.
if TASK == "classification":
forest_probabilities = best_forest.predict_proba(X_test)
forest_classes = best_forest.named_steps["model"].classes_
forest_probability_table = pd.DataFrame(
forest_probabilities,
columns=[f"estimated_p({label})" for label in forest_classes],
index=X_test.index,
)
forest_probability_table.insert(0, "predicted", forest_predictions)
forest_probability_table.insert(0, "actual", y_test)
display(forest_probability_table.head(12))
else:
print(
"Regression predicts numbers, so this model has no "
"class-probability output."
)For a yes/no task, a short table of example rows with the true answer, the forest's guess, and its confidence. The interesting rows are the surprises: a very confident wrong guess, or a correct guess the forest was unsure about. For a number-prediction task, you will just see a one-line note that this step does not apply — that is expected, not an error.
Write down what you learned
A forest is more accurate than a single tree, but harder to explain one row at a time — you cannot point to a single simple rule. This last cell gathers everything worth remembering into one tidy record you can save: your winning settings, the columns the forest leaned on, and its final score. When you decide which model to keep, weigh that score against how easy it is to explain and how bad different mistakes would be.
What this does: shows your running comparison of every model so far, then prints a compact summary card for this forest — its settings, its most-relied-on columns, and its final test result.
display(pd.DataFrame(results_log))
project_record = {
"model": "Random forest",
"task": TASK,
"primary_cv_metric": PRIMARY_SCORING,
"chosen_settings": forest_search.best_params_,
"most_relied_on_columns": (
importance_table.head(5)["original_feature"].tolist()
),
"test_result": forest_result,
}
display(pd.Series(project_record, name="value"))A comparison table with a row for every model you have tried, followed by a small summary card for this forest. There is no single "right" winner here — the best choice depends on what the mistakes cost in your real project, not just which row has the highest number.
- Did the forest improve on your simpler models by enough to be worth being harder to explain?
- Which group of rows, time period, or type of mistake still deserves a closer look?
- What might make the score slip as new data arrives down the road?
DEEPER PRACTICE
Want to keep exploring? Here's how, safely
None of this is required — it is just for when you are curious to poke further. The one golden rule stays the same: keep your test rows sealed while you experiment. This table pairs a thing to try with what it teaches and the data rule to keep it honest.
| Try next | Learn | Data rule |
|---|---|---|
| Repeat CV with another seed | Selection stability | Do not inspect test again |
| Compare larger leaf sizes | Bias–variance trade-off | Choose from training folds |
| Remove a suspected leak | How honest features change performance | Create a fresh final test after development |
YOUR PACE, YOUR PROJECT
Save the notes your future self will thank you for.
Jot down how you split the data, the settings you kept, the give-or-take spread you saw, the final score, which mistakes hurt most, which columns the forest leaned on, and the reminder that "leaned on" is not the same as "caused." When it feels complete, mark this lesson done and move on whenever you are ready.