It's really just a scorecard
Picture a scorecard. Each clue about a student — hours studied, attendance, and so on — gets a weight, a number that says how much that clue counts. You multiply each clue by its weight, add everything up, and out comes your predicted number. That's the whole idea.
Every clue gets a "how much does this matter?" score. The model adds up all those scores to make its guess. Clues that matter a lot get big weights; clues that barely matter get tiny ones.
Written out, the model's rule looks like this:
The complete rule
prediction = intercept + weight₁ × feature₁ + weight₂ × feature₂ + … + weightₚ × featureₚ
The intercept is just the starting point — the guess the model would make before looking at any clue. From there, a positive weight nudges the prediction up as that clue gets bigger, and a negative weight nudges it down. When there are several clues at once the picture is a little harder to draw, but the recipe never changes: multiply each clue by its weight, then add.
What's lovely about it
It's fast, and you can actually read the finished scorecard to see which clues mattered. It's often a surprisingly strong first guess for any number you're trying to predict.
Where it struggles
It draws straight lines only. It can't spot curves, tipping points, or clues that only matter in combination — unless you hand it clues that already capture those things.
Going deeper (optional): what "best fit" means
Plain Linear Regression picks the weights that make the training-set misses as small as possible — specifically, it squares each miss and adds them up, so a few big misses hurt far more than many tiny ones. We still report everything using MAE (average miss in your target's own units) because that's much easier to read, and it's the score your setup notebook already uses for regression.
A quick safety check
Before we do anything else, this first cell just makes sure your setup cells ran and that you really are predicting a number. It's a friendly gatekeeper, nothing more. One gentle reminder: don't turn a real category (like "cat / dog / bird") into made-up numbers just to squeeze into this lesson — if your answer is a category, the classification lessons are the right home for it.
What this does: Peeks to confirm the setup cells ran and that your task is set to predicting a number, then prints a tiny summary so you know all is well.
required_names = [
"TASK", "RANDOM_STATE",
"X_train", "X_test", "y_train", "y_test",
"make_preprocessor", "make_cv", "CV_FIT_PARAMS",
"PRIMARY_SCORING", "evaluate_predictions", "results_log",
]
missing_names = [name for name in required_names if name not in globals()]
if missing_names:
raise RuntimeError(
"Run the foundation cells first. Missing: "
+ ", ".join(missing_names)
)
if TASK != "regression":
raise ValueError(
"Linear Regression needs a numeric regression target. "
"Keep your task honest and choose a classification lesson."
)
if pd.Series(y_train).nunique(dropna=True) < 2:
raise ValueError("The training target needs at least two distinct values.")
print("Task:", TASK)
print("Training shape:", X_train.shape)
print("Testing shape: ", X_test.shape)
print("Primary tuning score:", PRIMARY_SCORING)A few short lines: the word regression, the size of your training and testing data, and a scoring name like neg_mean_absolute_error. If instead it complains that something is missing, don't worry — just click Runtime → Run all to run the setup cells again, then try this one. There's nothing to fix in this lesson.
Watch out for clues that secretly hold the answer
Here's the single most important idea in this whole lesson. Sometimes a clue quietly contains the answer itself. When that happens the model looks brilliant — near-perfect scores — but it has actually learned nothing useful, because in real life that clue wouldn't be available yet. This trap has a name: leakage.
Imagine trying to predict a student's final score, but one of your clues is "did they pass?" — which is decided by the final score. Of course the guess looks perfect; you accidentally handed the model the answer. A perfect score is a reason to be suspicious, not proud.
Linear models make this trap easy to spot, because a leaky clue tends to grab a huge weight — but by then you may have wasted time trusting a fake result. So take a slow, honest look at each clue before you fit. Use these questions:
This table pairs a question to ask about each clue with the kind of thing that should make you pause.
| Ask yourself | A sign to slow down |
|---|---|
| Would you actually know this clue at the moment you make the guess? | A status, measurement, or total that only gets recorded after the outcome happens |
| Was this clue built from the answer itself? | A ratio, band, flag, or copy that was calculated using the very thing you're predicting |
| Does it summarise other rows in a fair way? | A total or average that sneakily includes the current row or rows from the future |
| Could the same person or thing show up on both sides of the split? | The same people, devices, cases, or dates appearing in both training and testing |
| Is the data prep learned from training data only? | Filling gaps, scaling, or picking clues done before the data was split apart |
If it looks too good to be true, pause and check
A near-perfect score might be genuine — but treat it as a signal to look harder first. Hunt for clues built from the answer, fields recorded after the outcome, duplicate rows, and an unrealistic split. The pipeline you'll build below guards the data-prep steps for you, but only you, thinking about your own problem, can catch a clue that wouldn't exist yet in real life.
- For each clue, when in real life does its value actually become known?
- Could any clue have been built using the answer, using future rows, or using this row's own outcome?
- Is there a suspicious clue you'll drop from
FEATURESbefore going on?
Plain, Ridge, or Lasso?
All three build the exact same scorecard. The difference is how carefully they keep the weights in check. Here's the problem they solve: when several clues overlap and say almost the same thing, a plain scorecard can swing wildly — giving one clue a giant positive weight and its twin a giant negative one, so the two cancel out. That makes the model jumpy and easy to fool.
Ridge gently reminds every weight to stay humble, so no single clue runs away with the answer. Lasso is stricter: it can push a useless clue's weight all the way to zero, quietly crossing that clue off the scorecard for you. Both are ways of saying "don't get carried away" — the fancy word for that is regularisation.
Here's how the three compare at a glance.
| Which one | What it tries to do | Why you'd want it |
|---|---|---|
| Plain (LinearRegression) | Just make the training misses as small as it can | The simplest starting point, with no restraint on the weights |
| Ridge | Small misses and humble weights | Calms down clues that overlap or behave unpredictably |
| Lasso | Small misses and a firm nudge toward zero | Can drop useless clues entirely — automatic clue-picking |
There's one dial that sets how firm Ridge and Lasso are, called alpha. (A dial you choose yourself, rather than one the model figures out, is called a hyperparameter.) Turn alpha to zero and there's no restraint at all; turn it up very high and the model gets so cautious it barely reacts to any clue. The best cousin and the best alpha depend entirely on your data — so you won't guess them by hand. In the next cell, the notebook tries the options and picks for you.
Let the notebook pick the best cousin for you
This cell does the choosing so you don't have to. It quietly tries plain, several Ridge settings, and several Lasso settings, and keeps whichever does best. It makes that choice using cross-validation: it practises on part of your training data and checks on the part it held back, over and over, so the winner is chosen fairly and your real test set stays sealed.
It's like a taste test. The notebook tries each recipe on some of the training data, checks it on the rest, repeats a few times, and crowns the recipe that tastes best on average. The final test set is never touched during this — it's saved for one honest check later.
One quiet but important detail: before comparing weights, every clue is first put on a common scale (this is called standardising). Without that, Ridge and Lasso would unfairly punish a clue just for happening to be measured in bigger numbers.
What this does: Tries plain, Ridge, and Lasso at several strengths, uses the practise-and-check method to find the best combination, and prints which one it chose.
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import GridSearchCV
linear_pipeline = Pipeline([
("prepare", make_preprocessor(scale_numeric=True)),
("model", LinearRegression()),
])
# Ridge is largely scale-free in y; Lasso's useful alpha range
# changes with the target's units, so adapt it to this dataset.
ridge_alphas = np.logspace(-3, 3, 7)
target_scale = max(float(np.std(y_train)), np.finfo(float).eps)
lasso_alphas = target_scale * np.logspace(-4, 0, 5)
search_space = [
{
"model": [LinearRegression()],
},
{
"model": [Ridge()],
"model__alpha": ridge_alphas,
},
{
"model": [Lasso(max_iter=20000)],
"model__alpha": lasso_alphas,
},
]
linear_search = GridSearchCV(
estimator=linear_pipeline,
param_grid=search_space,
scoring=PRIMARY_SCORING,
cv=make_cv(),
n_jobs=-1,
return_train_score=True,
error_score="raise",
)
linear_search.fit(X_train, y_train, **CV_FIT_PARAMS)
raw_cv = pd.DataFrame(linear_search.cv_results_)
cv_summary = pd.DataFrame({
"family": [
params["model"].__class__.__name__
for params in raw_cv["params"]
],
"alpha": [
params.get("model__alpha", np.nan)
for params in raw_cv["params"]
],
"mean_validation_MAE": -raw_cv["mean_test_score"],
"fold_to_fold_variation": raw_cv["std_test_score"],
"rank": raw_cv["rank_test_score"],
})
display(
cv_summary
.sort_values(["rank", "family", "alpha"])
.reset_index(drop=True)
)
best_linear_pipeline = linear_search.best_estimator_
selected_model = best_linear_pipeline.named_steps["model"]
selected_family = selected_model.__class__.__name__
selected_alpha = getattr(selected_model, "alpha", "not used")
print("Selected family:", selected_family)
print("Selected alpha: ", selected_alpha)A little table listing plain Linear Regression once, plus several Ridge and Lasso options, each with its average miss (its MAE — the typical distance between guess and truth). Then a line naming the winner. Smaller MAE is better. If the top few are almost tied, or the numbers jump around a lot between rounds, that just means the ranking is a bit unsure — that's normal, and your own data will land on different numbers than any example.
Why this choice is trustworthy
Everything — the cousin, the alpha, the gap-filling, the scaling — was learned only from training data during the practise-and-check rounds. The CV_FIT_PARAMS from your setup also keeps any grouped rows from leaking across the split. Your test answers had no say in the choice, which is exactly what keeps the final result honest.
Going deeper (optional): a warning can be helpful, not scary
Lasso sometimes prints a note that it "did not converge" — usually when clues overlap heavily or the alpha it's testing is very weak. Don't hide that note. You can raise max_iter to give it more time, trim clues that repeat the same information, or narrow the range being searched — always deciding based on the training rounds, never by peeking at the test score.
Read the finished scorecard
Now for the fun part: you get to open up the winner and see the weights it chose. Each weight is a coefficient, which is just "how much this clue moves the prediction." A big number means that clue has a big say; a number near zero means it barely matters. The notebook already re-trained the winner on all your training data, so nothing here so much as glances at the test set.
What this does: Pulls out the starting point and each clue's weight, then shows the clues with the biggest say, largest first.
prepared_features = (
best_linear_pipeline
.named_steps["prepare"]
.get_feature_names_out()
)
coefficients = np.asarray(selected_model.coef_).reshape(-1)
intercept = float(
np.asarray(selected_model.intercept_).reshape(-1)[0]
)
coefficient_table = pd.DataFrame({
"prepared_feature": prepared_features,
"weight": coefficients,
})
coefficient_table["absolute_weight"] = (
coefficient_table["weight"].abs()
)
print(f"Intercept: {intercept:.6g}")
print("Selected family:", selected_family)
print(
"Weights near zero:",
int(np.isclose(coefficients, 0).sum()),
"of",
len(coefficients),
)
display(
coefficient_table
.sort_values("absolute_weight", ascending=False)
.head(20)
)A starting point (the intercept) and a list of up to twenty clues that had the biggest say, ranked by how much they move the guess. A small note: because the clues were put on a common scale — and because categories get split into yes/no columns — a weight of "1" doesn't mean the same real-world change for every clue. So read the list as "which clues matter most here," not as an exact price tag. Your own data will highlight different clues.
Weights show what this model leans on — not what causes what
Please resist reading these as causes. A big weight can come from a real link, from how a clue is scaled, from how a category was coded, from two clues overlapping — or from leakage. When two clues say almost the same thing they can trade weight back and forth while the predictions barely change. And a Lasso weight of zero only means "this model didn't use it here," not "this thing doesn't matter in real life."
- Does every big-weight clue make sense as something you'd actually know when you make the guess?
- Which weights look shaky because two clues carry overlapping information?
- Does the direction (up or down) match what you'd expect — and what innocent, non-causal reason could also explain it?
The one honest test
This is the moment we've been protecting. Everything is now locked in — the cousin, the alpha, the clues. We open the test set a single time, make predictions, and write the score down next to the baseline, the same humble yardstick every regression lesson uses. You test once and take the number as it comes; think of it as evidence, not a grade.
What this does: Makes the winner's predictions on the untouched test data and records the score in your results log, right beside the simple baseline.
linear_predictions = best_linear_pipeline.predict(X_test)
result_name = f"{selected_family} (CV-selected)"
linear_test_result = evaluate_predictions(
result_name,
y_test,
linear_predictions,
show_details=False,
)Your results log now shows two rows: the simple baseline and your chosen model. MAE and RMSE are the typical size of a miss, in your target's own units — smaller is better. R² is a 0-to-1 score of how much better than "just guess the average" you did. For a rough feel, on the example students the baseline lands around a 10-point average miss with R² near 0, and a good linear model reaches an R² of about 0.64 — but your own data will land somewhere else entirely, and there's no single "right" number. What matters is that you beat your baseline.
Look at the misses, not just the average
An average score hides a lot. This last cell draws pictures of where the model missed, so you can spot patterns. A residual is simply the real answer minus the guess. Positive means the model aimed too low; negative means it aimed too high. Scattered, patternless misses are healthy. A clear pattern is the model telling you something it couldn't capture.
What this does: Lists the biggest misses and draws three quick pictures so you can see whether the misses form a telling pattern.
outcomes = pd.DataFrame(
{
"__actual__": np.asarray(y_test),
"__predicted__": np.asarray(linear_predictions),
},
index=X_test.index,
)
residual_review = pd.concat([outcomes, X_test], axis=1)
residual_review["__residual__"] = (
residual_review["__actual__"]
- residual_review["__predicted__"]
)
residual_review["__absolute_error__"] = (
residual_review["__residual__"].abs()
)
display(
residual_review
.sort_values("__absolute_error__", ascending=False)
.head(10)
)
low = min(
residual_review["__actual__"].min(),
residual_review["__predicted__"].min(),
)
high = max(
residual_review["__actual__"].max(),
residual_review["__predicted__"].max(),
)
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
axes[0].scatter(
residual_review["__actual__"],
residual_review["__predicted__"],
alpha=0.5,
)
axes[0].plot([low, high], [low, high], "--", color="black")
axes[0].set_xlabel("Actual")
axes[0].set_ylabel("Predicted")
axes[0].set_title("Actual vs predicted")
axes[1].scatter(
residual_review["__predicted__"],
residual_review["__residual__"],
alpha=0.5,
)
axes[1].axhline(0, linestyle="--", color="black")
axes[1].set_xlabel("Predicted")
axes[1].set_ylabel("Residual")
axes[1].set_title("Residual pattern")
axes[2].hist(
residual_review["__residual__"],
bins=25,
edgecolor="white",
)
axes[2].axvline(0, linestyle="--", color="black")
axes[2].set_xlabel("Residual")
axes[2].set_title("Residual distribution")
plt.tight_layout()
plt.show()Three little charts and a list of the biggest misses. Don't worry about whether the dots look "good" — instead look for shapes: a gentle curve, a funnel that widens, clumps that sit apart, whole regions that lean too high or too low, or a handful of dramatic misses. A shape is a hint about something the straight-line scorecard couldn't capture. Your own charts will look different, and that's expected.
Here's a quick guide to the shapes you might notice and what each could be hinting at.
| If you see this shape | It might mean | A question to explore (training data only) |
|---|---|---|
| A curved band of misses | The straight-line scorecard is missing a bend in the real pattern | Would a sensible reshaping of a clue, or a model that allows curves, fit better? |
| A funnel (misses spread out) | The model is more precise for some sizes of answer than others | Would reshaping the target, or studying groups separately, help? |
| Separate clumps | One group of rows may follow a different pattern | Is there a meaningful group clue that's missing or poorly encoded? |
| A few enormous misses | Unusual rows, data-quality glitches, or rare situations | Are those rows real, important, and well represented in your training data? |
Don't tweak the model to please this chart
It's tempting, but resist. This chart is a diagnosis of your final result. If it inspires you to try another approach, that's fine — just judge the new attempt with the practise-and-check method on training data, and set aside a fresh, untouched slice of data before you make any new final claim. Once you've studied a test set, it's no longer an honest test.
- How big is the average miss (MAE) in the real units of your decision — is that close enough to be useful?
- Which miss-pattern stands out most, and what are two possible explanations for it?
- Does the model do noticeably better or worse for a certain group, time period, or range of answers?
- Would you actually use this model? Write down your decision and the evidence behind it.
What you can now explain to a friend
You covered a lot. Here's the short version, in plain words:
- A linear prediction is a scorecard: a starting point plus each clue times its weight.
- Plain regression lets the weights run free; Ridge keeps them humble; Lasso can cross useless clues off entirely.
- Putting clues on a common scale matters, especially once you're comparing weights across the cousins.
- You didn't hand-pick the cousin or its strength — the practise-and-check method did, using training data only.
- Weights show what the model leans on, not what causes what.
- Looking at the misses reveals problems that a single average score would hide.
- Leakage is a problem with how the question was set up, and no clever model can rescue it.
Going deeper (optional): building new clues without leaking
A linear model can happily use a sensible reshaping of a clue — a square, a logarithm, a rate, or two clues combined — as long as it's built only from information you'd genuinely have when making the guess. Add these inside the pipeline, jot down why each one makes sense for your problem, and compare them with the practise-and-check method. What you must never do is invent new clues after peeking at the final test misses and then still call that test "untouched."
YOUR PACE, YOUR PROJECT
Jot down a few notes before you move on.
Take a minute to write down: which cousin and strength won, how steady the practise-and-check scores were, how you compared against the baseline, two cautions about the weights, one miss-pattern you noticed, and what your leakage check concluded. Future-you will thank you.