A team that corrects itself.

Many small corrections, added together

Think back to the last lesson. A random forest grows lots of trees at the same time, each looking at the problem a little differently, and then it averages their votes — like asking a whole room for their opinion at once.

Gradient Boosting is different. It works one teammate at a time, in a line. Tree 1 makes a first rough guess. Tree 2 looks only at where tree 1 went wrong and learns a small correction. Tree 3 fixes whatever is still off. Each new tree studies the leftover mistakes and patches them, and their small fixes add up into a sharp final answer.

This model has two main dials you can turn. Here is what each one does in plain language.

These two dials pull against each other: gentle steps (a small learning rate) usually want more rounds, while bigger steps want fewer. In Step 2 you will let cross-validation try several combinations and pick the pair that works for your data.

Gradient Boosting works for both jobs — sorting things into groups (classification) and predicting a number (regression). It does not care whether your numbers are on the same scale, so we skip that step, but your shared preprocessor still quietly fills in missing values and turns text categories into numbers behind the scenes.

Set up the model for your task

This cell first does a quick safety check that your setup cells ran, then builds a pipeline: the cleaning steps and the model wrapped together so they always run in the right order. It automatically picks the right version of Gradient Boosting for your task — the sorting-into-groups version or the predicting-a-number version.

What this does: Confirms your setup cells ran, then builds the Gradient Boosting model paired with your data-cleaning steps.

# Check that the shared foundation is available.
required_names = [
    "TASK", "X_train", "X_test", "y_train", "y_test",
    "make_preprocessor", "make_cv", "PRIMARY_SCORING",
    "CV_FIT_PARAMS", "evaluate_predictions", "results_log",
]
missing_names = [name for name in required_names if name not in globals()]
if missing_names:
    raise RuntimeError(
        "Run the Start Here foundation cells first. Missing: "
        + ", ".join(missing_names)
    )

from sklearn.ensemble import (
    GradientBoostingClassifier,
    GradientBoostingRegressor,
)
from sklearn.pipeline import Pipeline

if TASK == "classification":
    gradient_estimator = GradientBoostingClassifier(
        random_state=RANDOM_STATE
    )
else:
    gradient_estimator = GradientBoostingRegressor(
        random_state=RANDOM_STATE
    )

gradient_pipeline = Pipeline([
    ("prepare", make_preprocessor(scale_numeric=False)),
    ("model", gradient_estimator),
])

print("Pipeline ready for:", TASK)

Let the model try dial settings on practice data

Now we let the computer try several combinations of the two dials and see which works best — but only using cross-validation on the practice data. Your real test rows stay sealed in an envelope, untouched, so the final grade later is honest.

Each combination of "number of rounds" and "size of each correction" is one hyperparameter choice. Start with the modest set of options below; if your dataset is large this can take a few minutes, which is completely normal — let it run.

What this does: Quietly tries several dial settings on practice data and keeps the combination that scored best.

from sklearn.model_selection import GridSearchCV

parameter_grid = {
    "model__n_estimators": [50, 100, 200],
    "model__learning_rate": [0.03, 0.1, 0.3],
    "model__max_depth": [2, 3],
}

gradient_search = GridSearchCV(
    estimator=gradient_pipeline,
    param_grid=parameter_grid,
    scoring=PRIMARY_SCORING,
    cv=make_cv(),
    n_jobs=-1,
    refit=True,
)
gradient_search.fit(X_train, y_train, **CV_FIT_PARAMS)

print("Best training-CV settings:")
for name, value in gradient_search.best_params_.items():
    print(f"  {name}: {value}")

if TASK == "classification":
    print(f"Best mean CV balanced accuracy: {gradient_search.best_score_:.3f}")
else:
    print(f"Best mean CV MAE: {-gradient_search.best_score_:.3f}")

Open the sealed test — just once

All the choosing is done. Now the winning model sits the exact same held-out exam that your baseline and your earlier models took. We open that envelope only once so the score stays trustworthy.

What this does: Asks the winning model to predict the sealed test rows and shows how well those predictions match the real answers.

best_gradient = gradient_search.best_estimator_
gradient_predictions = best_gradient.predict(X_test)

evaluate_predictions(
    "Gradient Boosting",
    y_test,
    gradient_predictions,
    show_details=True,
)

See how each dial setting did

Before you opened the test, the search quietly quizzed every dial combination on practice data. This cell lays those results out in a table, best at the top, so you can see the pattern instead of guessing.

What this does: Shows a tidy table of every dial combination the search tried and how each one scored on the practice quizzes.

cv_results = pd.DataFrame(gradient_search.cv_results_)

columns_to_show = [
    "param_model__n_estimators",
    "param_model__learning_rate",
    "param_model__max_depth",
    "mean_test_score",
    "std_test_score",
    "mean_fit_time",
]

cv_summary = cv_results[columns_to_show].copy()
if TASK == "regression":
    cv_summary["mean_test_score"] *= -1
    cv_summary = cv_summary.rename(
        columns={"mean_test_score": "mean_validation_MAE"}
    )
    cv_summary = cv_summary.sort_values("mean_validation_MAE")
else:
    cv_summary = cv_summary.rename(
        columns={"mean_test_score": "mean_validation_balanced_accuracy"}
    )
    cv_summary = cv_summary.sort_values(
        "mean_validation_balanced_accuracy", ascending=False
    )

display(cv_summary.head(10))

See which columns the model leans on

Which of your columns actually helped? This cell finds out with a clever trick called permutation importance: it scrambles one column at a time and watches how much the score falls. If scrambling a column barely changes the score, the model was not really using it. If the score collapses, that column was doing a lot of work.

What this does: Scrambles each column in turn, measures how much the score drops, and draws a bar chart ranking your columns from most to least useful.

from sklearn.inspection import permutation_importance

gradient_importance = permutation_importance(
    best_gradient,
    X_test,
    y_test,
    scoring=PRIMARY_SCORING,
    n_repeats=5,
    random_state=RANDOM_STATE,
    n_jobs=-1,
)

importance_table = pd.DataFrame({
    "feature": X_test.columns,
    "importance": gradient_importance.importances_mean,
    "variation": gradient_importance.importances_std,
}).sort_values("importance", ascending=False)

display(importance_table)

importance_table.sort_values("importance").plot.barh(
    x="feature",
    y="importance",
    xerr="variation",
    legend=False,
    figsize=(8, max(4, len(importance_table) * 0.35)),
)
plt.xlabel("Score lost after shuffling this feature")
plt.title("Gradient Boosting permutation importance")
plt.tight_layout()
plt.show()

Your own scoreboard for the whole course

This is the moment the course has been building toward. Your results_log has been quietly collecting a row for every model you tried in this notebook — the baseline and each lesson — all graded on the very same sealed test. This cell lays them out as one scoreboard so you can finally compare them fairly.

What this does: Gathers every model you have run into one table and sorts it, best first, by the measure that fits your task.

leaderboard = pd.DataFrame(results_log)

if TASK == "classification":
    leaderboard = leaderboard.sort_values(
        "balanced_accuracy", ascending=False
    )
else:
    leaderboard = leaderboard.sort_values("MAE")

display(leaderboard.reset_index(drop=True))

Before you crown a winner, it helps to ask a few honest questions. The highest score is not always the best choice — here is why the answer can change once you look past the number.

A peek at the model's "how sure" numbers

This one is optional — skip it with a clear conscience if you are ready to wrap up. If your project is a classification (sorting-into-groups) task, the model can also tell you how strongly it leans toward each group for a given row. These are handy for ranking rows by how sure the model seems, but they are estimates, not guarantees, so never read them as certainty.

What this does: For a classification project, shows the first ten test rows next to the true answer, the model's guess, and how strongly it leaned toward that guess.

if TASK != "classification":
    print("Probability output applies to classification only.")
else:
    class_names = best_gradient.named_steps["model"].classes_
    probabilities = best_gradient.predict_proba(X_test)

    probability_preview = X_test.head(10).copy()
    probability_preview["truth"] = y_test.head(10).values
    probability_preview["prediction"] = gradient_predictions[:10]
    probability_preview["largest_estimated_probability"] = (
        probabilities[:10].max(axis=1)
    )

    print("Probability column order:", list(class_names))
    display(probability_preview)

The real lesson was the routine, not any one model

Here is the honest secret of the whole course: the individual models matter less than the steady routine you now know by heart. If you can follow these seven steps, you can pick up almost any new model and use it safely.

1. Decide what one row means, what you are predicting, which columns you get to use, and which mistake would hurt most.
2. Split the data in a way that mirrors how it will really be used — by chance, by time, or by group.
3. Learn all cleaning steps from the practice data only, kept inside a pipeline so nothing leaks.
4. Build a plain baseline first, so every fancy model has something honest to beat.
5. Tune the dials with cross-validation on practice data — never on the final test.
6. Open the test just once, and look at where it goes wrong, not only the headline score.
7. Pick the model whose trade-offs actually fit the decision you have to make.