Project: An End-to-End Tabular Prediction System

Definition

Can you beat the baseline honestly, and explain how?

This project delivers a complete tabular prediction system: a framed problem with a cost-aware metric, a leak-free preprocessing pipeline, a baseline, several model families compared with appropriate cross-validation, a tuned final model, a threshold chosen from costs, one honest test evaluation, interpretability analysis and a written model card.

In simple words

Everything in this module, in one notebook. The example below predicts malignant tumours, where missing a malignant case is far worse than a false alarm. Replace the dataset with your own, such as loan default or customer churn, and follow the same steps.

Why it matters

Employers and reviewers judge ML skill by complete, honest projects rather than isolated algorithms. A documented, reproducible project with a model card is the standard portfolio piece.

How it works

Milestones and rubric

Problem framing: target, users, metric and error costs written down (10 per cent).
Data and leakage audit: what is known at prediction time, splits that mimic use (15 per cent).
Pipeline and baseline: preprocessing in a Pipeline, a trivial and a simple model (15 per cent).
Model comparison and tuning with proper cross-validation (20 per cent).
Threshold from costs, one test evaluation, error analysis (20 per cent).
Interpretability and a model card: intended use, data, metrics, limits, risks (20 per cent).

In code

A compact end-to-end run: baseline, two tuned candidates, a cost-based threshold on validation folds, one test evaluation and the top drivers.

tabular_project.py

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, recall_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_predict, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, 1 - data.target                    # 1 = malignant
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
COST_FP, COST_FN = 1, 20                              # a false alarm against a missed malignancy

def cost(y_true, pred):
    tn, fp, fn, tp = confusion_matrix(y_true, pred).ravel()
    return fp * COST_FP + fn * COST_FN

print("Baseline cost (never flag):", cost(y_te, DummyClassifier().fit(X_tr, y_tr).predict(X_te)))
candidates = {
    "logistic": GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)), {"logisticregression__C": [0.1, 1, 10]}, cv=cv, scoring="roc_auc"),
    "boosting": GridSearchCV(HistGradientBoostingClassifier(random_state=0), {"max_depth": [2, 4], "learning_rate": [0.05, 0.1]}, cv=cv, scoring="roc_auc"),
}
for name, search in candidates.items():
    search.fit(X_tr, y_tr)
    print(f"{name:>9}: CV ROC-AUC {search.best_score_:.3f}  {search.best_params_}")
best_name = max(candidates, key=lambda n: candidates[n].best_score_)
best = candidates[best_name].best_estimator_

oof = cross_val_predict(best, X_tr, y_tr, cv=cv, method="predict_proba")[:, 1]
threshold = min(np.linspace(0.05, 0.6, 56), key=lambda t: cost(y_tr, (oof >= t).astype(int)))
pred = (best.fit(X_tr, y_tr).predict_proba(X_te)[:, 1] >= threshold).astype(int)
print(f"Chosen: {best_name}, threshold {threshold:.2f}")
print(f"Test: recall {recall_score(y_te, pred):.3f}, cost {cost(y_te, pred)}, confusion {confusion_matrix(y_te, pred).ravel().tolist()}")
imp = permutation_importance(best, X_te, y_te, n_repeats=5, random_state=0).importances_mean
print("Top drivers:", [str(data.feature_names[j]) for j in np.argsort(imp)[::-1][:3]])

Output

Baseline cost (never flag): 1060
 logistic: CV ROC-AUC 0.995  {'logisticregression__C': 0.1}
 boosting: CV ROC-AUC 0.988  {'learning_rate': 0.1, 'max_depth': 2}
Chosen: logistic, threshold 0.11
Test: recall 1.000, cost 16, confusion [74, 16, 0, 53]
Top drivers: ['worst radius', 'radius error', 'worst texture']

Try it yourself

Write the model card

Fill in each section for your project in one or two sentences.

Intended use and users: who uses the predictions, for what decision?
Data: source, time period, size, known gaps and who is under-represented.
Metrics: test results with uncertainty, and results per important group.
Threshold and costs: what was chosen and why.
Limits and risks: where the model should not be used, and what could go wrong.

I've done this activity

Common mistakes

Tuning the threshold on the test set.
Comparing models on different splits.
Skipping the model card, so nobody knows the model's limits.

Check yourself

1. Where should the threshold be chosen?

2. What does a model card document?

3. Why compare against a baseline?

Summary

Frame, audit, pipeline, baseline, compare, tune, threshold, test once, explain, document.
Costs drive the metric and the threshold.
A model card makes the work usable and responsible.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Mitchell et al., Model Cards for Model Reporting (2019)