The End-to-End Supervised Workflow

Definition

How do you know a model is ready, and not just lucky on one split?

In simple words

Treat the test set like a sealed exam paper. You may practise as much as you like on training data and rehearse on validation data, but you open the exam only once. A baseline, such as always predicting the most common class, tells you whether your model is actually learning anything.

Why it matters

Most failed ML projects fail in the workflow, not the algorithm: the wrong metric, a test set used for tuning, or no baseline to compare against. A clean workflow is what makes a reported score trustworthy.

How it works

1. Frame: what is predicted, for whom, and which metric reflects the real cost of errors?
2. Split: train / validation / test (for example 60 / 20 / 20), stratified for classification.
3. Baseline: a trivial model, such as the majority class or the mean.
4. Train candidates and compare them on validation data or with cross-validation.
5. Tune the best, then evaluate once on the test set and report that number.
6. Analyse errors: which cases fail, and why? Then iterate on data and features.

In code

A complete workflow on the breast cancer dataset: stratified splits, a baseline, a candidate, validation-based selection and one final test score.

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, recall_score

X, y = load_breast_cancer(return_X_y=True)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=0)
print("Train / val / test sizes:", len(X_train), len(X_val), len(X_test))

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Baseline validation accuracy:", round(accuracy_score(y_val, baseline.predict(X_val)), 3))

candidates = {C: make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=1000)) for C in [0.01, 1, 100]}
scores = {C: accuracy_score(y_val, m.fit(X_train, y_train).predict(X_val)) for C, m in candidates.items()}
for C, s in scores.items():
    print(f"C = {C:<6} validation accuracy {s:.3f}")
best_C = max(scores, key=scores.get)

final = make_pipeline(StandardScaler(), LogisticRegression(C=best_C, max_iter=1000)).fit(X_temp, y_temp)
pred = final.predict(X_test)
print("Chosen C:", best_C)
print("Test accuracy:", round(accuracy_score(y_test, pred), 3), "| test recall:", round(recall_score(y_test, pred), 3))

Train / val / test sizes: 341 114 114
Baseline validation accuracy: 0.623
C = 0.01   validation accuracy 0.965
C = 1      validation accuracy 0.991
C = 100    validation accuracy 0.974
Chosen C: 1
Test accuracy: 0.982 | test recall: 1.0

Try it yourself

Common mistakes

• Tuning on the test set, which turns it into a second validation set and inflates the score.
• Skipping the baseline, so 95 per cent accuracy looks good even when 94 per cent is trivial.
• Choosing a metric for convenience rather than for the real cost of errors.

Check yourself

Summary

• Frame, split, baseline, train, tune, test once, analyse.
• Keep the test set sealed until the final evaluation.
• Always compare against a baseline.