Seeing Images, Explaining Decisions¶
CIFAR-10 image classification — Gareth Grant¶
Python · TensorFlow / Keras · Computer vision · Reproducible evaluation
This project compares simple baselines with convolutional neural networks, explains their errors, and connects the method to decisions in business, astronomy and government. Each numbered step has one main job, its own code cell, and a short explanation. Edit a code comment or insert a Markdown cell beneath any step to add your observations.
The code trains CIFAR-10 classifiers only. The later domain applications are proposed adaptations requiring different labelled data and independent validation. The distributed executed copy identifies its budget in Step 4. A quick mode is also available. This project demonstrates an understandable workflow, not state-of-the-art performance.
How to use the notebook¶
- Install the accompanying
requirements.txtin a Python 3.12 environment. - Open this file in Jupyter or VS Code and select that environment.
- Read Step 4, restart the kernel, and run all cells in order.
- Keras downloads the official CIFAR-10 archive on the first run (about 163 MiB).
- Read each plot and its limitations before quoting a result.
RUN_MODE = "full" uses all cleaned development rows and a 25-epoch ceiling per CNN.
RUN_MODE = "quick" uses 8,000 training and 2,000 validation images with a 6-epoch ceiling.
Both modes keep all 10,000 official test images. Training time depends on the hardware.
Changing settings requires restarting and running from the beginning.
Outputs are saved under results/<mode>_seed<seed>/. The model, class order, manifest,
plots, metrics, row predictions and development split record are reusable artifacts.
Review of the supplied notebook¶
| Finding | Why it mattered | Correction |
|---|---|---|
| Intel six-class data described; CIFAR-10 loaded | The interpretation described the wrong data | One consistent ten-class project |
| 587 lines in one code cell | Difficult to annotate, learn from, or debug | Small cells for separate operations |
Saved SyntaxError in the main cell |
The supplied code could not run | Valid Python and clean sequential execution |
| Test data used in early stopping and tuning | Evaluation information influenced development | Separate train and validation data; frozen decisions before test scoring |
improved_model built but the existing pipeline fitted again |
Scores were not clearly attached to that model | Named Keras models with their own predictions |
| Reused tuner names with inconsistent ranges | Search definitions were ambiguous | Two explicit learning-rate trials |
| Augmentation object not used by fitting | Its claimed benefit was not implemented | Training-only layers inside the model |
| Balanced classes said to imply no bias | Equal counts cannot rule out other bias | Coverage and deployment limitations explained |
| One-hot encoding described as essential | Sparse cross-entropy accepts integer labels | Keep integer labels consistently |
| Fixed claims without successful saved results | Text could disagree with outputs | Interpretation generated from measured predictions |
The supplied file contains an error, not a valid saved trained-model score. Any result below is generated by this replacement workflow.
Problem and evaluation plan¶
Question: how much does a compact CNN improve on a simple classifier, and where do its errors suggest additional work is needed?
CIFAR-10 contains ten mutually exclusive object labels and low-resolution RGB photographs. The dataset creators describe 60,000 images of 32 × 32 pixels, split into 50,000 development images and 10,000 official test images [1]. This makes a useful teaching benchmark; it does not represent commercial defects, galaxies or multispectral satellite imagery.
This is whole-image classification. It does not locate an object with a box, label individual pixels, recognise arbitrary new classes, or infer a scientific cause.
| Decision fixed before fitting | Rule |
|---|---|
| Development split | Stratified 80% train / 20% validation after exact-duplicate cleaning |
| Duplicate policy | Remove conflicting development hash groups, repeated rows, and development copies of test images using hashes only |
| Reference models | Majority class and regularised linear classifier on coarse RGB features |
| CNN experiments | Basic CNN; regularised CNN at learning rates 0.001 and 0.0005 |
| Checkpoint | Lowest validation loss; early stopping restores its weights |
| CNN choice | Highest validation macro-F1; fewer parameters break a tie |
| Review threshold | Validation-only accuracy lower-bound target of 90%, with at least 100 accepted cases |
| Test comparison | Report every declared model; do not select a new CNN from its test score |
| Uncertainty | Paired bootstrap of test-image hash groups, conditional on fitted models |
Macro-F1 weights classes equally. Accuracy describes the overall fraction correct. Precision, recall, cross-entropy and the error images describe different failure modes.
1. Setup and reproducibility¶
Optional first-time installation in a notebook: %pip install -r requirements.txt.
Restart the kernel afterwards. Record versions and seeds with the result. A fixed seed
reduces variation but cannot guarantee identical results on different software or hardware.
01. Import general utilities¶
These imports support paths, timing, hashes, tables, figures and image loading.
Python · cell 01
# No training or downloading happens in this cell.
import os
os.environ.setdefault("TF_CPP_MIN_LOG_LEVEL", "2")
import json, hashlib, platform, time
from pathlib import Path
from importlib.metadata import version, PackageNotFoundError
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from PIL import Image, ImageOps
from IPython.display import display, Markdown
02. Import splitting, baselines and metrics¶
All model comparisons use the same metric definitions and class order.
Python · cell 02
# Integer class IDs are retained throughout the project.
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
accuracy_score, f1_score, log_loss, classification_report,
confusion_matrix, ConfusionMatrixDisplay, roc_auc_score,
roc_curve, top_k_accuracy_score,
)
03. Import TensorFlow and Keras¶
Use the public model API directly instead of assigning a wrapper’s internal attributes.
Python · cell 03
# The CNNs are trained from scratch, not with transfer learning.
import tensorflow as tf
keras = tf.keras
layers = keras.layers
04. Choose the experiment budget¶
This is the main editable settings cell. Keep the test set intact in both modes.
Python · cell 04
# Use "quick" for a smaller first learning run.
RUN_MODE = "full"
SEED = 42
BATCH_SIZE = 128
CPU_THREADS = 8
EPOCHS = 6 if RUN_MODE == "quick" else 25
QUICK_TRAIN_SIZE, QUICK_VALIDATION_SIZE = 8_000, 2_000
TARGET_ACCEPTED_ACCURACY = 0.90
BOOTSTRAP_REPEATS = 500
LOCAL_DATA = Path(os.environ.get("CIFAR10_NPZ", "data/cifar10.npz"))
OUTPUT_DIR = Path("results") / f"{RUN_MODE}_seed{SEED}"
assert RUN_MODE in {"quick", "full"}
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
display(pd.Series({"mode": RUN_MODE, "seed": SEED, "batch_size": BATCH_SIZE,
"maximum_epochs_per_CNN": EPOCHS, "test_images": 10_000,
"output_folder": str(OUTPUT_DIR)}))
mode full seed 42 batch_size 128 maximum_epochs_per_CNN 25 test_images 10000 output_folder results/full_seed42 dtype: object
Your comment: Why start with a smaller experiment? What conclusions need a larger budget or repeated seeds?
05. Set seeds and CPU limits¶
Run this before model operations. Deterministic operations help reproducibility [2].
Python · cell 05
# Restart the kernel before resetting these after training.
keras.utils.set_random_seed(SEED)
tf.config.threading.set_intra_op_parallelism_threads(CPU_THREADS)
tf.config.threading.set_inter_op_parallelism_threads(2)
tf.config.experimental.enable_op_determinism()
print("Devices:", [device.device_type for device in tf.config.list_physical_devices()])
Devices: ['CPU']
06. Record software versions¶
These values identify the environment actually used, rather than assumed versions.
Python · cell 06
# Save these with the model manifest later.
software_versions = {"python": platform.python_version(), "tensorflow": tf.__version__}
for name in ["tensorflow-cpu", "keras", "numpy", "pandas", "scikit-learn",
"matplotlib", "Pillow", "nbformat", "nbconvert"]:
try:
software_versions[name] = version(name)
except PackageNotFoundError:
pass
display(pd.Series(software_versions, name="version"))
python 3.12.14 tensorflow 2.20.0 tensorflow-cpu 2.20.0 keras 3.15.1 numpy 2.3.5 pandas 2.2.3 scikit-learn 1.8.0 matplotlib 3.10.8 Pillow 12.3.0 nbformat 5.10.4 nbconvert 7.17.1 Name: version, dtype: object
07. Choose a consistent plot style¶
Readable navy and scarlet figures fit the portfolio’s palette.
Python · cell 07
# Style choices do not affect the trained model.
NAVY, SCARLET, SLATE = "#16324F", "#FF2400", "#667582"
plt.rcParams.update({"figure.dpi": 110, "savefig.dpi": 170,
"axes.spines.top": False, "axes.spines.right": False,
"axes.labelcolor": NAVY, "axes.titleweight": "bold", "font.size": 10})
08. Define a figure-saving helper¶
Display each chart and save a PNG suitable for a website.
Python · cell 08
# Each chart calls this helper once.
def finish_figure(figure, filename):
figure.tight_layout()
figure.savefig(OUTPUT_DIR / filename, bbox_inches="tight")
plt.show()
plt.close(figure)
2. Load and audit the real images¶
Keras supplies the official data [3]. An offline cache may instead contain x_train,
y_train, x_test, y_test in data/cifar10.npz with the official shapes and labels.
The creators publish archive MD5 c58f30108f718f92721af3b95e74349a [1].
Keep raw images as uint8. Sparse categorical cross-entropy accepts integer labels;
one-hot encoding is optional.
09. Define the class-name mapping¶
The name at each position must match the corresponding official class ID.
Python · cell 09
# Preserve this order in the exported model's manifest.
CLASS_NAMES = ["airplane", "automobile", "bird", "cat", "deer",
"dog", "frog", "horse", "ship", "truck"]
CLASS_IDS = np.arange(10)
display(pd.DataFrame({"class_id": CLASS_IDS, "class_name": CLASS_NAMES}))
| class_id | class_name | |
|---|---|---|
| 0 | 0 | airplane |
| 1 | 1 | automobile |
| 2 | 2 | bird |
| 3 | 3 | cat |
| 4 | 4 | deer |
| 5 | 5 | dog |
| 6 | 6 | frog |
| 7 | 7 | horse |
| 8 | 8 | ship |
| 9 | 9 | truck |
10. Load development and official test arrays¶
Test labels are loaded for later scoring and are not used for fitting or selection.
Python · cell 10
# An environment variable can point to a local offline cache.
if LOCAL_DATA.is_file():
with np.load(LOCAL_DATA, allow_pickle=False) as data:
x_development, y_development = data["x_train"], data["y_train"].reshape(-1)
x_test, y_test = data["x_test"], data["y_test"].reshape(-1)
data_route = "local CIFAR-10 NumPy cache"
else:
(x_development, y_development), (x_test, y_test) = keras.datasets.cifar10.load_data()
y_development, y_test = y_development.reshape(-1), y_test.reshape(-1)
data_route = "official CIFAR-10 via Keras"
print("Source:", data_route)
print("Development:", x_development.shape, "Test:", x_test.shape)
Source: local CIFAR-10 NumPy cache Development: (50000, 32, 32, 3) Test: (10000, 32, 32, 3)
11. Verify the data contract¶
Fail early if dimensions, dtype or development label counts do not match. These checks alone cannot authenticate an arbitrary local file.
Python · cell 11
# Input scaling stays inside the CNN.
assert x_development.shape == (50_000, 32, 32, 3)
assert x_test.shape == (10_000, 32, 32, 3)
assert x_development.dtype == x_test.dtype == np.uint8
assert y_development.shape == (50_000,) and y_test.shape == (10_000,)
assert np.issubdtype(y_development.dtype, np.integer)
assert np.issubdtype(y_test.dtype, np.integer)
assert np.array_equal(np.unique(y_development), CLASS_IDS)
assert np.array_equal(np.bincount(y_development), np.full(10, 5_000))
print("Image and development label contracts verified.")
Image and development label contracts verified.
12. Fingerprint exact images¶
Image hashes identify byte-identical copies, without consulting test labels.
Python · cell 12
# These hashes also define paired bootstrap groups later.
def image_hashes(images):
return np.array([hashlib.sha256(image.tobytes()).hexdigest() for image in images])
development_hashes = image_hashes(x_development)
test_hashes = image_hashes(x_test)
13. Check repeated images and conflicting development labels¶
The same image must not appear in both training and validation.
Python · cell 13
# Investigate development labels only at this stage.
development_audit = pd.DataFrame({"hash": development_hashes, "label": y_development})
labels_per_hash = development_audit.groupby("hash")["label"].nunique()
conflicting_hashes = set(labels_per_hash[labels_per_hash > 1].index)
repeated_development = int(development_audit.duplicated("hash").sum())
print("Repeated development rows:", repeated_development)
print("Conflicting development hash groups:", len(conflicting_hashes))
Repeated development rows: 0 Conflicting development hash groups: 0
14. Apply the declared duplicate policy¶
Remove development copies of test images using identity only. Official test rows stay intact.
Python · cell 14
# Exclusion counts may overlap; do not sum them.
overlaps_test = development_audit["hash"].isin(set(test_hashes))
conflicts = development_audit["hash"].isin(conflicting_hashes)
repeated = development_audit.duplicated("hash", keep="first")
clean_ids = np.flatnonzero((~(overlaps_test | conflicts | repeated)).to_numpy())
cleaning_summary = pd.Series({"original_development_rows": len(x_development),
"development_rows_matching_test": int(overlaps_test.sum()),
"conflicting_label_rows": int(conflicts.sum()),
"repeated_development_rows": repeated_development,
"retained_development_rows": len(clean_ids), "test_rows_retained": len(x_test)})
display(cleaning_summary)
original_development_rows 50000 development_rows_matching_test 0 conflicting_label_rows 0 repeated_development_rows 0 retained_development_rows 50000 test_rows_retained 10000 dtype: int64
Interpretation: exact hashes do not detect near duplicates or repeated views of one source object. Cleaning can also change comparability with standard benchmark protocols. Your comment: Which product IDs, astronomical object IDs or geographic identifiers would you need to prevent source-level leakage in a real application?
15. Create stratified train and validation subsets¶
Split cleaned development data, preserving approximate class proportions.
Python · cell 15
# The official test arrays do not enter this operation.
train_ids, validation_ids = train_test_split(clean_ids, test_size=0.20,
random_state=SEED, stratify=y_development[clean_ids])
16. Apply the optional quick-run sample¶
Sample within the already separate development subsets. Full mode retains every cleaned row.
Python · cell 16
# Quick sampling uses development labels only.
if RUN_MODE == "quick":
train_ids, _ = train_test_split(train_ids, train_size=QUICK_TRAIN_SIZE,
random_state=SEED, stratify=y_development[train_ids])
validation_ids, _ = train_test_split(validation_ids,
train_size=QUICK_VALIDATION_SIZE, random_state=SEED,
stratify=y_development[validation_ids])
x_train, y_train = x_development[train_ids], y_development[train_ids]
x_validation, y_validation = x_development[validation_ids], y_development[validation_ids]
17. Assert that image identities do not cross subsets¶
Make the leakage rule executable, not just a statement in the report.
Python · cell 17
# No selected development image can also be a test image.
train_hash_set = set(development_hashes[train_ids])
validation_hash_set = set(development_hashes[validation_ids])
assert set(train_ids).isdisjoint(validation_ids)
assert train_hash_set.isdisjoint(validation_hash_set)
assert train_hash_set.isdisjoint(test_hashes)
assert validation_hash_set.isdisjoint(test_hashes)
display(pd.DataFrame({"split": ["train", "validation", "official test"],
"images": [len(x_train), len(x_validation), len(x_test)]}))
| split | images | |
|---|---|---|
| 0 | train | 40000 |
| 1 | validation | 10000 |
| 2 | official test | 10000 |
18. Save the development split and cleaning record¶
Original row IDs and hashes allow another person to reconstruct the selected examples.
Python · cell 18
# Only the chosen development rows are listed in this record.
split_record = pd.concat([
pd.DataFrame({"original_row": train_ids, "split": "train"}),
pd.DataFrame({"original_row": validation_ids, "split": "validation"}),
], ignore_index=True)
split_record["image_sha256"] = development_hashes[split_record["original_row"]]
split_record.to_csv(OUTPUT_DIR / "development_split.csv", index=False)
cleaning_summary.to_csv(OUTPUT_DIR / "data_cleaning.csv", header=["value"])
print("Saved split identities and cleaning counts.")
Saved split identities and cleaning counts.
3. Explore training data¶
EDA describes selected training data, not a test set used for development. Equal class counts do not rule out background, viewpoint, source or population bias. A few displayed examples cannot verify every label.
19. Count training and validation classes¶
Confirm support before fitting. Test support is examined only in the final evaluation.
Python · cell 19
# Counts come from the actual subsets used in this run.
class_counts = pd.DataFrame({"class": CLASS_NAMES,
"train": np.bincount(y_train, minlength=10),
"validation": np.bincount(y_validation, minlength=10)})
display(class_counts)
| class | train | validation | |
|---|---|---|---|
| 0 | airplane | 4000 | 1000 |
| 1 | automobile | 4000 | 1000 |
| 2 | bird | 4000 | 1000 |
| 3 | cat | 4000 | 1000 |
| 4 | deer | 4000 | 1000 |
| 5 | dog | 4000 | 1000 |
| 6 | frog | 4000 | 1000 |
| 7 | horse | 4000 | 1000 |
| 8 | ship | 4000 | 1000 |
| 9 | truck | 4000 | 1000 |
20. Plot training class support¶
A count chart reveals whether sampling changed class balance.
Python · cell 20
# One chart describes one property: class support.
figure, axis = plt.subplots(figsize=(10, 4))
axis.bar(CLASS_NAMES, class_counts["train"], color=NAVY)
axis.set(title="Training images by class", ylabel="Images")
axis.tick_params(axis="x", rotation=35)
finish_figure(figure, "01_class_distribution.png")
Your comment: Which sources of bias could remain if every bar had the same height?
21. Select seeded examples per class¶
Use random selections rather than describing the first rows as random.
Python · cell 21
# Two examples per class make a compact balanced grid.
sample_rng = np.random.default_rng(SEED)
sample_ids = [index for class_id in CLASS_IDS for index in
sample_rng.choice(np.flatnonzero(y_train == class_id), size=2, replace=False)]
22. Display training examples¶
Look at backgrounds, pose, occlusion and detail. Enlarging a 32-pixel image does not recover missing information.
Python · cell 22
# These are training examples, not model predictions.
figure, axes = plt.subplots(4, 5, figsize=(10, 8))
for axis, index in zip(axes.flat, sample_ids):
axis.imshow(x_train[index], interpolation="nearest")
axis.set_title(CLASS_NAMES[y_train[index]])
axis.axis("off")
figure.suptitle("Seeded training examples", color=NAVY)
finish_figure(figure, "02_training_examples.png")
Your comment: Which images would also be difficult for a person to label at this resolution?
23. Summarise a training pixel sample¶
Avoid allocating a float copy of the entire benchmark just to describe intensities.
Python · cell 23
# These are descriptive statistics, not fitted preprocessing parameters.
statistics_ids = sample_rng.choice(len(x_train), size=min(2_000, len(x_train)), replace=False)
sampled_pixels = x_train[statistics_ids].astype(np.float32) / 255.0
pixel_statistics = pd.DataFrame({"channel": ["red", "green", "blue"],
"mean": sampled_pixels.mean(axis=(0, 1, 2)),
"standard_deviation": sampled_pixels.std(axis=(0, 1, 2))})
display(pixel_statistics)
| channel | mean | standard_deviation | |
|---|---|---|---|
| 0 | red | 0.491638 | 0.245968 |
| 1 | green | 0.483317 | 0.242473 |
| 2 | blue | 0.449165 | 0.261160 |
24. Plot sampled intensities¶
A histogram describes intensity; it does not prove that labels are separable.
Python · cell 24
# The sample is seeded and comes only from training.
figure, axis = plt.subplots(figsize=(8, 4))
for channel, colour in enumerate(["#C7353F", "#247B73", NAVY]):
axis.hist(sampled_pixels[..., channel].ravel()[::20], bins=40, alpha=0.4,
density=True, color=colour, label=pixel_statistics.loc[channel, "channel"])
axis.set(title="Sampled training intensities", xlabel="Pixel / 255", ylabel="Density")
axis.legend()
finish_figure(figure, "03_pixel_intensities.png")
4. Prepare batching and simple baselines¶
CNNs receive raw RGB values in 0–255. Rescaling is inside the model so export and training use the same transformation. The linear model uses block-averaged 8 × 8 RGB features, with a scaler fitted on training data only. Different model inputs are declared clearly.
25. Define TensorFlow batching¶
Shuffle training only; validation prediction order must remain aligned with its labels.
Python · cell 25
# Keep the private CPU thread pool bounded.
def make_dataset(images, labels, training=False):
dataset = tf.data.Dataset.from_tensor_slices((images, labels))
if training:
dataset = dataset.shuffle(len(images), seed=SEED, reshuffle_each_iteration=True)
options = tf.data.Options()
options.threading.private_threadpool_size = 2
options.threading.max_intra_op_parallelism = CPU_THREADS
options.experimental_deterministic = True
return dataset.batch(BATCH_SIZE).with_options(options).prefetch(1)
26. Create development batches¶
Test batches are created after the development decision is frozen.
Python · cell 26
# Validation data is not shuffled or augmented here.
train_dataset = make_dataset(x_train, y_train, training=True)
validation_dataset = make_dataset(x_validation, y_validation)
print("Training batches:", len(train_dataset), "Validation batches:", len(validation_dataset))
Training batches: 313 Validation batches: 79
27. Define consistent probability-based metrics¶
Check probability shape and class order before calculating quality metrics.
Python · cell 27
# Accuracy, macro-F1 and weighted F1 answer different questions.
def metric_row(name, labels, probabilities):
probabilities = np.asarray(probabilities)
assert probabilities.shape == (len(labels), 10)
assert np.isfinite(probabilities).all() and np.all(probabilities >= 0)
assert np.allclose(probabilities.sum(axis=1), 1, atol=1e-5)
predicted = probabilities.argmax(axis=1)
return {"model": name, "accuracy": accuracy_score(labels, predicted),
"macro_f1": f1_score(labels, predicted, average="macro", zero_division=0),
"weighted_f1": f1_score(labels, predicted, average="weighted", zero_division=0),
"log_loss": log_loss(labels, probabilities, labels=CLASS_IDS)}
28. Initialise named experiment records¶
Each score stays attached to its model, predictions and training history.
Python · cell 28
# These records start empty on each clean run.
validation_rows, validation_probabilities = [], {}
cnn_models, cnn_histories, training_seconds = {}, {}, {}
29. Fit a majority-class reference¶
Measure what is achieved by always guessing the most frequent training label.
Python · cell 29
# The dummy classifier does not learn image content.
majority_model = DummyClassifier(strategy="most_frequent")
majority_model.fit(np.zeros((len(y_train), 1)), y_train)
majority_validation = majority_model.predict_proba(np.zeros((len(y_validation), 1)))
assert np.array_equal(majority_model.classes_, CLASS_IDS)
row = metric_row("majority", y_validation, majority_validation)
row.update(parameters=0, training_seconds=0.0)
validation_rows.append(row)
validation_probabilities["majority"] = majority_validation
display(pd.Series(row))
model majority accuracy 0.1 macro_f1 0.018182 weighted_f1 0.018182 log_loss 32.439288 parameters 0 training_seconds 0.0 dtype: object
Interpretation: on balanced ten-class data, accuracy is about 10%. Macro-F1 is much lower because nine classes are never correctly predicted. Very high log loss reflects the dummy’s zero-probability assignments to the other classes.
30. Define coarse RGB features¶
A linear reference tests whether coarse colour and position already predict useful information.
Python · cell 30
# Each 4 x 4 RGB block becomes one average colour.
def coarse_features(images):
blocks = images.reshape(-1, 8, 4, 8, 4, 3)
return (blocks.mean(axis=(2, 4)) / 255.0).reshape(len(images), -1)
31. Fit a regularised linear model¶
Fit preprocessing with the classifier to keep validation statistics out of the scaler.
Python · cell 31
# C=1 is fixed in advance; there is no hidden linear-model search.
linear_model = make_pipeline(StandardScaler(),
LogisticRegression(C=1.0, solver="lbfgs", max_iter=1_000, random_state=SEED))
linear_start = time.perf_counter()
linear_model.fit(coarse_features(x_train), y_train)
linear_seconds = time.perf_counter() - linear_start
32. Record linear validation results¶
These are validation scores, not final test scores.
Python · cell 32
# Save the actual probabilities used in the comparison.
linear_validation = linear_model.predict_proba(coarse_features(x_validation))
assert np.array_equal(linear_model.classes_, CLASS_IDS)
row = metric_row("linear_RGB_8x8", y_validation, linear_validation)
linear_estimator = linear_model.named_steps["logisticregression"]
row.update(parameters=linear_estimator.coef_.size + linear_estimator.intercept_.size,
training_seconds=linear_seconds)
validation_rows.append(row)
validation_probabilities["linear_RGB_8x8"] = linear_validation
display(pd.Series(row))
model linear_RGB_8x8 accuracy 0.4154 macro_f1 0.412006 weighted_f1 0.412006 log_loss 1.719292 parameters 1930 training_seconds 10.40988 dtype: object
5. Train the basic CNN¶
Convolution learns filters shared over image positions. Pooling reduces spatial size. Dense layers combine learned features into ten scores, and softmax gives a distribution over the known labels. This compact CNN follows the general teaching approach in the TensorFlow tutorial [3]. It is trained from scratch.
33. Build the basic architecture¶
Keep input scaling inside the model. This reference CNN uses no augmentation or dropout.
Python · cell 33
# One constructor defines one architecture.
def build_basic_cnn():
return keras.Sequential([
keras.Input(shape=(32, 32, 3), name="raw_rgb_0_to_255"),
layers.Rescaling(1.0 / 255.0),
layers.Conv2D(32, 3, activation="relu"),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, activation="relu"),
layers.MaxPooling2D(),
layers.Conv2D(64, 3, activation="relu"),
layers.Flatten(), layers.Dense(64, activation="relu"),
layers.Dense(10, activation="softmax"),
], name="basic_cnn")
34. Define the common compilation rule¶
All CNNs use Adam and sparse categorical cross-entropy with integer labels.
Python · cell 34
# Only the declared learning rate changes across regularised trials.
def compile_cnn(model, learning_rate):
model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate),
loss=keras.losses.SparseCategoricalCrossentropy(),
metrics=[keras.metrics.SparseCategoricalAccuracy(name="accuracy")])
return model
35. Define a fresh early-stopping callback¶
Monitor validation loss and restore its best weights [4]. Create a new callback for every fit.
Python · cell 35
# Never give test data to this callback.
def early_stopping():
return keras.callbacks.EarlyStopping(monitor="val_loss", patience=3,
min_delta=1e-4, restore_best_weights=True, verbose=1)
36. Define one timed training operation¶
Keep the budget and validation source the same across CNNs.
Python · cell 36
# Training is already shuffled by the tf.data pipeline.
def train_cnn(model):
start = time.perf_counter()
history = model.fit(train_dataset, validation_data=validation_dataset,
epochs=EPOCHS, callbacks=[early_stopping()], shuffle=False, verbose=2)
return history.history, time.perf_counter() - start
37. Create and inspect the basic model¶
Inspect shapes and parameter count before training.
Python · cell 37
# Reset the seed before this model's initialisation.
keras.utils.set_random_seed(SEED)
basic_model = compile_cnn(build_basic_cnn(), learning_rate=1e-3)
basic_model.summary()
Model: "basic_cnn" ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ rescaling (Rescaling) │ (None, 32, 32, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d (Conv2D) │ (None, 30, 30, 32) │ 896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d (MaxPooling2D) │ (None, 15, 15, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_1 (Conv2D) │ (None, 13, 13, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_1 (MaxPooling2D) │ (None, 6, 6, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_2 (Conv2D) │ (None, 4, 4, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 64) │ 65,600 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 10) │ 650 │ └─────────────────────────────────┴────────────────────────┴───────────────┘ Total params: 122,570 (478.79 KB) Trainable params: 122,570 (478.79 KB) Non-trainable params: 0 (0.00 B)
38. Train the basic CNN¶
Fit only on training images; validation supports checkpointing and model development.
Python · cell 38
# This is the first CNN fit in the experiment.
basic_history, basic_seconds = train_cnn(basic_model)
cnn_models["basic_CNN"] = basic_model
cnn_histories["basic_CNN"] = basic_history
training_seconds["basic_CNN"] = basic_seconds
Epoch 1/25 313/313 - 6s - 20ms/step - accuracy: 0.3938 - loss: 1.6537 - val_accuracy: 0.4931 - val_loss: 1.3889 Epoch 2/25 313/313 - 5s - 17ms/step - accuracy: 0.5281 - loss: 1.3157 - val_accuracy: 0.5429 - val_loss: 1.2856 Epoch 3/25 313/313 - 5s - 17ms/step - accuracy: 0.5839 - loss: 1.1737 - val_accuracy: 0.5986 - val_loss: 1.1343 Epoch 4/25 313/313 - 5s - 18ms/step - accuracy: 0.6275 - loss: 1.0636 - val_accuracy: 0.6304 - val_loss: 1.0606 Epoch 5/25 313/313 - 6s - 18ms/step - accuracy: 0.6526 - loss: 0.9940 - val_accuracy: 0.6266 - val_loss: 1.0723 Epoch 6/25 313/313 - 5s - 17ms/step - accuracy: 0.6777 - loss: 0.9269 - val_accuracy: 0.6382 - val_loss: 1.0570 Epoch 7/25 313/313 - 6s - 18ms/step - accuracy: 0.6928 - loss: 0.8820 - val_accuracy: 0.6705 - val_loss: 0.9696 Epoch 8/25 313/313 - 5s - 17ms/step - accuracy: 0.7081 - loss: 0.8402 - val_accuracy: 0.6778 - val_loss: 0.9235 Epoch 9/25 313/313 - 5s - 18ms/step - accuracy: 0.7240 - loss: 0.7920 - val_accuracy: 0.6876 - val_loss: 0.9025 Epoch 10/25 313/313 - 5s - 17ms/step - accuracy: 0.7343 - loss: 0.7625 - val_accuracy: 0.6801 - val_loss: 0.9440 Epoch 11/25 313/313 - 5s - 17ms/step - accuracy: 0.7504 - loss: 0.7207 - val_accuracy: 0.6872 - val_loss: 0.9225 Epoch 12/25 313/313 - 5s - 17ms/step - accuracy: 0.7576 - loss: 0.6980 - val_accuracy: 0.6854 - val_loss: 0.9211 Epoch 12: early stopping Restoring model weights from the end of the best epoch: 9.
39. Record basic-CNN validation results¶
The trained model object directly generates these probabilities.
Python · cell 39
# Validation batches preserve the original label order.
basic_validation = basic_model.predict(validation_dataset, verbose=0)
row = metric_row("basic_CNN", y_validation, basic_validation)
row.update(parameters=basic_model.count_params(), training_seconds=basic_seconds)
validation_rows.append(row)
validation_probabilities["basic_CNN"] = basic_validation
display(pd.Series(row))
model basic_CNN accuracy 0.6876 macro_f1 0.685669 weighted_f1 0.685669 log_loss 0.902453 parameters 122570 training_seconds 65.907264 dtype: object
40. Plot basic-CNN accuracy¶
Look for validation improvement and its distance from the training trend.
Python · cell 40
# Epoch numbering starts at 1.
figure, axis = plt.subplots(figsize=(8, 4))
epochs_completed = np.arange(1, len(basic_history["accuracy"]) + 1)
axis.plot(epochs_completed, basic_history["accuracy"], color=NAVY, label="train")
axis.plot(epochs_completed, basic_history["val_accuracy"], color=SCARLET, label="validation")
axis.set(title="Basic CNN: accuracy", xlabel="Epoch", ylabel="Accuracy", ylim=(0, 1))
axis.legend()
finish_figure(figure, "04_basic_accuracy.png")
41. Plot basic-CNN loss¶
Rising validation loss alongside falling training loss suggests investigating overfitting.
Python · cell 41
# The restored weights need not correspond to the last plotted epoch.
figure, axis = plt.subplots(figsize=(8, 4))
axis.plot(epochs_completed, basic_history["loss"], color=NAVY, label="train")
axis.plot(epochs_completed, basic_history["val_loss"], color=SCARLET, label="validation")
axis.set(title="Basic CNN: cross-entropy", xlabel="Epoch", ylabel="Loss")
axis.legend()
finish_figure(figure, "05_basic_loss.png")
Your comment: Are the curves still improving? A short budget may leave a model undertrained; a gap alone does not establish overfitting.
6. Compare regularisation and learning rates¶
Horizontal flips and small translations can preserve these object labels; vertical flips can create implausible upright scenes. Keras augmentation layers are active during training, not inference [5]. Augmentation does not create new independent labelled observations.
The regularised architecture also changes normalisation, pooling, dropout and weight
penalties. This combined comparison cannot isolate which individual change caused a difference.
Batch normalisation uses batch statistics in training and moving statistics at inference [11].
momentum=0.90 lets moving values adapt faster than the default 0.99; short validation
runs can expose a training/inference mismatch. The setting is fixed for both trials.
42. Define training-only augmentation¶
Put augmentation inside the model so it cannot be accidentally omitted during fitting.
Python · cell 42
# These transformations still need domain-specific judgement.
def make_augmentation():
return keras.Sequential([
layers.RandomFlip("horizontal", seed=SEED),
layers.RandomTranslation(0.10, 0.10, fill_mode="reflect", seed=SEED + 1),
], name="training_augmentation")
43. Preview augmented training images¶
Check that transformations preserve the intended label before using them.
Python · cell 43
# A separate object previews raw 0-255 pixels.
preview_images = make_augmentation()(x_train[sample_ids[:8]], training=True).numpy()
figure, axes = plt.subplots(2, 4, figsize=(8, 4))
for axis, image, index in zip(axes.flat, preview_images, sample_ids[:8]):
axis.imshow(np.clip(image, 0, 255).astype(np.uint8))
axis.set_title(CLASS_NAMES[y_train[index]])
axis.axis("off")
figure.suptitle("Training-only flips and translations", color=NAVY)
finish_figure(figure, "06_augmentation_preview.png")
Your comment: Would these transformations preserve traffic-sign text, detector defects or calibrated astronomical measurements?
44. Build the regularised CNN¶
Global average pooling reduces the dense input size; dropout and L2 penalties aim to reduce memorisation.
Python · cell 44
# Both learning-rate trials use this exact constructor.
def build_regularised_cnn():
model = keras.Sequential([
keras.Input(shape=(32, 32, 3), name="raw_rgb_0_to_255"),
layers.Rescaling(1.0 / 255.0), make_augmentation(),
], name="regularised_cnn")
for filters in (32, 64, 128):
model.add(layers.Conv2D(filters, 3, padding="same", use_bias=False,
kernel_regularizer=keras.regularizers.l2(1e-4)))
model.add(layers.BatchNormalization(momentum=0.90))
model.add(layers.Activation("relu"))
model.add(layers.MaxPooling2D())
model.add(layers.GlobalAveragePooling2D())
model.add(layers.Dense(128, activation="relu"))
model.add(layers.Dropout(0.30, seed=SEED + 2))
model.add(layers.Dense(10, activation="softmax"))
return model
45. Declare the learning-rate trials¶
The search is small and explicit rather than an expensive ambiguous tuner.
Python · cell 45
# No trial settings are chosen from the test results.
trial_plan = pd.DataFrame([
{"model": "regularised_CNN_lr0.001", "learning_rate": 1e-3},
{"model": "regularised_CNN_lr0.0005", "learning_rate": 5e-4},
])
display(trial_plan.assign(maximum_epochs=EPOCHS))
| model | learning_rate | maximum_epochs | |
|---|---|---|---|
| 0 | regularised_CNN_lr0.001 | 0.0010 | 25 |
| 1 | regularised_CNN_lr0.0005 | 0.0005 | 25 |
46. Train and record the two trials¶
Keep each fitted model, history and validation probabilities under its own name.
Python · cell 46
# Identical seeds support a controlled learning-rate comparison.
for trial in trial_plan.to_dict("records"):
name = trial["model"]
keras.utils.set_random_seed(SEED)
model = compile_cnn(build_regularised_cnn(), trial["learning_rate"])
history, seconds = train_cnn(model)
probabilities = model.predict(validation_dataset, verbose=0)
row = metric_row(name, y_validation, probabilities)
row.update(parameters=model.count_params(), training_seconds=seconds)
validation_rows.append(row)
validation_probabilities[name] = probabilities
cnn_models[name], cnn_histories[name] = model, history
training_seconds[name] = seconds
print(name, "validation macro-F1:", round(row["macro_f1"], 4))
Epoch 1/25 313/313 - 16s - 50ms/step - accuracy: 0.4110 - loss: 1.6144 - val_accuracy: 0.4181 - val_loss: 1.6146 Epoch 2/25 313/313 - 15s - 47ms/step - accuracy: 0.5253 - loss: 1.3267 - val_accuracy: 0.4287 - val_loss: 1.6679 Epoch 3/25 313/313 - 14s - 45ms/step - accuracy: 0.5734 - loss: 1.2137 - val_accuracy: 0.5234 - val_loss: 1.3305 Epoch 4/25 313/313 - 13s - 43ms/step - accuracy: 0.5949 - loss: 1.1554 - val_accuracy: 0.4877 - val_loss: 1.5364 Epoch 5/25 313/313 - 14s - 44ms/step - accuracy: 0.6201 - loss: 1.0901 - val_accuracy: 0.5284 - val_loss: 1.4521 Epoch 6/25 313/313 - 14s - 46ms/step - accuracy: 0.6334 - loss: 1.0494 - val_accuracy: 0.5354 - val_loss: 1.4042 Epoch 6: early stopping Restoring model weights from the end of the best epoch: 3. regularised_CNN_lr0.001 validation macro-F1: 0.5106 Epoch 1/25 313/313 - 15s - 47ms/step - accuracy: 0.3977 - loss: 1.6569 - val_accuracy: 0.5225 - val_loss: 1.3432 Epoch 2/25 313/313 - 13s - 43ms/step - accuracy: 0.5122 - loss: 1.3650 - val_accuracy: 0.5201 - val_loss: 1.3516 Epoch 3/25 313/313 - 15s - 47ms/step - accuracy: 0.5539 - loss: 1.2566 - val_accuracy: 0.5148 - val_loss: 1.5107 Epoch 4/25 313/313 - 14s - 46ms/step - accuracy: 0.5813 - loss: 1.1874 - val_accuracy: 0.4337 - val_loss: 1.7202 Epoch 4: early stopping Restoring model weights from the end of the best epoch: 1. regularised_CNN_lr0.0005 validation macro-F1: 0.5102
7. Freeze development decisions¶
Select the CNN using validation macro-F1. Baselines remain informative references; if the linear model does better, report it. Do not assume a CNN is necessary. Do not widen the search after inspecting test scores while continuing to claim a fresh test estimate.
47. Compare validation quality, size and training cost¶
Show more than accuracy. Timing is a local measurement that includes first-fit overhead.
Python · cell 47
# Sort by the declared selection metric and tie-breaker.
validation_table = pd.DataFrame(validation_rows).sort_values(
["macro_f1", "parameters"], ascending=[False, True]).reset_index(drop=True)
validation_table.to_csv(OUTPUT_DIR / "validation_metrics.csv", index=False)
display(validation_table.round(4))
| model | accuracy | macro_f1 | weighted_f1 | log_loss | parameters | training_seconds | |
|---|---|---|---|---|---|---|---|
| 0 | basic_CNN | 0.6876 | 0.6857 | 0.6857 | 0.9025 | 122570 | 65.9073 |
| 1 | regularised_CNN_lr0.001 | 0.5234 | 0.5106 | 0.5106 | 1.3111 | 111722 | 86.3608 |
| 2 | regularised_CNN_lr0.0005 | 0.5225 | 0.5102 | 0.5102 | 1.3292 | 111722 | 57.4835 |
| 3 | linear_RGB_8x8 | 0.4154 | 0.4120 | 0.4120 | 1.7193 | 1930 | 10.4099 |
| 4 | majority | 0.1000 | 0.0182 | 0.0182 | 32.4393 | 0 | 0.0000 |
48. Select the CNN before test prediction¶
This selects the best CNN in this declared experiment, not the best possible model.
Python · cell 48
# The name remains fixed in the test evaluation.
cnn_ranking = validation_table[validation_table["model"].isin(cnn_models)]
champion_name = cnn_ranking.iloc[0]["model"]
champion_model = cnn_models[champion_name]
champion_validation = validation_probabilities[champion_name]
champion_history = cnn_histories[champion_name]
print("Validation-selected CNN:", champion_name)
print("Best overall validation reference:", validation_table.iloc[0]["model"])
Validation-selected CNN: basic_CNN Best overall validation reference: basic_CNN
49. Plot the selected model’s own history¶
Training-only augmentation and dropout can make training accuracy lower than evaluation accuracy.
Python · cell 49
# Accuracy and loss panels describe the same selected model.
figure, axes = plt.subplots(1, 2, figsize=(11, 4))
selected_epochs = np.arange(1, len(champion_history["loss"]) + 1)
for axis, metric in zip(axes, ["accuracy", "loss"]):
axis.plot(selected_epochs, champion_history[metric], color=NAVY, label="train")
axis.plot(selected_epochs, champion_history["val_" + metric], color=SCARLET, label="validation")
axis.set(title=f"Selected CNN: {metric}", xlabel="Epoch", ylabel=metric.title())
axis.legend()
finish_figure(figure, "07_selected_learning_curves.png")
50. Define a Wilson proportion interval¶
A lower bound is more cautious than a small accepted set’s accuracy point estimate.
Python · cell 50
# Standard 95% two-sided Wilson interval, conditional on the cases.
def wilson_interval(correct, total, z=1.96):
if total == 0:
return np.nan, np.nan
p = correct / total
denominator = 1 + z * z / total
centre = (p + z * z / (2 * total)) / denominator
radius = z * np.sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denominator
return float(centre - radius), float(centre + radius)
51. Define threshold coverage and accuracy¶
Coverage is the fraction automatically accepted; accepted accuracy excludes reviewed cases.
Python · cell 51
# A maximum softmax score is not guaranteed calibrated confidence.
def threshold_table(labels, probabilities, thresholds):
predicted, scores = probabilities.argmax(axis=1), probabilities.max(axis=1)
rows = []
for threshold in thresholds:
accepted = scores >= threshold
total = int(accepted.sum())
correct = int((predicted[accepted] == labels[accepted]).sum())
low, high = wilson_interval(correct, total)
rows.append({"threshold": float(threshold), "accepted": total,
"coverage": total / len(labels),
"accepted_accuracy": correct / total if total else np.nan,
"wilson_low": low, "wilson_high": high})
return pd.DataFrame(rows)
52. Select the review threshold using validation only¶
Require at least 100 accepted cases and a 90% accuracy lower bound; otherwise review everything.
Python · cell 52
# Threshold grid and target are declared before test scoring.
confidence_thresholds = np.arange(0.40, 1.00, 0.05)
validation_policy = threshold_table(y_validation, champion_validation, confidence_thresholds)
eligible = validation_policy[(validation_policy["accepted"] >= 100) &
(validation_policy["wilson_low"] >= TARGET_ACCEPTED_ACCURACY)]
review_threshold = None if eligible.empty else float(
eligible.sort_values("coverage", ascending=False).iloc[0]["threshold"])
validation_policy.to_csv(OUTPUT_DIR / "validation_review_policy.csv", index=False)
print("Frozen review threshold:", review_threshold, "(None means review all)")
display(validation_policy.round(4))
Frozen review threshold: 0.8499999999999999 (None means review all)
| threshold | accepted | coverage | accepted_accuracy | wilson_low | wilson_high | |
|---|---|---|---|---|---|---|
| 0 | 0.40 | 8991 | 0.8991 | 0.7296 | 0.7203 | 0.7387 |
| 1 | 0.45 | 8404 | 0.8404 | 0.7543 | 0.7450 | 0.7634 |
| 2 | 0.50 | 7789 | 0.7789 | 0.7793 | 0.7700 | 0.7884 |
| 3 | 0.55 | 7166 | 0.7166 | 0.8034 | 0.7940 | 0.8124 |
| 4 | 0.60 | 6602 | 0.6602 | 0.8244 | 0.8151 | 0.8334 |
| 5 | 0.65 | 5980 | 0.5980 | 0.8505 | 0.8412 | 0.8593 |
| 6 | 0.70 | 5447 | 0.5447 | 0.8715 | 0.8623 | 0.8801 |
| 7 | 0.75 | 4894 | 0.4894 | 0.8901 | 0.8810 | 0.8985 |
| 8 | 0.80 | 4350 | 0.4350 | 0.9087 | 0.8998 | 0.9169 |
| 9 | 0.85 | 3731 | 0.3731 | 0.9317 | 0.9231 | 0.9393 |
| 10 | 0.90 | 3045 | 0.3045 | 0.9488 | 0.9404 | 0.9560 |
| 11 | 0.95 | 2121 | 0.2121 | 0.9722 | 0.9643 | 0.9784 |
Limit: Wilson bounds are pointwise, not simultaneous guarantees across a threshold search. A separate policy-validation cohort is needed before deployment. A high score on an unknown image is still possible: this policy does not detect unknown classes.
53. Save the frozen decision¶
Record the choice before using test labels to score performance.
Python · cell 53
# This is an experiment decision, not production approval.
frozen_decision = {"cnn": champion_name, "selection_metric": "validation_macro_f1",
"review_threshold": review_threshold, "target_lower_bound": TARGET_ACCEPTED_ACCURACY,
"training_mode": RUN_MODE, "seed": SEED, "test_used_for_model_selection": False}
(OUTPUT_DIR / "frozen_decision.json").write_text(
json.dumps(frozen_decision, indent=2), encoding="utf-8")
display(frozen_decision)
{'cnn': 'basic_CNN',
'selection_metric': 'validation_macro_f1',
'review_threshold': 0.8499999999999999,
'target_lower_bound': 0.9,
'training_mode': 'full',
'seed': 42,
'test_used_for_model_selection': False}
8. Evaluate the frozen models on the official test set¶
Test labels are now used for model-performance evaluation. Every declared comparison is reported; the selected CNN remains the one chosen on validation. CIFAR-10 is public and repeatedly studied. A smaller authoring run was checked earlier. Treat this as a descriptive benchmark result, not a newly blind test or field validation.
54. Verify official test support¶
Check the final scoring labels against the official data contract.
Python · cell 54
# No fitting occurs in this section.
assert np.array_equal(np.unique(y_test), CLASS_IDS)
assert np.array_equal(np.bincount(y_test), np.full(10, 1_000))
test_dataset = make_dataset(x_test, y_test)
display(pd.DataFrame({"class": CLASS_NAMES, "test_support": np.bincount(y_test)}))
| class | test_support | |
|---|---|---|
| 0 | airplane | 1000 |
| 1 | automobile | 1000 |
| 2 | bird | 1000 |
| 3 | cat | 1000 |
| 4 | deer | 1000 |
| 5 | dog | 1000 |
| 6 | frog | 1000 |
| 7 | horse | 1000 |
| 8 | ship | 1000 |
| 9 | truck | 1000 |
55. Predict with every declared model¶
Store the probability matrices once and reuse them for all metrics and figures.
Python · cell 55
# The validation-selected name is not changed after this operation.
test_probabilities = {
"majority": majority_model.predict_proba(np.zeros((len(y_test), 1))),
"linear_RGB_8x8": linear_model.predict_proba(coarse_features(x_test)),
}
for name, model in cnn_models.items():
test_probabilities[name] = model.predict(test_dataset, verbose=0)
champion_probabilities = test_probabilities[champion_name]
champion_predictions = champion_probabilities.argmax(axis=1)
56. Create the measured test comparison¶
Report quality and development cost for the actual run budget.
Python · cell 56
# Highlight the existing selection rather than choosing a test winner.
cost_lookup = validation_table.set_index("model")[["parameters", "training_seconds"]]
test_table = pd.DataFrame([metric_row(name, y_test, probabilities)
for name, probabilities in test_probabilities.items()]).join(cost_lookup, on="model")
test_table["selected_on_validation"] = test_table["model"] == champion_name
test_table.to_csv(OUTPUT_DIR / "test_metrics.csv", index=False)
display(test_table.round(4))
| model | accuracy | macro_f1 | weighted_f1 | log_loss | parameters | training_seconds | selected_on_validation | |
|---|---|---|---|---|---|---|---|---|
| 0 | majority | 0.1000 | 0.0182 | 0.0182 | 32.4393 | 0 | 0.0000 | False |
| 1 | linear_RGB_8x8 | 0.4141 | 0.4105 | 0.4105 | 1.7069 | 1930 | 10.4099 | False |
| 2 | basic_CNN | 0.6922 | 0.6907 | 0.6907 | 0.9012 | 122570 | 65.9073 | True |
| 3 | regularised_CNN_lr0.001 | 0.5233 | 0.5085 | 0.5085 | 1.3227 | 111722 | 86.3608 | False |
| 4 | regularised_CNN_lr0.0005 | 0.5158 | 0.5035 | 0.5035 | 1.3348 | 111722 | 57.4835 | False |
57. Plot the fixed test comparison¶
Macro-F1 weights classes equally. Highlighting the selected CNN is descriptive.
Python · cell 57
# Exact values remain available in the table and CSV.
figure, axis = plt.subplots(figsize=(9, 4))
plot_table = test_table.sort_values("macro_f1")
colours = [SCARLET if name == champion_name else NAVY for name in plot_table["model"]]
axis.barh(plot_table["model"], plot_table["macro_f1"], color=colours)
axis.set(title="Official test macro-F1; selected CNN highlighted", xlabel="Macro-F1", xlim=(0, 1))
finish_figure(figure, "08_test_model_comparison.png")
58. Verify agreement with Keras evaluation¶
Catch mismatches between the model being evaluated and the predictions being reported.
Python · cell 58
# Keras total loss includes any L2 penalty; sklearn log loss does not.
keras_evaluation = champion_model.evaluate(test_dataset, verbose=0, return_dict=True)
computed_accuracy = accuracy_score(y_test, champion_predictions)
assert np.isclose(keras_evaluation["accuracy"], computed_accuracy, atol=1e-6)
print("Keras and saved-prediction accuracy agree:", round(computed_accuracy, 4))
print("Keras total loss:", round(keras_evaluation["loss"], 4))
Keras and saved-prediction accuracy agree: 0.6922 Keras total loss: 0.9012
59. Bootstrap test accuracy and the paired difference¶
Resample image-hash groups together to retain exact-repeat dependence and use the same cases in the comparison.
Python · cell 59
# This measures test sampling uncertainty, not retraining uncertainty.
_, test_group_ids = np.unique(test_hashes, return_inverse=True)
group_sizes = np.bincount(test_group_ids)
chosen_correct = (champion_predictions == y_test).astype(float)
basic_correct = (test_probabilities["basic_CNN"].argmax(axis=1) == y_test).astype(float)
group_correct = np.bincount(test_group_ids, weights=chosen_correct)
group_differences = np.bincount(test_group_ids, weights=chosen_correct - basic_correct)
bootstrap_rng = np.random.default_rng(SEED + 100)
bootstrap_accuracy, bootstrap_delta = [], []
for _ in range(BOOTSTRAP_REPEATS):
sampled = bootstrap_rng.integers(0, len(group_sizes), size=len(group_sizes))
denominator = group_sizes[sampled].sum()
bootstrap_accuracy.append(group_correct[sampled].sum() / denominator)
bootstrap_delta.append(group_differences[sampled].sum() / denominator)
accuracy_interval = np.quantile(bootstrap_accuracy, [0.025, 0.975])
delta_interval = np.quantile(bootstrap_delta, [0.025, 0.975])
60. Report uncertainty beside the estimate¶
Differences use percentage points. An interval containing zero does not establish a clear accuracy difference.
Python · cell 60
# Estimates are conditional on these fitted model objects.
accuracy_delta = float(np.mean(chosen_correct - basic_correct))
uncertainty_table = pd.DataFrame([
{"quantity": "selected_CNN_accuracy", "estimate": computed_accuracy,
"low_95": accuracy_interval[0], "high_95": accuracy_interval[1]},
{"quantity": "accuracy_difference_vs_basic_CNN", "estimate": accuracy_delta,
"low_95": delta_interval[0], "high_95": delta_interval[1]},
])
uncertainty_table.to_csv(OUTPUT_DIR / "accuracy_uncertainty.csv", index=False)
display(uncertainty_table.round(4))
print("Test image groups:", len(group_sizes), "Bootstrap repetitions:", BOOTSTRAP_REPEATS)
Test image groups: 10000 Bootstrap repetitions: 500
| quantity | estimate | low_95 | high_95 | |
|---|---|---|---|---|
| 0 | selected_CNN_accuracy | 0.6922 | 0.6834 | 0.7017 |
| 1 | accuracy_difference_vs_basic_CNN | 0.0000 | 0.0000 | 0.0000 |
Interpretation: this interval excludes uncertainty from retraining, model selection, domain shift or unknown source dependence. Hash groups address exact copies only. Your comment: What would repeated seeds, source-grouped validation and an independent new dataset each add to the evidence?
9. Explain class and image errors¶
Rows of a confusion matrix are true labels; columns are predicted labels. Recall asks how many real members of a class were found. Precision asks how many predictions of that class were correct. Read these values with their support counts.
61. Generate per-class test metrics¶
Use explicit class order and the same saved prediction vector.
Python · cell 61
# Include classes even if their predictions are poor.
report = classification_report(y_test, champion_predictions, labels=CLASS_IDS,
target_names=CLASS_NAMES, output_dict=True, zero_division=0)
per_class_report = pd.DataFrame(report).T.loc[CLASS_NAMES]
per_class_report.to_csv(OUTPUT_DIR / "per_class_test_metrics.csv", index_label="class")
display(per_class_report.round(4))
| precision | recall | f1-score | support | |
|---|---|---|---|---|
| airplane | 0.7828 | 0.692 | 0.7346 | 1000.0 |
| automobile | 0.8810 | 0.696 | 0.7777 | 1000.0 |
| bird | 0.6601 | 0.534 | 0.5904 | 1000.0 |
| cat | 0.5206 | 0.505 | 0.5127 | 1000.0 |
| deer | 0.6578 | 0.619 | 0.6378 | 1000.0 |
| dog | 0.5701 | 0.626 | 0.5968 | 1000.0 |
| frog | 0.6880 | 0.816 | 0.7466 | 1000.0 |
| horse | 0.7093 | 0.749 | 0.7286 | 1000.0 |
| ship | 0.7994 | 0.837 | 0.8178 | 1000.0 |
| truck | 0.6957 | 0.848 | 0.7643 | 1000.0 |
62. Plot confusion counts¶
These are actual case counts rather than proportions.
Python · cell 62
# Rows: actual; columns: predicted.
confusion_counts = confusion_matrix(y_test, champion_predictions, labels=CLASS_IDS)
figure, axis = plt.subplots(figsize=(9, 8))
ConfusionMatrixDisplay(confusion_counts, display_labels=CLASS_NAMES).plot(
ax=axis, cmap="Blues", colorbar=False, xticks_rotation=45, values_format="d")
axis.set_title("Selected CNN: test confusion counts")
finish_figure(figure, "09_confusion_counts.png")
63. Plot row-normalised confusion¶
Rows summing to one make class recall and error destinations easy to compare.
Python · cell 63
# Row normalisation does not produce a precision matrix.
confusion_rates = confusion_counts / confusion_counts.sum(axis=1, keepdims=True)
figure, axis = plt.subplots(figsize=(9, 8))
ConfusionMatrixDisplay(confusion_rates, display_labels=CLASS_NAMES).plot(
ax=axis, cmap="Blues", colorbar=False, xticks_rotation=45, values_format=".2f")
axis.set_title("Selected CNN: fraction of each true class")
finish_figure(figure, "10_confusion_rates.png")
64. List the largest error routes¶
Rank off-diagonal counts rather than guessing the hardest class pairs.
Python · cell 64
# Exclude the correct-prediction diagonal.
error_pairs = pd.DataFrame([
{"actual": CLASS_NAMES[actual], "predicted": CLASS_NAMES[predicted],
"count": int(confusion_counts[actual, predicted])}
for actual in CLASS_IDS for predicted in CLASS_IDS if actual != predicted
]).sort_values("count", ascending=False).head(8).reset_index(drop=True)
error_pairs.to_csv(OUTPUT_DIR / "largest_error_pairs.csv", index=False)
display(error_pairs)
| actual | predicted | count | |
|---|---|---|---|
| 0 | cat | dog | 181 |
| 1 | automobile | truck | 174 |
| 2 | dog | cat | 161 |
| 3 | bird | deer | 98 |
| 4 | bird | frog | 92 |
| 5 | deer | horse | 92 |
| 6 | airplane | ship | 91 |
| 7 | cat | frog | 89 |
65. Inspect confident mistakes¶
An average accuracy score can conceal predictions that are confidently wrong.
Python · cell 65
# Confidence is a score, not proof of correctness.
confidence = champion_probabilities.max(axis=1)
mistake_ids = np.flatnonzero(champion_predictions != y_test)
confident_mistakes = mistake_ids[np.argsort(confidence[mistake_ids])[::-1][:12]]
figure, axes = plt.subplots(3, 4, figsize=(10, 8))
for axis in axes.flat:
axis.axis("off")
for axis, index in zip(axes.flat, confident_mistakes):
axis.imshow(x_test[index], interpolation="nearest")
axis.set_title(f"True: {CLASS_NAMES[y_test[index]]}\n"
f"Pred: {CLASS_NAMES[champion_predictions[index]]} ({confidence[index]:.0%})", fontsize=9)
figure.suptitle("Most confident test mistakes", color=NAVY)
finish_figure(figure, "11_confident_mistakes.png")
Your comment: Describe what is visible first. Background reliance or label mistakes are hypotheses to investigate, not facts proved by one thumbnail.
66. Compute macro one-vs-rest AUC and top-three accuracy¶
AUC is a ranking measure [6]; top-three accuracy changes the decision task.
Python · cell 66
# Do not label a flattened micro curve as every class's performance.
macro_auc = roc_auc_score(y_test, champion_probabilities,
multi_class="ovr", average="macro", labels=CLASS_IDS)
top_three_accuracy = top_k_accuracy_score(y_test, champion_probabilities, k=3, labels=CLASS_IDS)
print(f"Macro one-vs-rest ROC-AUC: {macro_auc:.4f}")
print(f"Top-three accuracy: {top_three_accuracy:.4f}")
Macro one-vs-rest ROC-AUC: 0.9522 Top-three accuracy: 0.9192
67. Plot class-specific ROC curves¶
Each class is compared with the remaining classes. For rare events, also study precision-recall.
Python · cell 67
# Curves describe ranking, not the fraction correctly classified.
figure, axis = plt.subplots(figsize=(8, 6))
for class_id, name in enumerate(CLASS_NAMES):
binary_truth = (y_test == class_id).astype(int)
false_positive, true_positive, _ = roc_curve(binary_truth, champion_probabilities[:, class_id])
class_auc = roc_auc_score(binary_truth, champion_probabilities[:, class_id])
axis.plot(false_positive, true_positive, label=f"{name}: {class_auc:.2f}", linewidth=1.3)
axis.plot([0, 1], [0, 1], linestyle="--", color=SLATE)
axis.set(title="Selected CNN: one-vs-rest ROC", xlabel="False positive rate", ylabel="True positive rate")
axis.legend(fontsize=8, ncol=2)
finish_figure(figure, "12_per_class_roc.png")
10. Diagnose confidence and apply the frozen review policy¶
Softmax sums to one over known labels but can be overconfident, including on unfamiliar images. This section diagnoses confidence, not a calibrated-probability model or an unknown-class detector.
68. Compute confidence reliability bins¶
Compare average maximum score with observed correctness inside each bin.
Python · cell 68
# This is top-label reliability for multiclass predictions.
correct_predictions = champion_predictions == y_test
bin_ids = np.minimum((confidence * 10).astype(int), 9)
reliability_rows = []
for bin_id in range(10):
in_bin = bin_ids == bin_id
if in_bin.any():
reliability_rows.append({"bin": bin_id, "images": int(in_bin.sum()),
"mean_confidence": float(confidence[in_bin].mean()),
"observed_accuracy": float(correct_predictions[in_bin].mean())})
reliability_table = pd.DataFrame(reliability_rows)
calibration_error = np.sum(reliability_table["images"] / len(y_test) *
abs(reliability_table["mean_confidence"] - reliability_table["observed_accuracy"]))
reliability_table.to_csv(OUTPUT_DIR / "confidence_reliability.csv", index=False)
display(reliability_table.round(4))
print(f"10-bin top-label expected calibration error: {calibration_error:.4f}")
10-bin top-label expected calibration error: 0.0248
| bin | images | mean_confidence | observed_accuracy | |
|---|---|---|---|---|
| 0 | 1 | 6 | 0.1851 | 0.0000 |
| 1 | 2 | 220 | 0.2693 | 0.3182 |
| 2 | 3 | 767 | 0.3572 | 0.3507 |
| 3 | 4 | 1177 | 0.4514 | 0.4486 |
| 4 | 5 | 1291 | 0.5480 | 0.5174 |
| 5 | 6 | 1158 | 0.6489 | 0.6140 |
| 6 | 7 | 1112 | 0.7498 | 0.7320 |
| 7 | 8 | 1215 | 0.8532 | 0.8222 |
| 8 | 9 | 3054 | 0.9670 | 0.9375 |
69. Plot score-versus-accuracy reliability¶
Points below the diagonal indicate overconfidence in that bin. Small bins remain uncertain.
Python · cell 69
# ECE depends on the binning choice; its definition is recorded.
figure, axis = plt.subplots(figsize=(6, 5))
axis.plot([0, 1], [0, 1], linestyle="--", color=SLATE, label="perfect agreement")
axis.scatter(reliability_table["mean_confidence"], reliability_table["observed_accuracy"],
s=30 + 170 * reliability_table["images"] / reliability_table["images"].max(), color=SCARLET)
axis.set(title="Selected CNN: confidence reliability", xlabel="Mean maximum softmax score",
ylabel="Observed accuracy", xlim=(0, 1), ylim=(0, 1))
axis.legend()
finish_figure(figure, "13_reliability.png")
Your comment: Which bins look overconfident? Why is calibration a stronger claim than good accuracy?
70. Measure the frozen review policy on test cases¶
Apply the validation-selected threshold once. A failed validation rule remains review-all.
Python · cell 70
# There is no test-time threshold search.
if review_threshold is None:
policy_result = {"threshold": None, "coverage": 0.0,
"review_fraction": 1.0, "accepted_accuracy": None, "accepted": 0}
else:
policy_result = threshold_table(y_test, champion_probabilities,
[review_threshold]).iloc[0].to_dict()
policy_result["review_fraction"] = 1.0 - policy_result["coverage"]
(OUTPUT_DIR / "test_review_policy.json").write_text(json.dumps(policy_result, indent=2), encoding="utf-8")
display(pd.Series(policy_result))
threshold 0.850000 accepted 3714.000000 coverage 0.371400 accepted_accuracy 0.920032 wilson_low 0.910868 wilson_high 0.928328 review_fraction 0.628600 dtype: float64
71. Plot validation coverage versus accepted accuracy¶
This describes the development trade-off, not a new policy selected from test results.
Python · cell 71
# Omit undefined accuracy where no examples were accepted.
figure, axis = plt.subplots(figsize=(7, 4))
valid_policy_rows = validation_policy.dropna(subset=["accepted_accuracy"])
axis.plot(valid_policy_rows["coverage"], valid_policy_rows["accepted_accuracy"],
marker="o", color=NAVY, label="validation accepted accuracy")
axis.axhline(TARGET_ACCEPTED_ACCURACY, color=SCARLET, linestyle="--", label="target level")
axis.set(title="Validation: coverage and accuracy", xlabel="Coverage",
ylabel="Accuracy among accepted images", xlim=(0, 1), ylim=(0, 1))
axis.legend(fontsize=8)
finish_figure(figure, "14_validation_coverage.png")
72. Generate the interpretation from measured results¶
This narrative changes when predictions change. Keep the scope and limitations beside the score.
Python · cell 72
# Each number is computed from this run's predictions.
selected_row = test_table.set_index("model").loc[champion_name]
worst_class = per_class_report["recall"].idxmin()
largest_pair = error_pairs.iloc[0]
delta_evidence = "excludes zero" if delta_interval[0] > 0 or delta_interval[1] < 0 else "includes zero"
summary_text = f"""### Measured interpretation
**{RUN_MODE} mode:** {len(x_train):,} train, {len(x_validation):,} validation,
{len(x_test):,} official test images, at most {EPOCHS} epochs per CNN.
Validation selected **{champion_name}**. Test accuracy is **{computed_accuracy:.1%}**
(bootstrap interval **{accuracy_interval[0]:.1%}–{accuracy_interval[1]:.1%}**),
with macro-F1 **{selected_row['macro_f1']:.3f}**.
The accuracy difference from the basic CNN is **{accuracy_delta*100:+.2f} percentage points**;
its interval {delta_evidence}. The weakest recall is **{worst_class}**
({per_class_report.loc[worst_class, 'recall']:.1%}). The largest error route is
**{largest_pair['actual']} → {largest_pair['predicted']}** ({largest_pair['count']} images).
Macro AUC **{macro_auc:.3f}** describes ranking; top-three accuracy
**{top_three_accuracy:.1%}** describes a different task.
This demonstrates an evaluated benchmark workflow, not field reliability or a
trained astronomy, defect or satellite classifier.
"""
display(Markdown(summary_text))
(OUTPUT_DIR / "measured_interpretation.md").write_text(summary_text, encoding="utf-8")
Out[72]: 679
Measured interpretation¶
full mode: 40,000 train, 10,000 validation, 10,000 official test images, at most 25 epochs per CNN. Validation selected basic_CNN. Test accuracy is 69.2% (bootstrap interval 68.3%–70.2%), with macro-F1 0.691. The accuracy difference from the basic CNN is +0.00 percentage points; its interval includes zero. The weakest recall is cat (50.5%). The largest error route is cat → dog (181 images). Macro AUC 0.952 describes ranking; top-three accuracy 91.9% describes a different task. This demonstrates an evaluated benchmark workflow, not field reliability or a trained astronomy, defect or satellite classifier.
11. Export the model and evidence¶
Keep the class mapping, preprocessing, policy and model identity with its weights. The export is a reusable benchmark model, not a deployed service. Save/reload checks confirm the artifact matches the predictions reported above.
73. Save the selected model¶
Native Keras format retains its architecture and weights. Raw pixels are rescaled internally.
Python · cell 73
# Do not divide by 255 again when using this artifact.
model_path = OUTPUT_DIR / "selected_cifar10_cnn.keras"
champion_model.save(model_path)
model_sha256 = hashlib.sha256(model_path.read_bytes()).hexdigest()
print("Saved model:", model_path.name)
print("Model SHA-256:", model_sha256)
Saved model: selected_cifar10_cnn.keras Model SHA-256: bb7bd1d88c0599b03f26b83c667ea9ddefc947e88cbd13137da5bb8072064e1e
74. Reload and verify inference¶
A successful save should also preserve its actual predicted probabilities.
Python · cell 74
# Compare one raw-RGB inference batch with saved test probabilities.
reloaded_model = keras.models.load_model(model_path, compile=False)
roundtrip_probabilities = reloaded_model.predict(x_test[:32], verbose=0)
assert np.allclose(roundtrip_probabilities, champion_probabilities[:32], atol=1e-5)
print("Saved/reloaded probabilities agree within tolerance.")
Saved/reloaded probabilities agree within tolerance.
75. Save row-level predictions¶
These records make metrics recalculable without fitting the model again.
Python · cell 75
# Probability columns follow the official class mapping.
prediction_record = pd.DataFrame({"test_row": np.arange(len(y_test)),
"image_sha256": test_hashes, "actual_id": y_test,
"predicted_id": champion_predictions, "maximum_score": confidence})
for class_id, class_name in enumerate(CLASS_NAMES):
prediction_record["prob_" + class_name] = champion_probabilities[:, class_id]
prediction_record.to_csv(OUTPUT_DIR / "selected_test_predictions.csv", index=False)
print("Saved", len(prediction_record), "test predictions.")
Saved 10000 test predictions.
76. Save the reproducibility manifest¶
Record what was measured separately from proposed future work.
Python · cell 76
# Array hashes identify the actual image content for this run.
manifest = {"author": "Gareth Grant", "dataset": "CIFAR-10",
"dataset_reference": "https://www.cs.toronto.edu/~kriz/cifar.html",
"mode": RUN_MODE, "seed": SEED, "maximum_epochs": EPOCHS,
"batch_size": BATCH_SIZE, "train_images": len(x_train),
"validation_images": len(x_validation), "test_images": len(x_test),
"development_array_sha256": hashlib.sha256(x_development.tobytes()).hexdigest(),
"test_array_sha256": hashlib.sha256(x_test.tobytes()).hexdigest(),
"class_names": CLASS_NAMES, "input_shape": [32, 32, 3],
"input_scale": "raw RGB 0-255; model rescales internally",
"selection": frozen_decision, "model_sha256": model_sha256,
"software": software_versions, "bootstrap_repeats": BOOTSTRAP_REPEATS,
"test_review_policy": policy_result, "application_models_trained": False}
(OUTPUT_DIR / "run_manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
print("Saved model, manifest, plots, metrics and predictions.")
Saved model, manifest, plots, metrics and predictions.
77. Define inference for one new image¶
Convert to RGB, respect orientation metadata and resize to the benchmark input.
Python · cell 77
# File handling is not a domain-shift or unknown-class detector.
def predict_image(image_path):
with Image.open(image_path) as source:
image = ImageOps.exif_transpose(source).convert("RGB")
image = image.resize((32, 32), resample=Image.Resampling.BILINEAR)
pixels = np.asarray(image, dtype=np.uint8)
probabilities = reloaded_model.predict(pixels[None, ...], verbose=0)[0]
predicted_id = int(probabilities.argmax())
score = float(probabilities[predicted_id])
review = review_threshold is None or score < review_threshold
return {"label": CLASS_NAMES[predicted_id], "maximum_score": score,
"review_required": review}, probabilities
Example after running the notebook: decision, scores = predict_image("my_image.jpg").
Every input is forced into a known class; an unfamiliar galaxy could receive a confident
animal label. Resizing removes detail. Validate the actual intended imagery before use.
12. Business: connect classification to a decision and its cost¶
These are proposed adaptations of the workflow, not capabilities proved by CIFAR-10.
| Use | Decision supported | New data and design required | Pilot measures |
|---|---|---|---|
| Catalogue photos | Route a photo to a product category for review | Actual labelled catalogue images; product-ID split; unknown categories | Macro-F1, corrections per 1,000, review time, coverage |
| Manufacturing inspection | Send suspected defects to an inspector | Label defects and normal items; separate batches; detection if location matters | Defect recall, false alarms, missed-defect cost, workload |
| Asset-photo triage | Prioritise claims or maintenance photos | Labels tied to operational action; separate assets/sites/cameras; suitable resolution | Error cost, throughput, time to resolution |
A business needs to see the action following the prediction, the cost of each error, the remaining manual workload and improvement over its existing process. A benchmark score alone does not provide that evidence. Use a real operational pilot, split by product/batch/site rather than repeated photos of the same object, and compare simple and pretrained approaches with the current human process.
Auditing: keep input identity, model hash, mapping, score, threshold, review action and reviewer correction. That makes an action traceable; it does not prove the label is true or that a process meets a particular legal standard.
78. Declare an illustrative business scenario¶
These rand assumptions are fictional. They are not prices, measured savings or estimates from CIFAR-10.
Python · cell 78
# Replace these inputs with operational pilot measurements.
MONTHLY_IMAGES, MANUAL_SECONDS_PER_IMAGE = 50_000, 30
HOURLY_REVIEW_COST_R, MODEL_COST_PER_1_000_R = 180, 2
FIXED_MONTHLY_COST_R, COST_PER_AUTO_ERROR_R = 2_000, 20
business_scenarios = pd.DataFrame([
{"scenario": "downside", "automation_coverage": 0.30, "accepted_accuracy": 0.85},
{"scenario": "central", "automation_coverage": 0.60, "accepted_accuracy": 0.95},
{"scenario": "optimistic", "automation_coverage": 0.80, "accepted_accuracy": 0.98},
])
display(business_scenarios)
| scenario | automation_coverage | accepted_accuracy | |
|---|---|---|---|
| 0 | downside | 0.3 | 0.85 |
| 1 | central | 0.6 | 0.95 |
| 2 | optimistic | 0.8 | 0.98 |
79. Calculate conditional monthly net value¶
Count manual work that remains, automatic errors and operating costs. Automation can also lose money.
Python · cell 79
# Each exception still gets the full assumed manual review time.
manual_cost_r = MONTHLY_IMAGES * MANUAL_SECONDS_PER_IMAGE / 3_600 * HOURLY_REVIEW_COST_R
operating_cost_r = FIXED_MONTHLY_COST_R + MONTHLY_IMAGES / 1_000 * MODEL_COST_PER_1_000_R
business_scenarios["manual_review_cost_R"] = manual_cost_r * (1 - business_scenarios["automation_coverage"])
business_scenarios["expected_auto_errors"] = MONTHLY_IMAGES * business_scenarios["automation_coverage"] * (1 - business_scenarios["accepted_accuracy"])
business_scenarios["error_cost_R"] = business_scenarios["expected_auto_errors"] * COST_PER_AUTO_ERROR_R
business_scenarios["operating_cost_R"] = operating_cost_r
business_scenarios["net_monthly_value_R"] = manual_cost_r - (
business_scenarios["manual_review_cost_R"] + business_scenarios["error_cost_R"] + operating_cost_r)
business_scenarios.to_csv(OUTPUT_DIR / "illustrative_business_scenarios.csv", index=False)
print(f"Assumed all-manual monthly cost: R{manual_cost_r:,.0f}")
display(business_scenarios.round(2))
Assumed all-manual monthly cost: R75,000
| scenario | automation_coverage | accepted_accuracy | manual_review_cost_R | expected_auto_errors | error_cost_R | operating_cost_R | net_monthly_value_R | |
|---|---|---|---|---|---|---|---|---|
| 0 | downside | 0.3 | 0.85 | 52500.0 | 2250.0 | 45000.0 | 2100.0 | -24600.0 |
| 1 | central | 0.6 | 0.95 | 30000.0 | 1500.0 | 30000.0 | 2100.0 | 12900.0 |
| 2 | optimistic | 0.8 | 0.98 | 15000.0 | 800.0 | 16000.0 | 2100.0 | 41900.0 |
Interpretation: conditional arithmetic is not an ROI forecast. The example excludes implementation, labelling, integration, supervision, training, downtime, tax and revenue changes. Use a class-specific cost matrix when errors have different consequences. Your comment: Which measured assumption would change the conclusion most?
80. Plot the hypothetical business outcomes¶
Show negative as well as positive outcomes to make the cost assumptions visible.
Python · cell 80
# The title explicitly distinguishes assumptions from measured savings.
figure, axis = plt.subplots(figsize=(7, 4))
values = business_scenarios["net_monthly_value_R"]
axis.bar(business_scenarios["scenario"], values,
color=[SCARLET if value < 0 else NAVY for value in values])
axis.axhline(0, color=SLATE, linewidth=1)
axis.set(title="Illustrative monthly value — assumptions, not measured savings", ylabel="Rand")
finish_figure(figure, "15_illustrative_business_value.png")
13. Astronomy: survey classification and expert review¶
Related workflows can organise galaxy morphology or rank candidates for inspection. The Euclid Collaboration studied a Galaxy Zoo/Zoobot CNN adapted for emulated Euclid images [7]. Zoobot research also studies reusable galaxy representations for similarity, anomalies and new labels [8]. Those are separate research results, not this notebook’s result.
| Proposed use | Value to an astronomer | Changes required | Meaningful evaluation |
|---|---|---|---|
| Galaxy morphology | Build a catalogue of smooth, disk or featured objects | Galaxy images; expert/citizen labels; domain-trained representations | Per-class recall, label-distribution agreement, brightness/redshift strata |
| Candidate triage | Prioritise scientifically useful follow-up | Labels matching the research question and rare-event sampling | Purity, completeness, precision-recall, confirmed discoveries per review hour |
| Quality screening | Identify artefacts or poor observations | Instrument-specific labels and appropriate image scale | Missed artefacts, rejected real sources, cross-instrument validation |
Preserve FITS calibration metadata, photometric bands, intensity scales and sufficient resolution. Arbitrary RGB conversion or a 32 × 32 resize may destroy useful structure. Choose augmentations with an astronomer. Keep repeated observations of one object in one subset and test new sky regions, times, seeing conditions and instruments. Report performance by magnitude or signal-to-noise; bright familiar sources do not establish success on faint objects. A CIFAR-10 animal/object classifier cannot perform these scientific tasks without new data and training.
Your reflection: Which scientific signal might preprocessing destroy, and what would count as independent validation for your proposed astronomy project?
14. Government: inspection and environmental information¶
ESA describes satellite land-cover classification supporting agriculture, planning, environmental protection and crisis management [9]. NASA describes geospatial foundation models for applications including flood mapping, burn scars and crop classification [10]. These examples use different data and models from this RGB benchmark.
| Proposed use | Decision supported | Data and design required | Measures |
|---|---|---|---|
| Land-cover monitoring | Prioritise planning or environmental review | Multispectral imagery, georeferenced labels, separate regions/seasons; usually segmentation | Class recall, map accuracy, area-estimate uncertainty, regional results |
| Disaster assessment | Send possible flood/fire impacts to analysts | Event-specific data; validation on new events/sensors/cloud conditions | Missed areas, false alerts, latency, workload |
| Maintenance routing | Direct infrastructure photos to the right team | Local labelled assets; site/asset/time separation; detection if location matters | Routing accuracy, missed urgent cases, review time, resolution time |
This notebook does not create pixel maps, estimate flooded area, forecast disasters or establish whether an asset is unsafe. A public-sector pilot needs task-specific data, field validation and comparison with the existing process. Record versions, actions and corrections; examine regions and conditions separately. Keep accountable review for consequential actions. Where photographs include personal information, minimise collection and control access in the actual service design.
Your reflection: What action would a prediction trigger, who checks it, and which missed cases matter most?
15. What this portfolio can honestly demonstrate¶
The core achievement is a complete, readable evaluation workflow: contracts and leakage checks, useful baselines, explicit CNN experiments, validation-only decisions, error and confidence analysis, conditional uncertainty, reproducible exports and decision-focused applications. Quote scores with the run mode, training count, test count and metric definition.
Do not describe this as an Intel dataset analysis, transfer learning, a production classifier or a trained astronomy/government system.
Useful next experiments:
- Repeat seeds and run a larger search while recording compute cost.
- Use ablations to isolate augmentation, regularisation and pooling effects.
- Compare suitable pretrained features with their required preprocessing.
- Build one real domain pilot with source/location/time grouping and independent validation.
- Calibrate on a separate cohort and evaluate unfamiliar inputs.
These are future experiments, not implemented features. Add your final reflection: what did the simple baseline teach you, which errors matter most, and what evidence is missing?
References and attribution¶
Primary sources checked during authoring on 4 October 2026. Application examples are paraphrased context, not results for this experiment. Keep attribution when publishing and check dataset/source terms before redistributing third-party raw assets.
- Krizhevsky, Nair and Hinton. CIFAR-10 dataset page. Dataset report: Krizhevsky (2009), Learning Multiple Layers of Features from Tiny Images.
- TensorFlow. Operation determinism.
- TensorFlow. CNN tutorial.
- Keras. EarlyStopping.
- Keras. RandomFlip and RandomTranslation.
- Scikit-learn. roc_auc_score.
- Euclid Collaboration (2024). Measuring detailed galaxy morphologies for Euclid with machine learning.
- Walmsley et al. (2021). Practical Galaxy Morphology Tools from Deep Supervised Representation Learning.
- ESA (2020). Land-cover maps of Europe from the Cloud.
- NASA (2024). Foundation Models for Generalist Geospatial AI.
- Keras. BatchNormalization.