Skip to the analysis

GARETH GRANTTECHNOLOGY · DATA · INNOVATION Back to projects

Back to all projects

Portfolio case study · Fraud analytics · Python

Credit-card fraud.From risk score to defensible decision.

A reproducible fraud-risk workflow focused on chronological evaluation, operational thresholds, uncertainty and responsible model governance.

Gareth Grant · 5 October 2026 · Executed code, charts and interpretation

1.30Mtransactions

24original fields

87executed code cells

3locked data partitions

Evidence note: Extreme class imbalance makes raw accuracy misleading. Model value is judged through precision, recall, cost and leakage-safe chronological evaluation.

Read the interpretation. Open the code when useful. Show all code Hide all code Download notebook ↓

Table of contents

Table of contents

‹Hide

GARETH GRANT · ANALYTICS

OverviewCode & mathsResultsApplicationsModel card

Credit-card fraud: from a risk score to a defensible decision

Gareth Grant · Data analytics portfolio · Teaching edition · 5 October 2026

An interpretable fraud-screening workflow, with logistic regression implemented from first principles and compared with scikit-learn on the same data. Each numbered code cell has one purpose and a short explanation. You can add comments directly to that cell or place your own markdown underneath it.

The decision: which transactions should an analyst review, given the cost of missed fraud, the inconvenience of false alerts, and a limited review capacity?

What this project demonstrates: data validation, temporal evaluation, sparse preprocessing, numerical optimisation, rare-event metrics, threshold selection, uncertainty estimation, and communication for decision-makers.

Evidence boundary: the delivered outputs are a fresh run of the public CSV linked in the original notebook. They are educational benchmark results. Dataset provenance does not establish real-bank performance, causal effects, or savings in production. The cost assumptions are explicitly illustrative. Astronomy is discussed as a separate adaptation of the method, with no astronomy performance claim.

Reading routes

Reader	Suggested route
Recruiter or manager	Purpose → final results → business applications → model card
Student	Read every numbered explanation and execute the next cell
Technical reviewer	Data checks → chronological split → objective and gradient checks → frozen policies → test

Contents

1. Purpose and improvements
2. Reproducible setup
3. Data loading and quality
4. Chronological partitions
5. Training-data exploration
6. Features and preprocessing
7. Logistic regression from first principles
8. Model selection
9. Review policies
10. Final held-out results
11. Uncertainty and interpretation
12. Business, government, auditing, and astronomy
13. Model card and exports
14. Exercises and references

1. Purpose and improvements

Fraud is rare. A model can classify almost every transaction correctly while failing to identify a single fraudulent transaction. We therefore assess ranking, the transactions sent for review, and the fraud missed, alongside accuracy.

The original notebook reported 1,296,675 transactions, including 7,506 labelled fraud. Those counts implied approximately 99.42% accuracy for predicting legitimate on every row. This is a dataset-level arithmetic observation, not a held-out model result.

Issue in the original	Correction in this edition
Encoding, imputation, and SMOTE fitted before splitting	Fit learned transformations on training rows only; use no SMOTE
Evaluation on a 50:50 resampled population	Preserve observed prevalence in all future evaluation partitions
Male / Female mapped against stored M / F values	Exclude gender from predictors; avoid the faulty mapping
Text cleaned in df while encoding df_dropped	Treat category as a category; remove unused NLP processing
Unscaled scratch-model predictors	Reuse one fitted scaler and encoder for both implementations
Variables replaced with 100 make_moons rows	Keep transaction variables distinct; use no toy data for fraud results
Tuned model evaluated on its own training data	Select with future validation rows; reserve a final future test
ROC AUC calculated from hard predictions	Calculate ROC AUC and AP from continuous scores
Cross-validation after global scaling	Use explicit temporal partitions and train-only preprocessing
Loss sign error and repeated sigmoid definitions	Use a stable logistic loss; verify its gradient numerically
Prose disagrees with stored outputs	Generate result commentary from the evaluated metrics
Different datasets used in a model comparison	Compare the same training sample, features, policy data, and test rows

This is a deliberate redesign. It does not claim that historical and new scores are directly comparable, because the evaluation population and protocol have changed.

2. Reproducible setup

Use Python 3.11 or newer. Install the bundled requirements.txt, run python download_data.py, and open this notebook from the project folder. Alternatively, download the CSV manually from the Kaggle source and place it at data/credit_card_transactions.csv.

Run Restart Kernel and Run All. Changing predictors, costs, or a policy after examining final test results requires a new future holdout for a confirmatory claim. The default sample cap controls training time; validation and test data remain full.

Cell 01 · Import data tools

These libraries handle tables, arrays, paths, and reproducibility.

# Import data tools
from pathlib import Path
import hashlib
import importlib.metadata
import json
import platform
import time
import numpy as np
import pandas as pd

Cell 02 · Import visual tools

Markdown outputs keep the written interpretation tied to the numbers.

# Import visual tools
import matplotlib.pyplot as plt
from IPython.display import display, Markdown

Cell 03 · Import preprocessing tools

A column transformer applies different rules to numbers and categories.

# Import preprocessing tools
from scipy import sparse
from scipy.special import expit
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

Cell 04 · Import evaluation tools

Alias the library classifier so it cannot overwrite our scratch functions.

# Import evaluation tools
from sklearn.linear_model import LogisticRegression as SklearnLogisticRegression
from sklearn.metrics import (
    accuracy_score, average_precision_score, balanced_accuracy_score,
    brier_score_loss, confusion_matrix, f1_score, log_loss,
    precision_recall_curve, precision_score, recall_score, roc_auc_score, roc_curve,
)
from sklearn.calibration import calibration_curve
from threadpoolctl import threadpool_limits

Cell 05 · Choose file locations

Change the CSV path here. A missing file produces a clear error; no different dataset is substituted.

# Choose file locations
CSV_PATH = Path('data/credit_card_transactions.csv')
EXPORT_DIR = Path('outputs')
WRITE_EXPORTS = True

Cell 06 · Choose reproducible run settings

Uniform sampling applies only to training. The rare class is not artificially enlarged.

# Choose reproducible run settings
RANDOM_SEED = 42
MAX_TRAIN_ROWS = 150_000  # Set to None to train on the entire early period.
MAX_ITERATIONS = 1_500
GRADIENT_TOLERANCE = 1e-5
L2_CANDIDATES = (0.0001, 0.001, 0.01)
BOOTSTRAP_REPEATS = 200
threadpool_limits(limits=2)

Out[6]: <threadpoolctl.threadpool_limits at 0x7f21c2be4650>

Cell 07 · Declare illustrative review assumptions

MU means monetary units. These inputs are scenarios, not observed bank charges, exchange rates, or a savings forecast.

Learning question: Which of these assumptions would you need to obtain from a fraud operations team?

# Declare illustrative review assumptions
REVIEW_COST_MU = 2.0       # Every alert incurs review work.
FALSE_ALERT_FRICTION_MU = 10.0  # Extra cost for alerting a legitimate transaction.
MISSED_FRAUD_COST_MU = 500.0   # Constant loss per missed fraud: a simplifying assumption.
MAX_POLICY_ALERT_RATE = 0.02   # At most 2% of policy-validation transactions.
assert min(REVIEW_COST_MU, FALSE_ALERT_FRICTION_MU, MISSED_FRAUD_COST_MU) >= 0
assert 0 < MAX_POLICY_ALERT_RATE <= 1

Cell 08 · Set a consistent chart style

Readable labels, restrained colours, and closed figures make the notebook suitable for a website.

# Set a consistent chart style
NAVY, SCARLET, SILVER = '#16324F', '#FF2400', '#B8C2CC'
plt.rcParams.update({
    'figure.figsize': (9, 4.6), 'figure.dpi': 120,
    'axes.spines.top': False, 'axes.spines.right': False,
    'axes.titleweight': 'bold', 'axes.labelcolor': '#20252B',
    'axes.prop_cycle': plt.cycler(color=[NAVY, SCARLET, '#667582']),
})

Cell 09 · Record the environment

The exact dependency versions used in this run are recorded for reproducibility.

# Record the environment
environment = {'python': platform.python_version()}
for package in ['numpy', 'pandas', 'scipy', 'scikit-learn', 'matplotlib']:
    environment[package] = importlib.metadata.version(package)
display(pd.Series(environment, name='Version').to_frame())

	Version
python	3.12.14
numpy	2.3.5
pandas	2.2.3
scipy	1.17.0
scikit-learn	1.8.0
matplotlib	3.10.8

3. Data loading and quality

The source listing and the CSV are separate evidence. Trust programmatically verified counts, dates, and schema over a descriptive row-count claim. Record a file hash to identify exactly which bytes were analysed.

Names, full addresses, gender, jobs, and dates of birth are not loaded. Account and transaction identifiers are retained only for integrity and dependence checks; they are never predictors or printed in previews. The source does not substantiate a real-bank origin. Its schema and sample resemble the Sparkov simulation family; that resemblance is a provenance clue, not a verified chain of custody. Treat this as an educational benchmark and check source lineage before making operational claims.

Cell 10 · Check the file and header

The header is read first so missing required fields fail before an expensive load.

# Check the file and header
if not CSV_PATH.is_file():
    raise FileNotFoundError(f'Run download_data.py or place the source CSV at {CSV_PATH.resolve()}')
source_columns = pd.read_csv(CSV_PATH, nrows=0).columns.tolist()
required_columns = {
    'trans_date_trans_time', 'amt', 'category', 'lat', 'long',
    'merch_lat', 'merch_long', 'is_fraud',
}
optional_columns = {'cc_num', 'trans_num'}
missing_columns = required_columns - set(source_columns)
if missing_columns:
    raise ValueError(f'Missing required columns: {sorted(missing_columns)}')
print(f'CSV fields: {len(source_columns)}; size: {CSV_PATH.stat().st_size / 1e6:.1f} MB')

CSV fields: 24; size: 354.1 MB

Cell 11 · Load only the necessary columns

Reading identifiers as strings prevents accidental numeric rounding.

# Load only the necessary columns
raw = pd.read_csv(
    CSV_PATH, usecols=lambda name: name in required_columns | optional_columns,
    dtype={'cc_num': 'string', 'trans_num': 'string', 'category': 'string'},
)
print(f'Loaded {len(raw):,} rows and {raw.shape[1]} working columns.')

Loaded 1,296,675 rows and 10 working columns.

Cell 12 · Fingerprint the source file

A streamed hash avoids loading the whole CSV into memory a second time.

# Fingerprint the source file
def file_sha256(path):
    digest = hashlib.sha256()
    with Path(path).open('rb') as stream:
        for block in iter(lambda: stream.read(1024 * 1024), b''):
            digest.update(block)
    return digest.hexdigest()

source_sha256 = file_sha256(CSV_PATH)
print('Source SHA-256:', source_sha256)

Source SHA-256: 1a100d7d1e5e77442a5ce632293657c099d70ea0991797e92b55eb987da59f9f

Cell 13 · Inspect safe fields

The preview shows useful modelling fields without account numbers or names.

# Inspect safe fields
preview_columns = ['trans_date_trans_time', 'category', 'amt', 'is_fraud']
display(raw[preview_columns].head())

	trans_date_trans_time	category	amt	is_fraud
0	2019-01-01 00:00:18	misc_net	4.97	0
1	2019-01-01 00:00:44	grocery_pos	107.23	0
2	2019-01-01 00:00:51	entertainment	220.11	0
3	2019-01-01 00:01:16	gas_transport	45.00	0
4	2019-01-01 00:03:06	misc_pos	41.96	0

Cell 14 · Count missing inputs

Missingness is measured before cleaning. Numeric imputation will be learned later from training rows.

# Count missing inputs
missing_report = raw.isna().sum().rename('missing_rows').to_frame()
missing_report['missing_percent'] = 100 * missing_report['missing_rows'] / len(raw)
display(missing_report.sort_values('missing_rows', ascending=False))

	missing_rows	missing_percent
trans_date_trans_time	0	0.0
cc_num	0	0.0
category	0	0.0
amt	0	0.0
lat	0	0.0
long	0	0.0
trans_num	0	0.0
merch_lat	0	0.0
merch_long	0	0.0
is_fraud	0	0.0

Cell 15 · Validate binary labels

Unknown or absent outcomes cannot silently become legitimate transactions.

# Validate binary labels
labels = pd.to_numeric(raw['is_fraud'], errors='coerce')
if labels.isna().any() or not set(labels.unique()).issubset({0, 1}):
    raise ValueError('is_fraud must contain only known labels 0 and 1.')
raw['is_fraud'] = labels.astype('int8')
print(raw['is_fraud'].value_counts().rename(index={0: 'Legitimate', 1: 'Fraud'}))

is_fraud
Legitimate    1289169
Fraud            7506
Name: count, dtype: int64

Cell 16 · Parse transaction timestamps

The recorded transaction timestamp controls the split. We do not assume a timezone or reinterpret the redundant unix_time field.

# Parse transaction timestamps
raw['timestamp'] = pd.to_datetime(raw['trans_date_trans_time'], errors='coerce')
invalid_time_rows = int(raw['timestamp'].isna().sum())
print(f'Unparseable timestamps: {invalid_time_rows:,}')

Unparseable timestamps: 0

Cell 17 · Coerce numeric fields

Invalid coordinates become missing values. Negative or non-finite amounts are excluded in the next cell.

# Coerce numeric fields
numeric_source_columns = ['amt', 'lat', 'long', 'merch_lat', 'merch_long']
for column in numeric_source_columns:
    raw[column] = pd.to_numeric(raw[column], errors='coerce').replace([np.inf, -np.inf], np.nan)
for column, bound in [('lat', 90), ('merch_lat', 90), ('long', 180), ('merch_long', 180)]:
    raw.loc[raw[column].abs() > bound, column] = np.nan

Cell 18 · Exclude unusable transaction records

We require a timestamp for temporal evaluation and a valid non-negative amount for this analysis. Exclusions are counted.

# Exclude unusable transaction records
usable = raw['timestamp'].notna() & raw['amt'].notna() & raw['amt'].ge(0)
excluded_rows = int((~usable).sum())
data = raw.loc[usable].copy()
print(f'Excluded unusable records: {excluded_rows:,}')

Excluded unusable records: 0

Cell 19 · Remove exact duplicate records

The irrelevant CSV index was not loaded. Identical transaction records are removed before partitioning.

# Remove exact duplicate records
exact_duplicates = int(data.duplicated().sum())
data = data.drop_duplicates().copy()
print(f'Exact duplicates removed: {exact_duplicates:,}')

Exact duplicates removed: 0

Cell 20 · Check transaction-identifier integrity

Remaining repeated identifiers need investigation instead of silently keeping an arbitrary row.

# Check transaction-identifier integrity
if 'trans_num' in data:
    if data['trans_num'].isna().any() or data['trans_num'].duplicated().any():
        raise ValueError('Missing or repeated transaction IDs remain; investigate conflicting records.')
print('Transaction-ID integrity check passed where IDs are available.')

Transaction-ID integrity check passed where IDs are available.

Cell 21 · Order the cleaned data

A stable sort keeps the run reproducible. Equal timestamps will be assigned to the same temporal partition.

# Order the cleaned data
data = data.sort_values('timestamp', kind='stable').reset_index(drop=True)
data_quality = {
    'raw_rows': len(raw), 'usable_unique_rows': len(data),
    'excluded_unusable_rows': excluded_rows, 'exact_duplicates_removed': exact_duplicates,
    'source_column_count': len(source_columns), 'loaded_column_count': raw.shape[1] - 1,
}
display(pd.Series(data_quality, name='Count').to_frame())

	Count
raw_rows	1296675
usable_unique_rows	1296675
excluded_unusable_rows	0
exact_duplicates_removed	0
source_column_count	24
loaded_column_count	10

4. Chronological partitions

Four successive periods have different jobs. The first trains the model; the second chooses regularisation; the third chooses review thresholds and the final candidate; the last estimates future performance. All fractions are approximate row fractions, with tied timestamps kept together.

Period	Approximate share	Permitted use
Train pool	60%	EDA, fit preprocessing, fit model
Model validation	15%	Select regularisation using average precision, then log loss
Policy validation	10%	Select thresholds and candidate using the declared cost scenario
Final test	15%	Evaluate frozen choices; estimate uncertainty and diagnose limitations

Accounts may recur across periods: this evaluates future transactions in a partly known-customer population. It does not establish performance on entirely new customers. No account ID enters the model. A new-customer deployment needs an additional customer-disjoint evaluation and careful time ordering.

Cell 22 · Set temporal cut points

Cut points depend on recorded times, not on which split gives better model scores.

# Set temporal cut points
if len(data) < 1_000:
    raise ValueError('Too few records for this four-part teaching protocol.')
cut_train = data['timestamp'].iloc[int(len(data) * 0.60)]
cut_model = data['timestamp'].iloc[int(len(data) * 0.75)]
cut_policy = data['timestamp'].iloc[int(len(data) * 0.85)]
assert cut_train < cut_model < cut_policy

Cell 23 · Create the four partitions

Each record belongs to exactly one period.

# Create the four partitions
train_pool = data.loc[data['timestamp'] < cut_train].copy()
model_val = data.loc[data['timestamp'].ge(cut_train) & data['timestamp'].lt(cut_model)].copy()
policy_val = data.loc[data['timestamp'].ge(cut_model) & data['timestamp'].lt(cut_policy)].copy()
test = data.loc[data['timestamp'] >= cut_policy].copy()
partitions = {'Train pool': train_pool, 'Model validation': model_val,
              'Policy validation': policy_val, 'Final test': test}

Cell 24 · Verify separation and label support

These checks catch empty periods, reused rows, tied-time leakage, and missing classes.

# Verify separation and label support
assert sum(len(frame) for frame in partitions.values()) == len(data)
all_indices = np.concatenate([frame.index.to_numpy() for frame in partitions.values()])
assert len(np.unique(all_indices)) == len(data)
frames = list(partitions.values())
for earlier, later in zip(frames[:-1], frames[1:]):
    assert earlier['timestamp'].max() < later['timestamp'].min()
for frame in frames:
    assert set(frame['is_fraud'].unique()) == {0, 1}, 'Both classes are required in each period.'
print('Temporal separation, row disjointness, and class-support checks passed.')

Temporal separation, row disjointness, and class-support checks passed.

Cell 25 · Describe the partitions

Counts and prevalence are descriptive split checks, not final model scores.

# Describe the partitions
split_summary = pd.DataFrame([
    {'period': name, 'rows': len(frame), 'frauds': int(frame['is_fraud'].sum()),
     'fraud_rate': frame['is_fraud'].mean(), 'start': frame['timestamp'].min(),
     'end': frame['timestamp'].max()}
    for name, frame in partitions.items()
]).set_index('period')
display(split_summary.style.format({'fraud_rate': '{:.3%}', 'rows': '{:,}', 'frauds': '{:,}'}))

	rows	frauds	fraud_rate	start	end
period
Train pool	778,005	4,585	0.589%	2019-01-01 00:00:18	2019-11-29 19:37:11
Model validation	194,501	938	0.482%	2019-11-29 19:37:13	2020-01-28 15:01:15
Policy validation	129,667	850	0.656%	2020-01-28 15:04:36	2020-04-03 17:53:52
Final test	194,502	1,133	0.583%	2020-04-03 17:54:44	2020-06-21 12:13:37

Cell 26 · Cap only the training data

Both implementations train on exactly the same uniform sample. No class balancing changes the holdouts.

# Cap only the training data
if MAX_TRAIN_ROWS is not None and len(train_pool) > MAX_TRAIN_ROWS:
    train = train_pool.sample(MAX_TRAIN_ROWS, random_state=RANDOM_SEED).sort_index().copy()
else:
    train = train_pool.copy()
assert set(train['is_fraud'].unique()) == {0, 1}
print(f'Training fit uses {len(train):,} rows, {int(train.is_fraud.sum()):,} frauds '
      f'({train.is_fraud.mean():.3%}).')

Training fit uses 150,000 rows, 892 frauds (0.595%).

Cell 27 · Check customer overlap without printing IDs

A count communicates the evaluation population while protecting direct identifiers.

# Check customer overlap without printing IDs
customer_overlap = None
if 'cc_num' in data and data['cc_num'].notna().all():
    train_customers = set(train_pool['cc_num'])
    future_customers = set(test['cc_num'])
    customer_overlap = len(train_customers & future_customers) / len(future_customers)
    print(f'Fraction of final-test customer IDs also in the train pool: {customer_overlap:.1%}')
else:
    print('Customer overlap cannot be assessed from the available identifier field.')

Fraction of final-test customer IDs also in the train pool: 98.4%

5. Training-data exploration

All plots in this section use the early training pool. Exploration informs understanding, not a causal claim. Later model and policy validation rows remain reserved for their declared selection tasks.

Cell 28 · Show why accuracy needs a baseline

A logarithmic count axis makes the rare class visible.

# Show why accuracy needs a baseline
counts = train_pool['is_fraud'].value_counts().reindex([0, 1])
fig, ax = plt.subplots()
ax.bar(['Legitimate', 'Fraud'], counts, color=[NAVY, SCARLET])
ax.set_yscale('log')
ax.set(ylabel='Transactions (log scale)', title='Fraud is a small share of the training period')
for i, count in enumerate(counts):
    ax.text(i, count * 1.12, f'{count:,}', ha='center')
plt.tight_layout(); plt.show(); plt.close(fig)

Cell 29 · Calculate the majority-rule accuracy

This rule catches no fraud even if its accuracy looks impressive.

# Calculate the majority-rule accuracy
training_fraud_rate = train_pool['is_fraud'].mean()
display(Markdown(
    f'Predicting **legitimate for every training-pool transaction** gives '
    f'**{1 - training_fraud_rate:.2%} accuracy** and **0% fraud recall**. '
    'The final test will have its own independently reported baseline.'
))

Predicting legitimate for every training-pool transaction gives 99.41% accuracy and 0% fraud recall. The final test will have its own independently reported baseline.

Cell 30 · Summarise transaction amounts by label

Medians and quantiles reveal scale and skew without assuming a currency conversion.

# Summarise transaction amounts by label
amount_summary = train_pool.groupby('is_fraud')['amt'].describe(percentiles=[.25, .5, .75, .95])
amount_summary.index = ['Legitimate', 'Fraud']
display(amount_summary.round(2))

	count	mean	std	min	25%	50%	75%	95%	max
Legitimate	773420.0	67.55	151.86	1.00	9.61	47.19	82.46	189.66	26544.12
Fraud	4585.0	525.51	393.42	1.18	207.26	369.09	902.67	1086.78	1371.81

Cell 31 · Plot the amount distributions

Taking log(1 + amount) lets small and large purchases appear on one readable axis.

# Plot the amount distributions
fig, ax = plt.subplots()
for label, name, colour in [(0, 'Legitimate', NAVY), (1, 'Fraud', SCARLET)]:
    values = np.log1p(train_pool.loc[train_pool['is_fraud'].eq(label), 'amt'])
    ax.hist(values, bins=45, density=True, alpha=.55, label=name, color=colour)
ax.set(xlabel='log(1 + transaction amount)', ylabel='Density', title='Amounts overlap: amount alone is not proof of fraud')
ax.legend(); plt.tight_layout(); plt.show(); plt.close(fig)

Cell 32 · Calculate category fraud rates with denominators

A high rate based on a tiny category should not be interpreted like a high rate based on many observations.

# Calculate category fraud rates with denominators
category_summary = train_pool.groupby('category', observed=True)['is_fraud'].agg(rows='size', frauds='sum')
category_summary['fraud_rate'] = category_summary['frauds'] / category_summary['rows']
category_summary = category_summary.sort_values('fraud_rate')
display(category_summary.style.format({'rows': '{:,}', 'frauds': '{:,}', 'fraud_rate': '{:.3%}'}))

	rows	frauds	fraud_rate
category
home	73,806	110	0.149%
health_fitness	51,450	81	0.157%
food_dining	55,201	90	0.163%
kids_pets	67,986	153	0.225%
personal_care	54,646	126	0.231%
entertainment	56,320	138	0.245%
grocery_net	27,152	84	0.309%
misc_pos	47,738	154	0.323%
travel	24,337	80	0.329%
gas_transport	78,990	404	0.511%
shopping_pos	69,929	507	0.725%
misc_net	37,980	547	1.440%
grocery_pos	74,038	1,068	1.443%
shopping_net	58,432	1,043	1.785%

Cell 33 · Plot category rates

This is an association in the training benchmark, not a claim that a merchant category causes fraud.

# Plot category rates
fig, ax = plt.subplots(figsize=(9, 5))
ax.barh(category_summary.index, 100 * category_summary['fraud_rate'], color=NAVY)
ax.set(xlabel='Transactions labelled fraud (%)', title='Fraud rates vary by category in the training period')
plt.tight_layout(); plt.show(); plt.close(fig)

Cell 34 · Inspect hour-of-day rates

Recorded local clock patterns can support features. No timezone or universal behaviour is inferred.

# Inspect hour-of-day rates
hour_summary = train_pool.assign(hour=train_pool['timestamp'].dt.hour).groupby('hour')['is_fraud'].agg(rows='size', frauds='sum')
hour_summary['fraud_rate'] = hour_summary['frauds'] / hour_summary['rows']
fig, ax = plt.subplots()
ax.plot(hour_summary.index, 100 * hour_summary.fraud_rate, marker='o')
ax.set(xlabel='Recorded hour', ylabel='Labelled fraud (%)', title='Training-period pattern across the recorded day', xticks=range(0, 24, 3))
plt.tight_layout(); plt.show(); plt.close(fig)

6. Features and preprocessing

Five numeric features and one low-cardinality category keep the first model readable: log amount, log location-distance proxy, sine and cosine of hour, weekend, and transaction category. Category coefficients are relative to an omitted reference category. We avoid high-cardinality merchant names and direct customer identifiers.

The distance is between source cardholder and merchant coordinates. It is not a verified journey or transaction-terminal GPS measurement. Time-of-day uses the clock as recorded. Hour sine/cosine makes 23:00 close to 00:00 rather than far apart.

We handle imbalance through prevalence-preserving training, ranking metrics, and review-threshold selection. This avoids manufacturing fractional one-hot categories with ordinary SMOTE and avoids interpreting balanced-sample scores as live fraud probabilities. These choices are a baseline design, not a claim that resampling or class weights are always inappropriate.

Cell 35 · Define the location-distance proxy

The haversine formula approximates great-circle distance. Missing inputs remain missing.

# Define the location-distance proxy
def haversine_km(lat1, lon1, lat2, lon2):
    lat1, lon1, lat2, lon2 = [np.radians(np.asarray(v, dtype=float)) for v in (lat1, lon1, lat2, lon2)]
    a = np.sin((lat2 - lat1) / 2) ** 2
    a += np.cos(lat1) * np.cos(lat2) * np.sin((lon2 - lon1) / 2) ** 2
    return 6371.0088 * 2 * np.arcsin(np.sqrt(np.clip(a, 0, 1)))

Cell 36 · Verify the distance calculation

A same-point check and a known quarter-circumference catch unit and formula mistakes.

# Verify the distance calculation
assert np.isclose(haversine_km([0], [0], [0], [0])[0], 0)
assert np.isclose(haversine_km([0], [0], [0], [90])[0], 10007.5572, atol=.01)
print('Distance checks passed.')

Distance checks passed.

Cell 37 · Define one reusable feature rule

The same deterministic transformation must run at training, validation, test, and inference time.

# Define one reusable feature rule
def engineer_features(frame):
    timestamp = pd.to_datetime(frame['trans_date_trans_time'], errors='coerce')
    if timestamp.isna().any():
        raise ValueError('Feature construction requires valid transaction timestamps.')
    amount = pd.to_numeric(frame['amt'], errors='coerce').to_numpy(dtype=float)
    if not np.isfinite(amount).all() or (amount < 0).any():
        raise ValueError('Feature construction requires valid non-negative amounts.')
    coords = [pd.to_numeric(frame[c], errors='coerce').to_numpy(dtype=float) for c in ['lat', 'long', 'merch_lat', 'merch_long']]
    for values, bound in zip(coords, [90, 180, 90, 180]):
        values[np.abs(values) > bound] = np.nan
    distance = haversine_km(*coords)
    angle = 2 * np.pi * timestamp.dt.hour.to_numpy() / 24
    category = frame['category'].astype(object).where(frame['category'].notna(), np.nan)
    return pd.DataFrame({
        'log_amount': np.log1p(amount), 'log_distance': np.log1p(distance),
        'hour_sin': np.sin(angle), 'hour_cos': np.cos(angle),
        'weekend': (timestamp.dt.dayofweek.to_numpy() >= 5).astype(float),
        'category': category.to_numpy(),
    }, index=frame.index)

Cell 38 · Construct training and selection features

Final test features are deliberately constructed only after the policies have been frozen.

# Construct training and selection features
train_features = engineer_features(train)
model_val_features = engineer_features(model_val)
policy_val_features = engineer_features(policy_val)
display(train_features.head().round(3))

	log_amount	log_distance	hour_sin	hour_cos	weekend	category
6	3.819	4.780	0.0	1.0	0.0	grocery_net
7	4.286	2.622	0.0	1.0	0.0	gas_transport
14	2.171	4.205	0.0	1.0	0.0	shopping_pos
18	4.160	4.521	0.0	1.0	0.0	food_dining
24	3.176	4.389	0.0	1.0	0.0	shopping_pos

Cell 39 · Specify numeric preprocessing

Median replacement and scaling learn their values from training rows only.

# Specify numeric preprocessing
numeric_features = ['log_amount', 'log_distance', 'hour_sin', 'hour_cos', 'weekend']
numeric_pipeline = Pipeline([
    ('impute', SimpleImputer(strategy='median', keep_empty_features=True)),
    ('scale', StandardScaler()),
])

Cell 40 · Specify categorical preprocessing

An unseen category becomes an all-zero category vector. Monitor that situation rather than silently treating it as familiar.

# Specify categorical preprocessing
categorical_pipeline = Pipeline([
    ('impute', SimpleImputer(strategy='constant', fill_value='__missing__')),
    ('encode', OneHotEncoder(drop='first', handle_unknown='ignore', sparse_output=True)),
])

Cell 41 · Combine the preprocessing rules

Keeping one fitted transformer prevents different implementations from seeing different feature spaces.

# Combine the preprocessing rules
preprocessor = ColumnTransformer([
    ('numeric', numeric_pipeline, numeric_features),
    ('category', categorical_pipeline, ['category']),
], sparse_threshold=1.0, verbose_feature_names_out=False)

Cell 42 · Fit preprocessing on training rows

This is the only preprocessing fit. Stored means, medians, and category levels are reused later.

# Fit preprocessing on training rows
X_train = sparse.csr_matrix(preprocessor.fit_transform(train_features), dtype=np.float64)
y_train = train['is_fraud'].to_numpy(dtype=float)
feature_names = preprocessor.get_feature_names_out()
print(f'Transformed training matrix: {X_train.shape}; sparse non-zero fraction: {X_train.nnz / np.prod(X_train.shape):.2%}')

Transformed training matrix: (150000, 18); sparse non-zero fraction: 32.92%

Cell 43 · Transform the two selection periods

Transform applies learned rules without learning anything from future rows.

# Transform the two selection periods
X_model_val = sparse.csr_matrix(preprocessor.transform(model_val_features), dtype=np.float64)
X_policy_val = sparse.csr_matrix(preprocessor.transform(policy_val_features), dtype=np.float64)
y_model_val = model_val['is_fraud'].to_numpy(dtype=int)
y_policy_val = policy_val['is_fraud'].to_numpy(dtype=int)

Cell 44 · Verify the transformed matrices

Check that columns align and the optimiser will not receive infinities or missing values.

# Verify the transformed matrices
for matrix in [X_train, X_model_val, X_policy_val]:
    assert matrix.shape[1] == len(feature_names)
    assert np.isfinite(matrix.data).all()
reference_category = preprocessor.named_transformers_['category'].named_steps['encode'].categories_[0][0]
print('Preprocessing checks passed; category reference:', reference_category)

Preprocessing checks passed; category reference: entertainment

7. Logistic regression from first principles

For one transaction, a weighted sum of the features produces a score:

$$z = x^T w + b, \qquad p = \sigma(z) = \frac{1}{1+e^{-z}}.$$

The sigmoid maps a score to a number between zero and one. A fitted score is not automatically a well-calibrated fraud probability; we check calibration later. This illustration uses a score axis only and is not a separate fraud experiment.

Cell 45 · Define a numerically stable sigmoid

Using expit avoids overflow at extreme positive or negative scores.

# Define a numerically stable sigmoid
def sigmoid(scores):
    return expit(scores)

Cell 46 · Plot the sigmoid

Moving the threshold changes decisions while leaving the underlying risk ranking unchanged.

# Plot the sigmoid
z_demo = np.linspace(-8, 8, 300)
fig, ax = plt.subplots()
ax.plot(z_demo, sigmoid(z_demo))
ax.axhline(.5, color=SILVER, linestyle='--')
ax.set(xlabel='Linear score z', ylabel='Sigmoid output', title='How a logistic score becomes a probability-shaped output')
plt.tight_layout(); plt.show(); plt.close(fig)

The training objective is mean binary cross-entropy plus L2 regularisation:

$$L = \frac{1}{n}\sum_i[\log(1+e^{z_i})-y_i z_i] +\frac{\lambda}{2}\lVert w\rVert_2^2.$$

logaddexp(0, z) computes the logarithmic term stably. The intercept is not penalised. The gradients are $X^T(p-y)/n + \lambda w$ for weights and the mean of $p-y$ for bias. The loss encourages useful probability estimates; L2 shrinks large coefficients.

Cell 47 · Define the objective

This formula is equivalent to correctly signed binary cross-entropy without taking log(0).

# Define the objective
def logistic_objective(X, y, weights, bias, l2):
    scores = np.asarray(X @ weights).ravel() + bias
    return float(np.mean(np.logaddexp(0, scores) - y * scores) + l2 * np.dot(weights, weights) / 2)

Cell 48 · Define the gradients

These derivatives tell gradient descent which direction reduces the objective.

# Define the gradients
def logistic_gradients(X, y, weights, bias, l2):
    probabilities = sigmoid(np.asarray(X @ weights).ravel() + bias)
    errors = probabilities - y
    weight_gradient = np.asarray(X.T @ errors).ravel() / len(y) + l2 * weights
    bias_gradient = float(errors.mean())
    return weight_gradient, bias_gradient

Cell 49 · Check the analytic gradient

Finite differences provide an independent check of the loss sign and regularisation derivative.

# Check the analytic gradient
check_X = np.array([[1., -.5], [0., 1.], [2., .2], [-1., .4]])
check_y, check_w, check_b, check_l2 = np.array([0., 1., 1., 0.]), np.array([.2, -.3]), .1, .01
analytic_w, analytic_b = logistic_gradients(check_X, check_y, check_w, check_b, check_l2)
epsilon = 1e-6
finite_w = []
for j in range(len(check_w)):
    delta = np.zeros_like(check_w); delta[j] = epsilon
    finite_w.append((logistic_objective(check_X, check_y, check_w + delta, check_b, check_l2)
                     - logistic_objective(check_X, check_y, check_w - delta, check_b, check_l2)) / (2 * epsilon))
finite_b = (logistic_objective(check_X, check_y, check_w, check_b + epsilon, check_l2)
            - logistic_objective(check_X, check_y, check_w, check_b - epsilon, check_l2)) / (2 * epsilon)
assert np.allclose(analytic_w, finite_w, atol=1e-7) and np.isclose(analytic_b, finite_b, atol=1e-7)
assert np.isfinite(sigmoid(np.array([-1e6, 0, 1e6]))).all()
print('Analytic gradient, bias gradient, and extreme-score checks passed.')

Analytic gradient, bias gradient, and extreme-score checks passed.

Cell 50 · Define a safe gradient-descent update

Backtracking reduces an over-large step until it gives sufficient descent. This makes the optimisation stable rather than tuning learning rate on final outcomes.

# Define a safe gradient-descent update
def descent_update(X, y, weights, bias, l2, step_size):
    dw, db = logistic_gradients(X, y, weights, bias, l2)
    old_loss = logistic_objective(X, y, weights, bias, l2)
    squared_norm = float(np.dot(dw, dw) + db * db)
    for _ in range(50):
        new_w, new_b = weights - step_size * dw, bias - step_size * db
        new_loss = logistic_objective(X, y, new_w, new_b, l2)
        if np.isfinite(new_loss) and new_loss <= old_loss - 1e-4 * step_size * squared_norm:
            return new_w, new_b, new_loss, step_size, max(np.abs(dw).max(), abs(db))
        step_size *= .5
    raise RuntimeError('Backtracking failed; inspect the features and objective.')

Cell 51 · Define the training loop

One cohesive function trains one model. The initial bias reflects the training prevalence and the stopping rule checks the gradient.

# Define the training loop
def fit_scratch(X, y, l2, max_iterations=MAX_ITERATIONS, tolerance=GRADIENT_TOLERANCE):
    weights = np.zeros(X.shape[1], dtype=float)
    prevalence = np.clip(y.mean(), 1e-8, 1 - 1e-8)
    bias = float(np.log(prevalence / (1 - prevalence)))
    history, step_size = [], 16.0
    start = time.perf_counter()
    for iteration in range(1, max_iterations + 1):
        weights, bias, loss, accepted_step, gradient_norm = descent_update(X, y, weights, bias, l2, step_size)
        if iteration == 1 or iteration % 10 == 0 or gradient_norm < tolerance:
            history.append({'iteration': iteration, 'objective': loss, 'step': accepted_step})
        step_size = min(accepted_step * 1.15, 64.0)
        if gradient_norm < tolerance:
            break
    dw, db = logistic_gradients(X, y, weights, bias, l2)
    final_gradient = float(max(np.abs(dw).max(), abs(db)))
    return {'weights': weights, 'bias': bias, 'l2': l2, 'history': pd.DataFrame(history),
            'iterations': iteration, 'gradient_norm': final_gradient,
            'converged': final_gradient < tolerance, 'fit_seconds': time.perf_counter() - start}

Cell 52 · Define scratch probability prediction

Keep probability scoring separate from threshold-based review decisions.

# Define scratch probability prediction
def scratch_probability(model, X):
    return sigmoid(np.asarray(X @ model['weights']).ravel() + model['bias'])

8. Model selection

Select each implementation's L2 value on the model-validation period using average precision (AP), with log loss as the tie breaker. AP summarises the precision–recall curve; it is not the trapezoidal PR area. Both models use the same features, training sample, unweighted labels, and candidate regularisation strengths.

Scikit-learn's inverse penalty parameter is matched to our mean-loss convention: C = 1 / (lambda * n_train). The optimiser differs; the objective does not. Solver agreement is also checked on a fixed regularisation value, separately from selection. No test scores guide either search.

Cell 53 · Train the scratch candidates

The small candidate set makes the search readable and its computational cost explicit.

# Train the scratch candidates
scratch_candidates = {}
for l2 in L2_CANDIDATES:
    scratch_candidates[l2] = fit_scratch(X_train, y_train, l2)
    fitted = scratch_candidates[l2]
    print(f"Scratch lambda={l2:g}: {fitted['iterations']} iterations, gradient={fitted['gradient_norm']:.2g}, converged={fitted['converged']}")

Scratch lambda=0.0001: 235 iterations, gradient=9.9e-06, converged=True
Scratch lambda=0.001: 78 iterations, gradient=9.3e-06, converged=True
Scratch lambda=0.01: 22 iterations, gradient=5.3e-06, converged=True

Cell 54 · Score scratch candidates on model validation

These are selection scores, not final claims of performance.

# Score scratch candidates on model validation
scratch_selection = pd.DataFrame([
    {'l2': l2, 'validation_ap': average_precision_score(y_model_val, scratch_probability(model, X_model_val)),
     'validation_log_loss': log_loss(y_model_val, scratch_probability(model, X_model_val)),
     'iterations': model['iterations'], 'converged': model['converged']}
    for l2, model in scratch_candidates.items()
]).sort_values(['validation_ap', 'validation_log_loss'], ascending=[False, True])
display(scratch_selection)

	l2	validation_ap	validation_log_loss	iterations	converged
0	0.0001	0.345141	0.019169	235	True
1	0.0010	0.313585	0.020013	78	True
2	0.0100	0.295729	0.024857	22	True

Cell 55 · Freeze the scratch regularisation choice

The selected model remains fitted on the original training sample; validation rows are not added to its training.

# Freeze the scratch regularisation choice
scratch_l2 = float(scratch_selection.iloc[0]['l2'])
scratch_model = scratch_candidates[scratch_l2]
print('Selected scratch lambda:', scratch_l2)
if not scratch_model['converged']:
    print('Convergence tolerance was not reached; treat the solver comparison as provisional.')

Selected scratch lambda: 0.0001

Cell 56 · Train the matched library candidates

The alias prevents namespace conflicts. Default L2 regularisation avoids a deprecated penalty argument.

# Train the matched library candidates
library_candidates = {}
for l2 in L2_CANDIDATES:
    fitted = SklearnLogisticRegression(C=1 / (l2 * len(y_train)), solver='lbfgs', max_iter=2_000, tol=1e-9)
    fitted.fit(X_train, y_train)
    library_candidates[l2] = fitted
    print(f'Library lambda={l2:g}: {int(fitted.n_iter_[0])} solver iterations')

Library lambda=0.0001: 52 solver iterations
Library lambda=0.001: 28 solver iterations
Library lambda=0.01: 15 solver iterations

Cell 57 · Score library candidates on model validation

The metric and selection rules match the scratch search.

# Score library candidates on model validation
library_selection = pd.DataFrame([
    {'l2': l2, 'validation_ap': average_precision_score(y_model_val, model.predict_proba(X_model_val)[:, 1]),
     'validation_log_loss': log_loss(y_model_val, model.predict_proba(X_model_val)[:, 1]),
     'iterations': int(model.n_iter_[0])}
    for l2, model in library_candidates.items()
]).sort_values(['validation_ap', 'validation_log_loss'], ascending=[False, True])
display(library_selection)

	l2	validation_ap	validation_log_loss	iterations
0	0.0001	0.345275	0.019158	52
1	0.0010	0.313714	0.020003	28
2	0.0100	0.295732	0.024854	15

Cell 58 · Freeze the library regularisation choice

Neither implementation is chosen by looking at final test performance.

# Freeze the library regularisation choice
library_l2 = float(library_selection.iloc[0]['l2'])
library_model = library_candidates[library_l2]
print('Selected library lambda:', library_l2)

Selected library lambda: 0.0001

Cell 59 · Check agreement for the same objective

Comparing predicted scores on a fixed lambda verifies that the scratch implementation approximates the library solution.

# Check agreement for the same objective
parity_l2 = L2_CANDIDATES[1]
parity_scratch = scratch_probability(scratch_candidates[parity_l2], X_model_val)
parity_library = library_candidates[parity_l2].predict_proba(X_model_val)[:, 1]
parity_mean_gap = float(np.mean(np.abs(parity_scratch - parity_library)))
print(f'Mean absolute probability difference at lambda={parity_l2:g}: {parity_mean_gap:.8f}')
assert parity_mean_gap < 0.005, 'Solver discrepancy is too large; inspect convergence before proceeding.'

Mean absolute probability difference at lambda=0.001: 0.00001518

Cell 60 · Plot the scratch optimisation trace

A decreasing objective supports numerical stability. It does not demonstrate generalisation by itself.

# Plot the scratch optimisation trace
history = scratch_model['history']
fig, ax = plt.subplots()
ax.plot(history.iteration, history.objective)
ax.set(xlabel='Gradient-descent iteration', ylabel='Penalised training objective', title='Scratch optimiser: learning on training rows only')
plt.tight_layout(); plt.show(); plt.close(fig)

9. Review policies

The same score can support different operational decisions. In the declared scenario:

cost = review_cost × all_alerts + false_alert_friction × false_alerts + missed_fraud_cost × missed_frauds.

On the policy-validation period, select the threshold with the smallest cost among policies alerting at most the declared fraction of transactions. Ties prefer fewer alerts and then a higher threshold. A no-alert policy is a valid candidate. Choose the final implementation by its selected policy cost, with a library preference if costs tie. All choices are frozen before final testing.

This estimates a retrospective scenario. It assumes every flagged fraud avoids the constant missed-loss amount, ignores residual losses and implementation costs, and does not establish real savings. A fraction cap is a proxy for workload; a live team would need explicit per-day or per-shift capacity and investigation effectiveness.

Cell 61 · Score the policy-validation period

This period was used neither to train models nor to select their regularisation.

# Score the policy-validation period
policy_probabilities = {
    'Scratch': scratch_probability(scratch_model, X_policy_val),
    'Scikit-learn': library_model.predict_proba(X_policy_val)[:, 1],
}

Cell 62 · Define an exact threshold sweep

Sorting once and using cumulative counts evaluates every distinct score efficiently. Equal scores receive the same decision.

# Define an exact threshold sweep
def threshold_sweep(y, probabilities):
    order = np.argsort(-probabilities, kind='stable')
    p, labels = probabilities[order], y[order]
    endpoints = np.r_[np.flatnonzero(np.diff(p) != 0), len(p) - 1]
    tp = np.cumsum(labels)[endpoints]
    alerts = endpoints + 1
    fp, fn = alerts - tp, int(y.sum()) - tp
    table = pd.DataFrame({'threshold': p[endpoints], 'tp': tp, 'fp': fp, 'fn': fn, 'alerts': alerts})
    none = pd.DataFrame([{'threshold': np.nextafter(float(p.max()), np.inf), 'tp': 0, 'fp': 0, 'fn': int(y.sum()), 'alerts': 0}])
    table = pd.concat([none, table], ignore_index=True)
    table['alert_rate'] = table.alerts / len(y)
    table['precision'] = np.divide(table.tp, table.alerts, out=np.zeros(len(table)), where=table.alerts.to_numpy() > 0)
    table['recall'] = table.tp / y.sum()
    table['cost_mu'] = REVIEW_COST_MU * table.alerts + FALSE_ALERT_FRICTION_MU * table.fp + MISSED_FRAUD_COST_MU * table.fn
    return table

Cell 63 · Verify tie handling and the no-alert option

This independent example checks the cumulative logic against explicit decisions.

# Verify tie handling and the no-alert option
tie_y, tie_p = np.array([1, 0, 1, 0]), np.array([.8, .8, .2, .1])
tie_table = threshold_sweep(tie_y, tie_p)
for row in tie_table.itertuples():
    prediction = tie_p >= row.threshold
    assert row.tp == np.sum(prediction & (tie_y == 1))
    assert row.fp == np.sum(prediction & (tie_y == 0))
assert tie_table.iloc[0].alerts == 0
print('Threshold tie and confusion-count checks passed.')

Threshold tie and confusion-count checks passed.

Cell 64 · Evaluate policy candidates

Each sweep is an internal selection table; the entire table is not printed.

# Evaluate policy candidates
policy_sweeps = {name: threshold_sweep(y_policy_val, probabilities)
                 for name, probabilities in policy_probabilities.items()}

Cell 65 · Select review thresholds under the capacity cap

The cap is checked on policy validation. Future alert volume can exceed it if distributions change.

# Select review thresholds under the capacity cap
policy_rows = []
for name, table in policy_sweeps.items():
    feasible = table.loc[table.alert_rate <= MAX_POLICY_ALERT_RATE]
    choice = feasible.sort_values(['cost_mu', 'alerts', 'threshold'], ascending=[True, True, False]).iloc[0]
    policy_rows.append({'implementation': name, **choice.to_dict()})
policy_choices = pd.DataFrame(policy_rows).set_index('implementation')
display(policy_choices[['threshold', 'precision', 'recall', 'alert_rate', 'cost_mu']].round(5))

	threshold	precision	recall	alert_rate	cost_mu
implementation
Scratch	0.05073	0.24609	0.68471	0.01824	156560.0
Scikit-learn	0.05079	0.24671	0.68471	0.01819	156488.0

Cell 66 · Freeze the final candidate and thresholds

The implementation choice is made here, before any final test model predictions exist.

# Freeze the final candidate and thresholds
ranked_policies = policy_choices.assign(tie_preference=[1 if name == 'Scratch' else 0 for name in policy_choices.index])
selected_name = ranked_policies.sort_values(['cost_mu', 'tie_preference']).index[0]
frozen_thresholds = policy_choices.threshold.astype(float).to_dict()
selected_threshold = frozen_thresholds[selected_name]
print(f'Frozen candidate: {selected_name}; review threshold: {selected_threshold:.6f}')

Frozen candidate: Scikit-learn; review threshold: 0.050788

Cell 67 · Visualise the threshold trade-off

The shaded feasible region expresses the declared capacity constraint, not a guarantee of future workload.

# Visualise the threshold trade-off
table = policy_sweeps[selected_name]
plot_table = table.loc[table.alert_rate <= .05].iloc[::max(1, int((table.alert_rate <= .05).sum() / 600))]
fig, ax = plt.subplots()
ax.plot(100 * plot_table.alert_rate, plot_table.cost_mu / len(y_policy_val) * 10_000)
ax.axvline(100 * MAX_POLICY_ALERT_RATE, color=SCARLET, linestyle='--', label='Validation capacity cap')
choice = policy_choices.loc[selected_name]
ax.scatter([100 * choice.alert_rate], [choice.cost_mu / len(y_policy_val) * 10_000], color=SCARLET, zorder=3)
ax.set(xlabel='Transactions sent for review (%)', ylabel='Illustrative cost (MU per 10,000)', title='Policy-validation choice: cost and workload')
ax.legend(); plt.tight_layout(); plt.show(); plt.close(fig)

10. Final held-out results

The final period is opened now. The two models and thresholds are already frozen. We report both the default 0.5 threshold and each chosen review policy so readers can see how changing a decision rule differs from changing a model.

Metric	Interpretation
Average precision (AP)	How well high-ranked rows concentrate fraud; compare with prevalence
ROC AUC	Ranking discrimination across thresholds; use continuous scores
Precision	Of alerts, the fraction labelled fraud
Recall	Of labelled fraud, the fraction alerted
Alert rate	Review workload as a fraction of transactions
Brier score / log loss	Probability errors; smaller is better
Accuracy	Overall correctness; read alongside the majority rule
Cost / 10,000	The declared retrospective cost scenario, not measured savings

The majority baseline predicts legitimate everywhere and uses the sampled training prevalence as its constant probability score. Its AP on the test set equals test prevalence and its ROC AUC is 0.5.

Cell 68 · Construct and transform final test features

The held-out period receives the same deterministic feature rules and already fitted preprocessor.

# Construct and transform final test features
test_features = engineer_features(test)
X_test = sparse.csr_matrix(preprocessor.transform(test_features), dtype=np.float64)
y_test = test['is_fraud'].to_numpy(dtype=int)
assert X_test.shape[1] == X_train.shape[1] and np.isfinite(X_test.data).all()

Cell 69 · Score the final test period

Both implementations score exactly the same rows.

# Score the final test period
test_probabilities = {
    'Scratch': scratch_probability(scratch_model, X_test),
    'Scikit-learn': library_model.predict_proba(X_test)[:, 1],
}
baseline_probability = np.full(len(test), y_train.mean())

Cell 70 · Define one evaluation rule

One function prevents metrics in tables, charts, and prose from using inconsistent labels or thresholds.

# Define one evaluation rule
def evaluate_policy(y, probability, threshold, name):
    predicted = (probability >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y, predicted, labels=[0, 1]).ravel()
    alerts = int(tp + fp)
    cost = REVIEW_COST_MU * alerts + FALSE_ALERT_FRICTION_MU * fp + MISSED_FRAUD_COST_MU * fn
    return {'policy': name, 'threshold': threshold, 'accuracy': accuracy_score(y, predicted),
            'balanced_accuracy': balanced_accuracy_score(y, predicted),
            'precision': precision_score(y, predicted, zero_division=0),
            'recall': recall_score(y, predicted, zero_division=0), 'f1': f1_score(y, predicted, zero_division=0),
            'average_precision': average_precision_score(y, probability), 'roc_auc': roc_auc_score(y, probability),
            'brier_score': brier_score_loss(y, probability), 'log_loss': log_loss(y, probability),
            'alert_rate': alerts / len(y), 'tn': int(tn), 'fp': int(fp), 'fn': int(fn), 'tp': int(tp),
            'cost_mu_per_10000': cost / len(y) * 10_000}

Cell 71 · Evaluate the baseline and both model policies

Threshold 1.0 gives the baseline no alerts. The chosen model remains the earlier frozen candidate regardless of this table.

# Evaluate the baseline and both model policies
test_rows = [evaluate_policy(y_test, baseline_probability, 1.0, 'Always legitimate')]
for name, probability in test_probabilities.items():
    test_rows.append(evaluate_policy(y_test, probability, .5, f'{name} at 0.5'))
    test_rows.append(evaluate_policy(y_test, probability, frozen_thresholds[name], f'{name} review policy'))
test_metrics = pd.DataFrame(test_rows).set_index('policy')
display(test_metrics[['average_precision', 'roc_auc', 'accuracy', 'precision', 'recall', 'alert_rate', 'cost_mu_per_10000']].round(5))

	average_precision	roc_auc	accuracy	precision	recall	alert_rate	cost_mu_per_10000
policy
Always legitimate	0.00583	0.50000	0.99417	0.00000	0.00000	0.00000	29125.66452
Scratch at 0.5	0.36411	0.88660	0.99408	0.38750	0.02736	0.00041	28362.17622
Scratch review policy	0.36411	0.88660	0.98425	0.22023	0.67079	0.01774	11326.97864
Scikit-learn at 0.5	0.36456	0.88673	0.99409	0.40244	0.02913	0.00042	28310.96852
Scikit-learn review policy	0.36456	0.88673	0.98434	0.22128	0.66990	0.01763	11340.24329

Cell 72 · Define the selected test decisions

This name still comes from policy validation, not from choosing the best-looking row in the final table.

# Define the selected test decisions
selected_probability = test_probabilities[selected_name]
selected_prediction = (selected_probability >= selected_threshold).astype(int)
selected_metrics = test_metrics.loc[f'{selected_name} review policy']
selected_matrix = confusion_matrix(y_test, selected_prediction, labels=[0, 1])

Cell 73 · Show the selected confusion matrix

False alerts mean analyst work and possible customer friction; missed fraud means loss exposure.

# Show the selected confusion matrix
fig, ax = plt.subplots(figsize=(6.5, 5))
ax.imshow(np.log1p(selected_matrix), cmap='Blues')
for (i, j), value in np.ndenumerate(selected_matrix):
    ax.text(j, i, f'{value:,}', ha='center', va='center', color='black' if value < selected_matrix.max() / 2 else 'white', fontsize=13)
ax.set(xticks=[0, 1], yticks=[0, 1], xticklabels=['No alert', 'Review'], yticklabels=['Legitimate', 'Fraud'],
       xlabel='Decision', ylabel='Observed label', title=f'Final test: {selected_name} frozen policy')
plt.tight_layout(); plt.show(); plt.close(fig)

Cell 74 · Plot precision–recall curves

The horizontal line is test prevalence. AP is reported from average_precision_score rather than trapezoidal integration.

# Plot precision–recall curves
fig, ax = plt.subplots()
for name, probability in test_probabilities.items():
    precision, recall, _ = precision_recall_curve(y_test, probability)
    ax.plot(recall, precision, label=f'{name}: AP={average_precision_score(y_test, probability):.3f}')
ax.axhline(y_test.mean(), linestyle='--', color=SILVER, label=f'Prevalence={y_test.mean():.3%}')
ax.set(xlabel='Fraud recall', ylabel='Alert precision', title='Final test: concentration of fraud in high-ranked transactions', xlim=(0, 1), ylim=(0, 1))
ax.legend(); plt.tight_layout(); plt.show(); plt.close(fig)

Cell 75 · Plot ROC curves using scores

AUC reflects ranking and does not by itself specify acceptable false-alert volume.

# Plot ROC curves using scores
fig, ax = plt.subplots()
for name, probability in test_probabilities.items():
    fpr, tpr, _ = roc_curve(y_test, probability)
    ax.plot(fpr, tpr, label=f'{name}: AUC={roc_auc_score(y_test, probability):.3f}')
ax.plot([0, 1], [0, 1], linestyle='--', color=SILVER)
ax.set(xlabel='False-positive rate', ylabel='True-positive rate', title='Final test: ranking discrimination')
ax.legend(); plt.tight_layout(); plt.show(); plt.close(fig)

Cell 76 · Inspect probability calibration

Quantile bins give each bin observations in a rare-event setting. Calibration on this test is descriptive; it is not used to refit or adjust scores.

# Inspect probability calibration
observed_rate, mean_score = calibration_curve(y_test, selected_probability, n_bins=10, strategy='quantile')
fig, ax = plt.subplots()
ax.plot(mean_score, observed_rate, marker='o', label=selected_name)
upper = max(float(mean_score.max()), float(observed_rate.max())) * 1.1
ax.plot([0, upper], [0, upper], linestyle='--', color=SILVER, label='Ideal calibration')
ax.set(xlabel='Mean model score in bin', ylabel='Observed fraud rate in bin', title='Final test: are probability-shaped scores reliable?')
ax.legend(); plt.tight_layout(); plt.show(); plt.close(fig)

11. Uncertainty and interpretation

Repeated transactions from one customer are dependent. We therefore resample customer clusters, retaining their transactions together, rather than treating every row as an independent customer. Percentile intervals describe uncertainty across customers within this fixed test period, conditional on the trained model and threshold. They do not include training uncertainty, label errors, temporal shocks, or cross-customer dependence. A 200-repeat bootstrap is a teaching-scale estimate; a publication should assess interval stability with more repetitions.

If usable customer IDs are absent, this notebook explicitly skips cluster intervals rather than silently substituting an IID row bootstrap.

Cell 77 · Define the cluster bootstrap

Multiplicities weight whole customers without physically copying millions of rows.

# Define the cluster bootstrap
def bootstrap_customers(frame, labels, probability, prediction, repeats, seed):
    codes, customers = pd.factorize(frame['cc_num'], sort=True)
    if (codes < 0).any() or len(customers) < 2:
        raise ValueError('Cluster bootstrap requires complete customer IDs and at least two clusters.')
    rng, records = np.random.default_rng(seed), []
    for _ in range(repeats):
        multiplicities = rng.multinomial(len(customers), np.full(len(customers), 1 / len(customers)))
        weights = multiplicities[codes].astype(float)
        fraud_weight = float(np.sum(weights * labels))
        legitimate_weight = float(np.sum(weights * (1 - labels)))
        if fraud_weight == 0 or legitimate_weight == 0:
            continue
        tp = float(np.sum(weights * labels * prediction))
        fp = float(np.sum(weights * (1 - labels) * prediction))
        fn, alerts = fraud_weight - tp, tp + fp
        cost = REVIEW_COST_MU * alerts + FALSE_ALERT_FRICTION_MU * fp + MISSED_FRAUD_COST_MU * fn
        records.append({'average_precision': average_precision_score(labels, probability, sample_weight=weights),
                        'recall': tp / fraud_weight, 'precision': tp / alerts if alerts else 0.0,
                        'alert_rate': alerts / weights.sum(), 'cost_mu_per_10000': cost / weights.sum() * 10_000})
    return pd.DataFrame(records), len(customers)

Cell 78 · Estimate uncertainty for the frozen policy

The bootstrap evaluates the fixed final candidate. It never reselects a threshold on test rows.

# Estimate uncertainty for the frozen policy
bootstrap_samples = pd.DataFrame()
bootstrap_clusters = None
if 'cc_num' in test and test['cc_num'].notna().all():
    bootstrap_samples, bootstrap_clusters = bootstrap_customers(test, y_test, selected_probability, selected_prediction, BOOTSTRAP_REPEATS, RANDOM_SEED)
    print(f'Usable bootstrap replicates: {len(bootstrap_samples)} / {BOOTSTRAP_REPEATS}; customer clusters: {bootstrap_clusters}')
else:
    print('Cluster bootstrap skipped: complete customer IDs were not supplied.')

Usable bootstrap replicates: 200 / 200; customer clusters: 923

Cell 79 · Display percentile intervals

Intervals are approximate and conditional. Broad intervals should reduce confidence in a precise operational claim.

# Display percentile intervals
uncertainty_intervals = pd.DataFrame()
if not bootstrap_samples.empty:
    uncertainty_intervals = bootstrap_samples.quantile([.025, .975]).T
    uncertainty_intervals.columns = ['lower_95', 'upper_95']
    uncertainty_intervals.insert(0, 'point_estimate', selected_metrics[uncertainty_intervals.index].astype(float))
    display(uncertainty_intervals.round(5))

	point_estimate	lower_95	upper_95
average_precision	0.36456	0.31456	0.41429
recall	0.66990	0.64036	0.71025
precision	0.22128	0.18794	0.25759
alert_rate	0.01763	0.01658	0.01875
cost_mu_per_10000	11340.24329	9268.37797	13554.48204

Cell 80 · Interpret the held-out result in plain language

This text is computed from the actual final metrics, so it cannot retain stale scores from an earlier run.

# Interpret the held-out result in plain language
baseline_cost = float(test_metrics.loc['Always legitimate', 'cost_mu_per_10000'])
cost_difference = baseline_cost - float(selected_metrics.cost_mu_per_10000)
direction = 'lower' if cost_difference >= 0 else 'higher'
capacity_status = 'within' if selected_metrics.alert_rate <= MAX_POLICY_ALERT_RATE else 'above'
display(Markdown(
    f'**Frozen candidate: {selected_name}.** The final period contains **{len(test):,} transactions** '
    f'and **{int(y_test.sum()):,} labelled frauds** ({y_test.mean():.3%}). '
    f'The review policy flags **{int(selected_metrics.tp + selected_metrics.fp):,} transactions**, '
    f'including **{int(selected_metrics.tp):,} frauds** and **{int(selected_metrics.fp):,} legitimate transactions**. '
    f'It misses **{int(selected_metrics.fn):,} frauds**. Recall is **{selected_metrics.recall:.1%}** '
    f'and alert precision is **{selected_metrics.precision:.1%}**. '
    f'That means **{1 - selected_metrics.precision:.1%} of alerts are legitimate**, '
    f'and **{1 - selected_metrics.recall:.1%} of labelled fraud remains missed**. '
    'These limitations matter when deciding whether the queue is useful to an investigation team.\n\n'
    f'AP is **{selected_metrics.average_precision:.3f}**, compared with a prevalence baseline of '
    f'**{y_test.mean():.4f}**. Under the declared scenario, cost is **{abs(cost_difference):,.1f} MU per 10,000 {direction}** '
    'than the no-alert baseline. This is scenario arithmetic, not realised savings. '
    f'Test alert volume is **{capacity_status}** the {MAX_POLICY_ALERT_RATE:.0%} validation cap; '
    'a validation capacity constraint cannot guarantee future capacity.'
))

Frozen candidate: Scikit-learn. The final period contains 194,502 transactions and 1,133 labelled frauds (0.583%). The review policy flags 3,430 transactions, including 759 frauds and 2,671 legitimate transactions. It misses 374 frauds. Recall is 67.0% and alert precision is 22.1%. That means 77.9% of alerts are legitimate, and 33.0% of labelled fraud remains missed. These limitations matter when deciding whether the queue is useful to an investigation team.

AP is 0.365, compared with a prevalence baseline of 0.0058. Under the declared scenario, cost is 17,785.4 MU per 10,000 lower than the no-alert baseline. This is scenario arithmetic, not realised savings. Test alert volume is within the 2% validation cap; a validation capacity constraint cannot guarantee future capacity.

Cell 81 · Summarise amounts associated with detected fraud

Labelled amounts screened are not the same as amounts recovered or losses prevented.

# Summarise amounts associated with detected fraud
fraud_amounts = test['amt'].to_numpy()[y_test == 1]
screened_amount = float(fraud_amounts[selected_prediction[y_test == 1] == 1].sum())
total_fraud_amount = float(fraud_amounts.sum())
amount_coverage = screened_amount / total_fraud_amount if total_fraud_amount else np.nan
display(pd.Series({'labelled_fraud_amount': total_fraud_amount, 'labelled_amount_alerted': screened_amount,
                   'labelled_amount_coverage': amount_coverage}, name='Source amount units').to_frame())

	Source amount units
labelled_fraud_amount	604108.480000
labelled_amount_alerted	546177.700000
labelled_amount_coverage	0.904105

Cell 82 · Inspect outcomes over the final period

Monthly counts expose drift and sparse positive support. They diagnose limitations without revising the frozen model.

# Inspect outcomes over the final period
monthly = test[['timestamp', 'is_fraud']].assign(alert=selected_prediction, caught=y_test * selected_prediction)
monthly['month'] = monthly['timestamp'].dt.to_period('M').astype(str)
monthly_summary = monthly.groupby('month').agg(rows=('is_fraud', 'size'), frauds=('is_fraud', 'sum'), alerts=('alert', 'sum'), caught=('caught', 'sum'))
monthly_summary['alert_rate'] = monthly_summary.alerts / monthly_summary.rows
monthly_summary['recall'] = monthly_summary.caught / monthly_summary.frauds.replace(0, np.nan)
display(monthly_summary.round(4))

	rows	frauds	alerts	caught	alert_rate	recall
month
2020-04	62412	272	1048	184	0.0168	0.6765
2020-05	74343	527	1383	358	0.0186	0.6793
2020-06	57747	334	999	217	0.0173	0.6497

Cell 83 · Read coefficients as associations

Numeric odds ratios refer to a one-training-standard-deviation increase. Category odds ratios compare with the omitted reference; neither is a causal effect.

# Read coefficients as associations
selected_weights = scratch_model['weights'] if selected_name == 'Scratch' else library_model.coef_[0]
coefficients = pd.DataFrame({'feature': feature_names, 'coefficient': selected_weights})
coefficients['conditional_odds_ratio'] = np.exp(coefficients.coefficient)
coefficients['contrast'] = ['one training SD' if name in numeric_features else f'versus category {reference_category}' for name in feature_names]
coefficients = coefficients.sort_values('coefficient', ascending=False)
display(coefficients.round(4))

	feature	coefficient	conditional_odds_ratio	contrast
0	log_amount	1.8577	6.4090	one training SD
3	hour_cos	1.2923	3.6410	one training SD
8	category_grocery_pos	0.8542	2.3494	versus category entertainment
12	category_misc_net	0.7568	2.1314	versus category entertainment
15	category_shopping_net	0.7504	2.1179	versus category entertainment
6	category_gas_transport	0.4546	1.5756	versus category entertainment
7	category_grocery_net	0.2247	1.2519	versus category entertainment
5	category_food_dining	0.0471	1.0482	versus category entertainment
1	log_distance	0.0354	1.0360	one training SD
14	category_personal_care	-0.0182	0.9820	versus category entertainment
4	weekend	-0.0253	0.9750	one training SD
16	category_shopping_pos	-0.1042	0.9011	versus category entertainment
11	category_kids_pets	-0.1973	0.8209	versus category entertainment
13	category_misc_pos	-0.2345	0.7909	versus category entertainment
9	category_health_fitness	-0.2429	0.7844	versus category entertainment
2	hour_sin	-0.3041	0.7378	one training SD
10	category_home	-0.5436	0.5807	versus category entertainment
17	category_travel	-0.7892	0.4542	versus category entertainment

What the result does and does not support. This run demonstrates a reproducible screening workflow and a correctly separated evaluation. The source resembles simulation data, so strong benchmark results may reflect generator rules. Label delays, disputes, recovery, real fraud prevalence, customer behaviour, and geographic context would all require independent verification. We did not estimate the effect of preventing fraud, benchmark production latency, or evaluate entirely new customers. The coefficient table and calibration plot are descriptive, not formal causal or inferential claims. Excluding gender and names does not prove fairness: location, time, and category can still act as proxies. A live project needs relevant subgroup and outcome audits on authorised data.

12. Business, government, auditing, and astronomy

These are proposed applications of the workflow. No organisation's deployment or return on investment has been measured in this project.

Audience	Decision this method could support	Value to measure	Additional evidence needed
Bank or payment provider	Rank transactions for an analyst review queue	Fraud value identified, review precision, workload, customer friction	Authorised transaction history, delayed confirmed labels, actual loss/recovery and investigation costs
Online retailer	Prioritise suspicious orders before fulfilment	Chargebacks, fulfilment losses, false holds, conversion impact	Order and fulfilment features, chargeback labels, a prospective controlled pilot
Government programme	Triage anomalous payments or claims for investigation	Confirmed irregularities per review and costs of missed cases	Programme-specific transactions, eligibility rules, verified outcomes, access controls, appeal process
Auditor	Prioritise tests while preserving a record of the selection rule	Yield of confirmed findings and coverage of the population	Evidence trails, independently checked findings, complementary random sampling, professional judgement
Astronomer	Prioritise rare candidate events for telescope follow-up	Confirmed-target yield, rare-event recall, observing time used	Labelled light curves or spectra, detection quality, object and time separation, independent observing-season evaluation

Business: turn a metric into a decision

A useful presentation to a fraud manager states how many transactions need review, how many labelled frauds enter that queue, what remains missed, and whether staff can handle the volume. This notebook calculates those quantities. Its cost inputs must be replaced with operational evidence before proposing a budget or savings claim. An initial pilot would run in shadow mode, collect investigation outcomes, and compare with existing rules without automatically declining customers.

Sensitivity analysis should vary missed-loss costs, review time, false-alert friction, and the capacity limit on selection data or a new period. Do not optimise those assumptions to make the final test look profitable. Source amounts should not be relabeled as rand without a verified currency basis and conversion.

Government: protect public funds through investigation support

GAO's Fraud Risk Management Framework emphasises prevention, detection, response, and continued monitoring and feedback. The proposed connection here is to use a transparent score as one input to payment or claim investigation, with reviewers checking evidence. Card-fraud labels do not establish benefit fraud, procurement misconduct, tax evasion, or corruption. Those are different targets and require new programme-specific datasets and validation.

For a South African application, local transaction populations, confirmed findings, institutional procedures, and applicable data-governance requirements would need to be established. This benchmark cannot estimate the country's fraud rate or fiscal losses. Broad national claims cannot be inferred from this transaction sample.

Auditing: make every selection reviewable

The hash, quality counts, split dates, fitted preprocessing, candidate search, threshold assumptions, and model card help a reviewer trace the analysis. A flagged transaction is a lead for checking, not evidence of misconduct by itself. Keep a complementary random or independently designed sample so risk-based selection does not hide low-scored failures or distort estimates of population error rates. Log investigation outcomes separately and incorporate feedback using future evaluation.

Astronomy: reuse the design, rebuild the model

Rubin Observatory's alert-stream explanation describes brokers adding contextual information and machine-learning classifications to help scientists prioritise follow-up observations. The inference from that source is that this project's ranking, rare-event evaluation, and capacity-aware decision design could be adapted to a follow-up queue. The card-trained coefficients have no astronomical meaning and cannot score stars or transients.

An astronomy version would define a target such as a confirmed transient class, build physical features from light curves and measurement uncertainties, group by astronomical object, and hold out future observing periods. Precision controls wasted telescope time; recall describes missed targets. Telescope time and missed-science costs replace financial losses. Selection bias, incomplete confirmation labels, and survey sensitivity must be assessed. This notebook implements none of those domain tasks; it illustrates the transferable evaluation logic.

13. Model card and exports

The model card documents the evaluated candidate, population, assumptions, and limitations. Exports are aggregates and settings; they contain no customer IDs or transaction-level prediction file. The complete source code remains in the notebook.

Cell 84 · Assemble the model card

Record the frozen decision rule alongside results and limitations, rather than publishing a score without its population.

# Assemble the model card
model_card = {
    'project': 'Credit-card fraud screening teaching portfolio', 'author': 'Gareth Grant',
    'source_url': 'https://www.kaggle.com/datasets/priyamchoksi/credit-card-transactions-dataset',
    'source_dataset_version': 1, 'source_sha256': source_sha256,
    'provenance': 'Public educational benchmark; real-bank origin unverified; schema resembles simulation data',
    'selected_implementation': selected_name, 'threshold': selected_threshold,
    'scratch_l2': scratch_l2, 'library_l2': library_l2,
    'train_fit_rows': len(train), 'train_fit_frauds': int(y_train.sum()),
    'features': list(feature_names), 'category_reference': str(reference_category),
    'temporal_protocol': split_summary.reset_index().astype(str).to_dict(orient='records'),
    'cost_assumptions_mu': {'review': REVIEW_COST_MU, 'false_alert_friction': FALSE_ALERT_FRICTION_MU, 'missed_fraud': MISSED_FRAUD_COST_MU},
    'validation_capacity_fraction': MAX_POLICY_ALERT_RATE,
    'final_test_metrics': {key: float(value) for key, value in selected_metrics.items()},
    'bootstrap': {'method': 'Customer-cluster percentile', 'repeats_requested': BOOTSTRAP_REPEATS,
                  'usable_repeats': len(bootstrap_samples), 'clusters': bootstrap_clusters,
                  'intervals': uncertainty_intervals.to_dict(orient='index')},
    'data_quality': data_quality, 'environment': environment,
    'intended_use': 'Learning and portfolio review; investigation-support prototype design',
    'limitations': ['No verified real-bank or astronomical deployment', 'No causal savings estimate',
                    'No new-customer performance guarantee', 'Constant illustrative loss assumption',
                    'No prospective operational, fairness, or latency validation'],
}

Cell 85 · Display a compact model card

These fields give a reviewer the key evidence boundary at a glance.

# Display a compact model card
display(pd.Series({
    'Use': model_card['intended_use'], 'Candidate': selected_name,
    'Threshold': f'{selected_threshold:.6f}', 'Training fit rows': f'{len(train):,}',
    'Final test rows': f'{len(test):,}', 'Provenance': model_card['provenance'],
    'Evaluation': 'Future transactions; repeat customers may occur',
    'Cost basis': 'Illustrative MU inputs, not observed savings',
    'Uncertainty': 'Conditional customer-cluster bootstrap within final period',
}, name='Model card').to_frame())

	Model card
Use	Learning and portfolio review; investigation-s...
Candidate	Scikit-learn
Threshold	0.050788
Training fit rows	150,000
Final test rows	194,502
Provenance	Public educational benchmark; real-bank origin...
Evaluation	Future transactions; repeat customers may occur
Cost basis	Illustrative MU inputs, not observed savings
Uncertainty	Conditional customer-cluster bootstrap within ...

Cell 86 · Export aggregate evidence

Setting WRITE_EXPORTS to False leaves the analysis in memory. CSV and JSON outputs are easy to inspect without rerunning the model.

# Export aggregate evidence
if WRITE_EXPORTS:
    EXPORT_DIR.mkdir(parents=True, exist_ok=True)
    test_metrics.to_csv(EXPORT_DIR / 'test_metrics.csv')
    split_summary.to_csv(EXPORT_DIR / 'split_summary.csv')
    policy_choices.to_csv(EXPORT_DIR / 'policy_choices.csv')
    coefficients.to_csv(EXPORT_DIR / 'coefficients.csv', index=False)
    uncertainty_intervals.to_csv(EXPORT_DIR / 'uncertainty_intervals.csv')
    scratch_selection.to_csv(EXPORT_DIR / 'scratch_selection.csv', index=False)
    library_selection.to_csv(EXPORT_DIR / 'library_selection.csv', index=False)
    (EXPORT_DIR / 'model_card.json').write_text(json.dumps(model_card, indent=2, allow_nan=False), encoding='utf-8')
    print('Saved aggregate evidence:', ', '.join(sorted(path.name for path in EXPORT_DIR.iterdir() if path.is_file())))

Saved aggregate evidence: coefficients.csv, library_selection.csv, model_card.json, policy_choices.csv, portfolio_summary.md, scratch_selection.csv, split_summary.csv, test_metrics.csv, uncertainty_intervals.csv

Cell 87 · Create a portfolio summary from this run

A short website introduction should communicate the work and its limits without exaggerating impact.

# Create a portfolio summary from this run
portfolio_summary = (
    'I built an interpretable fraud-screening workflow with logistic regression implemented '
    'from first principles and checked against scikit-learn. The project separates training, '
    'model selection, review-policy selection, and future testing; preserves rare-event prevalence; '
    'and reports alert workload, missed fraud, probability errors, and customer-cluster uncertainty. '
    f'On the held-out benchmark period, the frozen {selected_name} policy achieved '
    f'{selected_metrics.recall:.1%} recall and {selected_metrics.precision:.1%} alert precision '
    f'while flagging {selected_metrics.alert_rate:.2%} of transactions. '
    'These are educational benchmark results; the cost scenario is illustrative and no live savings are claimed.'
)
display(Markdown(portfolio_summary))
if WRITE_EXPORTS:
    (EXPORT_DIR / 'portfolio_summary.md').write_text(portfolio_summary + '\n', encoding='utf-8')

I built an interpretable fraud-screening workflow with logistic regression implemented from first principles and checked against scikit-learn. The project separates training, model selection, review-policy selection, and future testing; preserves rare-event prevalence; and reports alert workload, missed fraud, probability errors, and customer-cluster uncertainty. On the held-out benchmark period, the frozen Scikit-learn policy achieved 67.0% recall and 22.1% alert precision while flagging 1.76% of transactions. These are educational benchmark results; the cost scenario is illustrative and no live savings are claimed.

14. Exercises and references

Keep each extension in new, clearly titled cells and give it a new confirmatory future test if it changes the model or decision policy.

1. Explain why the no-alert baseline has high accuracy and zero fraud recall.
2. Set a different training cap and assess optimisation time on training data.
3. Examine whether an amount-by-category interaction improves model-validation AP.
4. Compare a nonlinear challenger using exactly the same temporal protocol.
5. Collect operational cost inputs and study policy sensitivity on selection data.
6. Add a customer-disjoint evaluation for a new-customer use case.
7. Use rolling future evaluation windows to study seasonal stability.
8. Design an astronomy project using object-grouped light-curve data; start with a domain-specific target definition rather than transferring card coefficients.

Sources and technical documentation

Source links were checked on 5 October 2026. They support methodology and application context; they do not validate this project's performance or impact claims.

Source	What it supports
Kaggle: Credit Card Transactions Dataset	Public dataset named in the original notebook; verify extracted rows and lineage
Sparkov data generator: original repository	Simulation-family context; resemblance does not establish this CSV's exact lineage
scikit-learn: common pitfalls	Split before fitting learned transformations; consistent preprocessing
scikit-learn: LogisticRegression	Library classifier, regularisation, sparse input, solver settings
scikit-learn: average precision	AP from scores and the distinction from trapezoidal PR area
scikit-learn: decision thresholds	Separate score estimation from decisions; tune thresholds on independent selection rows
GAO: Fraud Risk Management Framework	Government prevention, detection, response, monitoring, and feedback context
Rubin Observatory: alert stream	Broker classification and follow-up prioritisation as an astronomy adaptation context

Publishing the project

Use the executed HTML as a readable page or embed it in the portfolio. Keep the benchmark provenance statement near the headline results, and link the notebook for readers who want to run the code. Publish only aggregate results. The downloadable bundle includes setup instructions and a source-data download helper; the large CSV is not bundled. Neither the notebook nor HTML installs a website comments service.

© Gareth Grant · Data analytics portfolioExplore another project →

Code copied to clipboard