Fraud made up less than 0.2% of the transaction data. That one fact shaped the entire evaluation.
If a model predicted every transaction as legitimate, its accuracy would still look excellent. I used F1, precision, recall, the confusion matrix, and ROC-AUC because each exposed a different part of the model’s behaviour.
ROC-AUC answered a narrower question than I needed.
ROC-AUC measured whether the model ranked fraudulent transactions above legitimate ones across many possible thresholds. That was useful. It showed that the signal existed and that the model separated the classes well.
The fraud team would never operate across every threshold. It would choose one cutoff and live with the alerts created by that choice. A strong ROC-AUC could not decide which false positives were affordable or how much missed fraud the business would accept.
The threshold is where a model becomes an operating decision.
The confusion matrix made the workload visible.
I moved from the default 0.5 cutoff toward roughly 0.4. The lower threshold was intended to catch more fraud while accepting a manageable increase in manual reviews. That recommendation still needs real cost data before production use, but it made the trade clear.
The embarrassing result was the useful one.
An early version of the model produced a perfect result. The target column had accidentally remained in the feature set. The model was reading the answer.
Fixing that leakage brought the score back to reality and changed my process. I now inspect the exact training columns, test the pipeline on deliberately simple baselines, and treat surprising performance as something to explain before celebrating it.
What I would measure next
The next version needs a cost model: average fraud loss, cost per review, review capacity, and the value of catching a case sooner. That would let the threshold follow the economics of the system instead of a generic metric.
ROC-AUC was useful evidence. The decision still lived in the threshold, the confusion matrix, and the work created for the people using the model.