Machine learning · Python
A fraud detector designed around the cost of being wrong.
How do you detect a rare event without creating an expensive stream of false alarms?
01 / The problem
Rare-event accuracy was the wrong target.
Fraud made up less than 0.2% of the transaction data. That imbalance made overall accuracy nearly useless. The useful question was whether the model could catch fraud while keeping manual review manageable.
A model can look excellent by predicting every transaction as legitimate. I needed evaluation that reflected the cost of missed fraud and the operational cost of false positives.
02 / The approach
Evaluate the model around the review decision.
- 01
Cleaned and standardized the anonymized transaction features.
- 02
Used five-fold cross-validation with F1 as the primary metric.
- 03
Trained a Random Forest and reviewed precision, recall, ROC-AUC, and the confusion matrix together.
- 04
Compared decision thresholds based on the review burden they would create.
03 / The result
The threshold sets the workload.
The model score was only the start. The threshold determines how risk policy becomes day-to-day review work, so it should be chosen with the people handling the alerts.