BAF Fraud Modeling

Rob Wiederstein

February 23, 2026

Introduction

Bank Account Fraud Dataset

• Synthetic online account applications
  
• 1M rows (Base)
  
• 8 months (0–7)
  
• Base + 5 biased variants
  
• Label: Fraud vs Legit
• Fraud \(\approx 1\%\)

What it is (plain English): each row is a bank account opening application submitted online. Fraudsters may impersonate someone (identity theft) or invent a person; once approved they quickly exploit the credit line or use the account to move illicit funds.

Why it exists: the BAF suite was created as a large, realistic benchmark to stress-test ML performance and fairness under dynamic / drifting conditions and “extreme” class imbalance. The variants introduce controlled bias patterns; the Base set has no induced bias.

How it was made: the released data are synthetic (generated from a CTGAN trained on an anonymized, feature-engineered real dataset). Privacy protections mean no row corresponds to a real identifiable person.

Time structure: month ranges 0–7 (eight months). This is why we use chronological evaluation (train early months, test late months).

Target variable: datasheet label is fraud_bool (0/1). In our pipeline we rename/recode to outcome with labels “Legit” and “Fraud” for readability.

Typical Scenario

Fraudsters will

1. Impersonate someone or

2. Create fake identity then

3. Max out the line or

4. receive illicit payment

Data Cleaning

• Relabel outcome.
• -1 → NA.
• Negative amount → NA.
• Write clean Parquet.

Outcome - fraud_bool (0/1) → outcome (“Legit”/“Fraud”); drop fraud_bool.

Missing encoded as values - Recode -1 to NA for: - prev_address_months_count - current_address_months_count - bank_months_count - session_length_in_minutes - device_distinct_emails (your data uses device_distinct_emails_8w; function handles either name)

Range constraint - intended_balcon_amount < 0 → NA (negative values are missing-encoding).

Output - Saved cleaned dataset as Parquet under 03_primary/variant=Base/ partitioned by month.

Explore

Variable Importance

Figure 1: Top 15 features driving the diagnostic model.

The diagnostic LightGBM model shows that behavior and identity structure dominate the early splits.

Feature Interaction

Figure 2: Interaction between Credit Risk Score and Address History.

Fraud clusters noticeably in high credit risk profiles combined with specific address tenure patterns.

Missingness Signal

Figure 3: Missingness rates by outcome.

Fraudsters are systematically omitting key tenure details (like previous address and bank history) compared to legitimate applicants.

Numeric Correlation

Figure 4: Core numeric correlation matrix.

The structural anchor of the synthetic data is visible here, particularly the relationship between credit score and proposed limit.

LightGBM

About

• Originally released in 2016
• Maintained by Microsoft
• Over 18,000 stars on GitHub
• King of Kaggle for tabular data
• Announcing paper over 23,000 citations
• Sped up similar gradient boosting algorithms 20x

Academic Support

• Standard
• Accurate
• Efficient

For tabular supervised learning, gradient boosted decision trees—most notably XGBoost and LightGBM—are strong, low-latency baselines because they exploit hand-engineered behavioral features; LightGBM remains a standard reference point for card and e-commerce fraud tasks [1]

[W]e found that the LightGBM approach had the highest detection accuracy of fraudulent activity with 97% in the experiments conducted. An additional key objective of reducing false alerts was accomplished, as the number of false alarms went from 13,024 to 6,249[2]

[W]e choose LightGBM as the base machine learning model due to its efficiency and widespread use in handling large-scale and structured datasets, particularly in financial domains such as credit card fraud detection.[3]

Unbalanced Classes

The Challenge

The scarce occurrences of rare events impair the detection task …

Citation: Guo, H., Li, Y., Shang, J., Gu, M., Huang, Y., & Gong, B. (2017). Learning from class-imbalanced data: Review of methods and applications. Expert Systems with Applications, 73, 220–239. https://doi.org/10.1016/j.eswa.2016.12.035

Bank Fraud Prevalence

Figure 5: Fraudulent versus legitimate applications by month.

Fraud represents approximately one percent of applications.

Fraud Prevalence

Table 1: Something

Month	Fraud	Legit	Total	% Fraud
0	1,500	130,940	132,440	1.13
1	1,198	126,422	127,620	0.94
2	1,198	135,781	136,979	0.87
3	1,392	149,544	150,936	0.92
4	1,452	126,239	127,691	1.14
5	1,411	117,912	119,323	1.18
6	1,450	106,718	108,168	1.34
7	1,428	95,415	96,843	1.47

Methods Tested

• Standard: Baseline (No sampling).
• Weighted: Cost-sensitive learning (\(4\times\) penalty).
• Undersampling: Random removal of majority class.
• SMOTE: Synthetic Minority Over-sampling Technique.
• ADASYN: Adaptive Synthetic Sampling (hard examples).
• Tomek Links: Cleaning boundary ambiguity.

Standard: The control group. We let the gradient booster handle the 1% imbalance naturally.

Weighted: We used scale_pos_weight to tell LightGBM that missing a Fraud case is 4x worse than a false alarm.

Undersampling: We threw away about 75% of the Legit cases to balance the ratio. Fast, but risky.

SMOTE & ADASYN: The “heavy hitters.” These generate fake fraud data based on nearest neighbors. Adasyn focuses specifically on “hard to learn” fraud cases.

Tomek: A cleaning method that removes Legit cases that are “too close” to Fraud cases, theoretically making the decision boundary clearer.

Strategy Showdown: Results

Table 2: Performance comparison across imbalance strategies using 3-month rolling windows.

Class Imbalance Strategy Showdown
Paired t-test comparison against 'Standard' baseline
recipe	avg_pr_auc	avg_runtime	p_val_vs_std	significance
Smote	0.1635	3.9054546	0.9032	No (ns)
Standard	0.1631	2.3006599	1.0000	-
Adasyn	0.1627	3.7494911	0.8361	No (ns)
Weighted	0.1614	2.2227480	0.3971	No (ns)
Tomek	0.1497	2.5417861	0.0501	No (ns)
Under	0.1403	0.9416666	0.0505	No (ns)

The “Standard” baseline is statistically indistinguishable from more complex methods like SMOTE and Adasyn (p > 0.05). Complex sampling provides no significant predictive gain for this dataset.

Sampling Compared

Figure 6: PR-AUC performance versus computational training time.

The Standard strategy represents the “Efficient Frontier.” It achieves near-peak performance while being nearly twice as fast as SMOTE or Adasyn. Tomek sampling actually degraded performance while increasing compute time.

Sampling Methods Discarded

• No statistical gain

• Resource intensive

• Scalability

Complex sampling methods like SMOTE and Adasyn do not outperform the baseline “Standard” model, as shown by their non-significant p-values (p > 0.05).

Synthetic generation and neighbor calculations nearly double the average training time per fold compared to the standard approach.

For larger file sizes, simplicity helps avoid memory bottlenecks and excessive compute costs.

Future model performance gains should focus in places other than sampling techniques.

Feature Creation

Final Results

The Confusion Matrix

Figure 7

The confusion matrix is the foundation of all classification metrics. Every metric is derived from these four cells.

In the fraud context: - TN: Legitimate application correctly approved. No harm done. - FP: Legitimate application flagged as fraud. Customer friction, potential churn. - FN: Fraud case missed. Direct financial loss — the costliest error. - TP: Fraud correctly caught. The goal.

The key insight: not all errors are equal. A missed fraud case (FN) costs far more than a false alarm (FP). Our threshold and metric choices reflect this asymmetry.

Precision & Recall

\[\text{Recall} = \frac{TP}{TP + FN}\]

Of all actual frauds, how many did we catch?

\[\text{Precision} = \frac{TP}{TP + FP}\]

Of all flagged cases, how many were real fraud?

Recall is the primary metric for fraud detection. Missing a fraud case (FN) is costly, so we want Recall as high as possible. A model that flags every application gets perfect Recall — but at the cost of Precision.

Precision captures that cost: if we flag everything, every legitimate customer gets rejected. Precision measures how trustworthy our fraud flags actually are.

The Precision-Recall tradeoff is the core tension in fraud modeling. Lowering the decision threshold increases Recall (catch more fraud) but decreases Precision (more false alarms). The right balance depends on the operational cost of each error type.

Our model targets ~49% Recall at a 5% False Positive Rate — a deliberate operating point chosen to limit customer friction while catching nearly half of fraud.

ROC vs Precision-Recall AUC

• ROC AUC
• PR AUC

• Plots Recall vs False Positive Rate
• AUC = 0.5 is random; 1.0 is perfect
• Optimistic under class imbalance
• Inflated by the large TN pool

• Plots Precision vs Recall
• Focuses entirely on the minority class
• Harder to game with a large Legit majority
• Preferred metric for fraud detection

Why ROC AUC can mislead on imbalanced data: with 99% legitimate applications, even a naive model achieves a low False Positive Rate simply because the TN pool is enormous. ROC AUC rewards this, making models look better than they are.

PR AUC ignores true negatives entirely. It only asks: of the positive (fraud) predictions, how precise were we, and how much fraud did we recall? This makes it a far more honest scoreboard when positives are rare.

Rule of thumb: use ROC AUC for balanced classes; use PR AUC for imbalanced fraud/anomaly detection tasks. We report both, but optimise for PR AUC.

Final Model Evaluation

Figure 8: Confusion Matrix Heatmap (5% Decision Threshold)

Diagnostic Metrics

Figure 9: ROC and Precision-Recall Curves for Out-of-Sample Data

References

[1]

G. Aminian et al., “FraudTransformer: Time-Aware GPT for Transaction Fraud Detection.” arXiv, Oct. 2025. doi: 10.48550/arXiv.2509.23712.

[2]

C. Iscan, O. Kumas, F. P. Akbulut, and A. Akbulut, “Wallet-Based Transaction Fraud Prevention Through LightGBM With the Focus on Minimizing False Alarms,” IEEE Access, vol. 11, pp. 131465–131474, 2023, doi: 10.1109/ACCESS.2023.3321666.

[3]

X. Zhao, Y. Liu, and Q. Zhao, “Improved LightGBM for Extremely Imbalanced Data and Application to Credit Card Fraud Detection,” IEEE Access, vol. 12, pp. 159316–159335, 2024, doi: 10.1109/ACCESS.2024.3487212.