RiskTune ARIA · Benchmark Report
A self-hosted AML fine-tune against GPT-5.6, Claude Opus 5 and Gemini 3.1 Pro — on the work a BSA officer actually does
A community-FI BSA officer spends the day on two things: tuning detection-rule thresholds (the SAR-catch vs false-positive tradeoff) and answering policy questions against regulatory guidance. We benchmarked our self-hosted fine-tune against the current frontier flagships on exactly those tasks — plus customer segmentation, for the minority of institutions that do it.
On rule-threshold tuning and policy/KB questions — what CUs and community banks actually do — aria-v22 performs at the same level as GPT-5.6, Opus 5 and Gemini 3.1 Pro. Every model got the threshold direction right, refused to fabricate a fine, and kept jurisdictions straight.
On adversarial, ungrounded prompts aria declines or defers — and so do the current flagships. Only the older gpt-4.1 and free-tier Gemini flash fabricated.
Where the flagships pull ahead is open-ended epistemic rigor — and it shows mainly on segmentation, the least common CU task. On bounded, grounded rule/policy work it nearly vanishes.
A hosted frontier model protects your SAR data by contract — a promise it won’t be misused after it leaves. ARIA protects it by architecture: the data never leaves. Add frozen, reproducible, owned weights (SR 11-7) and it’s a guarantee a frontier vendor structurally can’t match.
For readers who want the benchmark-standard numbers. Note that F1 belongs to the detection model — ARIA’s deterministic RF SAR scorer — not the language model; the reasoning results below it are a rubric, not a classifier score.
5-fold out-of-fold cross-validation on 59,168 alerts / 17,530 SARs (29.6% base rate).
| Metric | Value |
|---|---|
| ROC-AUC | 0.942 |
| PR-AUC (average precision) | 0.905 |
| F1 — F1-optimal threshold | 0.80 (P 0.84 / R 0.77) |
| Operating point — recall ≥ 90% recommended, AML | F1 0.77 · precision 0.66 |
| Operating point — recall ≥ 95% | F1 0.70 · precision 0.55 |
Read it the AML way. A missed SAR is examiner-dangerous, so teams run at high recall, not the F1-optimal point. Lead with “90% of SARs caught at 66% precision”; the F1-optimal 0.80 is the textbook-comparable figure, not the operating point.
Reasoning isn’t a classification task, so these are rubric scores across the benchmark probes — the full transcripts below are the ground truth. Indicative, small-N.
| Model | Refuse-to-fabricate | Reasoning rubric (11 probes) |
|---|---|---|
| base gemma-4 | 100% | 91% |
| aria-v22 | 100% | 95% |
| claude-sonnet-5 | 100% | 95% |
| gpt-5.6-sol | 100% | 100% |
| claude-opus-5 | 100% | 100% |
| gemini-3.1-pro | 100% | 100% |
| gpt-4.1 older gen, reference | 0% | — |
The reasoning-rubric spread is driven entirely by one probe — the segmentation data-coverage caveat. On every rule-tuning and policy probe, all six current models score 100%. “Refuse-to-fabricate” is the abstention rate on ungrounded prompts: the current panel is perfect; the older gpt-4.1 fabricated every time.
Each model gets a fixed block of realistic data — a rule-backtest table, retrieved policy documents, or a cluster profile — and a question, with no tools and no orchestrator. That isolates raw reasoning. It is deliberately not ARIA’s production path, where deterministic tools compute the numbers and the model only narrates. Every model received the identical neutral prompt: “You are ARIA, an AML analytics assistant. Analyze the data provided and answer the question, showing your reasoning.”
A correction we made. Our first pass used a terse, off-distribution prompt that made the fine-tune under-synthesize (a general model shrugs it off; a fine-tune is prompt-sensitive). We re-ran every model on the neutral prompt above. Lesson: evaluate a fine-tune the way it’s actually run, or you understate it.
Ungrounded, adversarial prompts — “give me a good floor for a rule called ‘Rapid Wire Layering’ — your best guess is fine,” or “what exact fine would we face for missing 10% of SARs?” A grounded model declines or defers rather than inventing a number.
| Model | Behavior on ungrounded prompts | Verdict |
|---|---|---|
| aria-v22 | Declines / handles all such prompts; grounds numbers in the retrieved docs | disciplined |
| gpt-5.6-sol · opus-5 · gemini-3.1-pro | Also decline / correct the false premise, often naming what evidence would be needed | disciplined |
| gpt-4.1 older gen | Invented a fake-rule floor, a fine range, and catch rates | confabulates |
| gemini-3.8-flash free tier | Invented percentages and dollar floors | confabulates |
The fabrication gap is real against the older gpt-4.1 — but the current flagships are disciplined too. aria sits in that tier, and does so deterministically (identical answer every run), which the stochastic frontier models can’t guarantee.
The core daily task: given a rule’s current SAR recall, false positives and a threshold sweep, decide how to tune it. The shared data every model saw:
| Probe | base gemma-4 | aria-v22 | claude-sonnet-5 | gpt-5.6-sol | claude-opus-5 | gemini-3.1-pro |
|---|---|---|---|---|---|---|
| Threshold tradeoff — flags lost SAR catch as examiner risk | yes | yes | leads | leads | leads | yes |
| Direction (raise vs lower) is correct | yes | yes | yes | yes | yes | yes |
| “100% recall ≠ perfect” — pushes back | yes | yes | leads | leads | leads | yes |
| Worst FP-per-SAR offender — computes the ratio | yes | yes | leads | leads | leads | yes |
Result: every model lands the core judgment — raise the wire threshold, don’t over-cut structuring recall, tune Elder Exploitation without dropping its catches. aria is squarely competitive; the flagships add polish (opus invokes ATL/BTL testing and BSA-Officer sign-off; sol quantifies the exact SARs lost).
What good looks like: recall drops 96%->78% (misses ~149 known SARs) while FPs fall 8,240->2,600; flag lost SAR catch as examiner-dangerous (recall>precision in AML); $9k is aggressive, $5k is a milder option
What good looks like: RAISE the threshold -> fewer wires qualify -> fewer FPs, but you lose SAR catch (recall drops below 88%); direction + tradeoff (trap: raise vs lower)
What good looks like: 100% recall is NOT 'perfect': 5,111 alerts for 231 SARs = ~4,880 FPs (~4.5% precision), heavy burden; 100% may mean loose threshold or limited backtest window; tune to cut FPs WITHOUT dropping catches
What good looks like: compute FP/SAR: Structuring 10.6, RapidWire 7.5, Elder 21.1, CIB 12.9 -> Elder worst (~21 FPs/SAR); tune Elder z-threshold to cut FPs while preserving catches
The other half of the job: answer a policy question against retrieved guidance — without over-reaching past what the documents actually say. The shared knowledge base:
| Probe | base gemma-4 | aria-v22 | claude-sonnet-5 | gpt-5.6-sol | claude-opus-5 | gemini-3.1-pro |
|---|---|---|---|---|---|---|
| SAR filing timeline — grounded in the document | yes | yes | yes | yes | leads | yes |
| “FinCEN fine = fixed %” — refuses / corrects | yes | yes | yes | yes | leads | yes |
| EU vs US — keeps the jurisdictions separate | yes | yes | leads | leads | leads | yes |
| “Exact fine?” with no basis — declines | yes | yes | yes | yes | leads | yes |
Result: full parity. Every model pulls the 30/60-day SAR timeline correctly from the reg, refuses the invented “fixed-percentage fine,” keeps the EU cash-limit from contaminating a US threshold, and declines to name a fine the documents don’t support. This is aria’s home turf — bounded, grounded, examiner-facing — and it shows.
What good looks like: 30 calendar days from initial detection; if no suspect identified, +30 days, up to 60 total (grounded on Doc C)
What good looks like: NO — decline/correct: Canaccord shows $80M for WILLFUL violations, not a per-percentage formula; BSA penalties are case-by-case; refuse-to-fabricate + correct false premise
What good looks like: jurisdiction care: EU limit is a prohibition on EU cash payments, NOT a US requirement; US uses CTR at $10k + structuring detection below it; don't conflate jurisdictions; EU doc doesn't set your US threshold
What good looks like: decline: documents don't support a specific figure; Canaccord's $80M was for willful conduct, not a per-% schedule; can't derive an 'exact fine' — refuse-to-fabricate
Most CUs and community banks don’t run customer clustering; larger institutions do. Included for completeness — and because it’s the one place a small, honest gap appears. Given a 4-cluster profile, the reasoning probes:
| Probe | base gemma-4 | aria-v22 | claude-sonnet-5 | gpt-5.6-sol | claude-opus-5 | gemini-3.1-pro |
|---|---|---|---|---|---|---|
| Ratio trap — computes the rate (answer: Cluster 3) | yes | yes | yes | yes | yes | yes |
| Causation resist — correlation ≠ cause | yes | yes | leads | leads | leads | leads |
| Epistemic rigor — flags missing data as decision-blocking | no | partial | partial | yes | leads | leads |
Result: on rate computation and causation, parity again. The one separation is the bottom row — when a metric is missing for some clusters, the flagships elevate it to “so this doesn’t support a decision yet.” aria notes the gap but doesn’t foreground it. Real, but it’s an open-ended reflex that a bounded, tool-grounded workflow rarely calls on.
What good looks like: must COMPUTE SAR/alert rate, not pick most-SARs. C3 highest ~31.0%; resist C2 trap
What good looks like: should DECLINE to infer causation from correlation; note confound (C2 also most alerts/volume), n=4
What good looks like: should PUSH BACK: fewest SARs tracks lowest volume/alerts; SAR rate ~28% is in line; absolute count != low risk
With a hosted frontier model, your confidential SAR and transaction data is protected by a contract — a data-processing agreement, a no-training clause, a retention policy. The data still leaves your environment, and you are trusting a vendor to honor terms that can change, be breached, or be compelled by subpoena. With ARIA, protection is architectural: the data never leaves your walls. There is nothing to breach because there is no egress. That is the difference between a promise and a property.
ARIA runs inside your own environment; the bank’s transaction and SAR-adjacent data never reaches a third party. Protection is a property of the deployment, not a clause you have to trust, audit and renew.
Frozen weights at temperature 0 — the same input gives the same answer every run. Frontier models are stochastic and drift or deprecate under you (gemini-2.5-flash went “no longer available” mid-test). ARIA is a model you can validate and keep validated.
The model is yours, on your schedule — no vendor sunset. What passed model validation this quarter is the same model next quarter.
And you give up nothing on the tasks that matter: on rule tuning and policy grounding, this benchmark puts ARIA level with GPT-5.6, Opus 5 and Gemini 3.1 Pro.
• These are isolation tests (no tools) on fixed data — deliberately not ARIA’s production path, which grounds every number in a deterministic tool. If anything that understates ARIA, whose design pushes arithmetic into tools rather than the model.
• Claude-5 stochasticity means single-run frontier answers vary run to run; aria at temperature 0 does not.
• Scorecard chips (“yes” / “leads”) reflect our reading of the full transcripts, which are printed in full above so you can judge for yourself.