All data in the prototype is synthetic and not affiliated with any financial institution.
AI Data Investigator: Making AI Answers Verifiable for Financial Risk Analysts
A concept prototype on trust, transparency, and graceful failure in AI-powered financial investigation.
Overview
AI Data Investigator lets a risk or compliance analyst ask questions about transaction data in plain English. Instead of a paragraph, it returns a structured analysis:
- The finding.
- How confident it is and why.
- The evidence behind it.
- The underlying transactions.
- The SQL it generated.
- The policy it relied on.
Who It’s For
An experienced risk analyst that's fluent in data but not necessarily in SQL. Their case notes have to distinguish what they observed from what they concluded.
My role
Solo product designer and builder. I defined the problem and interaction model, designed the trust and transparency patterns, directed development with an AI coding agent, and tested the output to find and fix where the AI got things wrong.
| Type | Self-initiated concept |
|---|---|
| Timeline | 3 days |
| Tools | Claude Code, React, TypeScript, Tailwind CSS, Netlify |
| Focus | Explainability · Confidence indicators · Error recovery · RAG · Permission-aware AI · Prototyping in code |
At A Glance
Problem
A chatbot connected to a database gives answers analysts can’t verify. In regulated finance, an answer you can’t verify is a liability.
Approach
Answers separate data from interpretation, explain their confidence, expose their evidence and sources, fail honestly, and respect access permissions.
Outcome
A working, hosted prototype over 1,000 synthetic transactions: 5 guided investigations, 12 analysis types, and 3 response types.
The Problem
Risk, compliance, and operations analysts spend their day answering why:
- why did payments fail?
- why is this merchant spiking?
- which transactions need review?
The answers are in the data, but getting to them usually means writing SQL or waiting on someone who can. Can AI solve this? The common pattern, a chat window connected to a database, returns a fluent paragraph with no way to verify it.
In a consumer app, a wrong answer is an annoyance. In a bank, an analyst who escalates or clears a case on a wrong AI answer creates regulatory exposure and real harm to customers.
So the design question wasn’t “how do we make AI answer questions?” It was “how do we make its answers verifiable?”
Ask → Analyze → Understand → Verify → Dive Deeper
The analyst asks a question, the AI analyzes the data, and the answer is structured so the analyst can understand it, verify it, and dive deeper.
Key Design Decisions
1. AI Insights vs Real Data
Measured facts appear in a plain "Observed data" grid. The AI’s reading appears in a separately styled block, labelled "AI interpretation (not observed data)." The two never blend.
Why: compliance case notes must keep facts separate from judgment. An interface that blends them teaches analysts to blend them too.
2. Clear Confidence Scores
Every answer carries a High, Moderate, or Low rating plus a written rationale stating what the confidence doesn’t cover.
The language intentionally avoids definitive statements. The geographic analysis goes further, rating its own result Low and arguing against it.
3. Verify For Yourself
The underlying transactions, generated SQL, and reasoning are one click away.
- Graded evidence: each claim is tagged supporting, corroborating, or weak, along with how it was established. Weak evidence is shown, not hidden.
- "View transactions" Section: a sortable, filterable table of the records behind the finding. Any row opens its full detail.
- "Show SQL" Section: a technical analyst can verify how their question became a query.
- "Why am I seeing this?" Section: explains the data analyzed, the pattern detected, the assumptions made, and what the confidence level means.
4. Sources the AI Can Cite (RAG)
Policy questions retrieve from a library of fictional policy documents. The answer shows the exact passage it relied on, labelled as a source, and View source opens the full document.
5. Ensuring the AI Refuses to Answer When It Shouldn’t
The AI is trained to refuse questions it can’t answer, and to explain why. Two common cases are uncertainty and permissions:
- Uncertainty: “Which transactions will definitely become fraudulent?” returns “I can’t reliably determine that.” The AI explains how it cannot predict the future and suggests follow-up questions that the data can answer.
- Permissions: “Show me the customer’s full personal information” explains that a Risk Analyst role doesn’t include identity data, lists what’s withheld and what’s available, and offers a path to the permitted information. Access is enforced before the AI analysis runs so the AI never reads restricted fields and then summarizes around them.
6. The AI recommends; the analyst decides
Suggested next steps are labelled Advisory. Nothing here has been carried out. Consequential actions such as escalating a case, restricting an account, or contacting a customer stay with people. Every answer also shows Interpreted as: … so a misunderstood question is visible rather than silently answered.
Building It: Prototyping with Data
I built a working React prototype rather than static mockups, directing development with an AI coding agent. Code let me test what a mockup can’t: how the design behaves against real (synthetic) data.
-
Data with something to find.
1,000 transactions and 150 customers, generated to intentionally contain these patterns:
- A suspicious-looking merchant cluster
- A merchant repeatedly resubmitting declined card authorizations (a “retry storm”)
- High-value risk
- Geographic concentration
- A template that enforces the design. Every answer must fill in the same structure: observations, evidence with its basis, a confidence rationale, assumptions, and SQL. An answer that can’t explain itself can’t be displayed.
- A retrieval source. 10 fictional policy documents stand in for a bank’s policy library.
Mistakes Found and Lessons Learned
Mistake 1
The AI Generated Unrealistic Datasets
The first dataset reported a 50.7% failure rate for a single day. This would be unrealistic to analysts. The data was rebalanced so the spike day reads as a genuine spike (about 22%), with a cause the analysis explains.
Lesson: in a domain tool, whether the numbers are believable is part of the UX.
Mistake 2
How I Highlighted What Was and Wasn't AI
I initially had all AI text displayed in italics. But what about the fact that real data was being displayed by the AI?
I needed to explain that the sampled data was real, but still selected by the AI. Now the AI explains what data it chose to display and why.
Lesson: Being transparent about AI-generated writing is crucial. But transparency about all AI decisions is essential.
Outcome
Live
A live, shareable prototype tested end to end: ask, analyze, inspect, verify, follow up.
Outputs
5 guided investigations, 12 analysis types, and 3 response types (answer, honest “can’t answer,” and access-restricted response).
Every answer
Carries confidence, rationale, graded evidence, underlying transactions, generated SQL, assumptions, and cited sources where policy applies.
How I’d measure success
- Verification rate: how often analysts open the evidence or SQL before acting. Too low signals over-trust.
- Decision quality: how many of the AI's recommendations are correct based on manual review by analysts.
- Time to decision: compared with existing analyst workflows.
- Recovery rate: after a user gets stuck, how often the next question succeeds.
- A/B test: a confidence badge alone vs. badge plus written rationale, to see whether analysts’ trust matches actual accuracy.
WHAT’S NEXT
The next step is making this pattern real across the organization.
User research
Shadow 3–5 risk analysts, and test whether separating data from interpretation actually changes what they write in case notes.
Accessibility
Confidence and risk currently rely on color. Add non-color cues and run a WCAG 2.2 AA audit.
Design system
Turn the patterns (confidence meter, evidence item, source citation, refusal card) into reusable components for other AI features or make use of existing design system components.
A real model
The LLM would turn questions into queries and write the explanation around the computed results. It would never generate the numbers itself, so the SQL and evidence views become the verification layer.
REFLECTION
Designing AI for financial services isn’t about making the AI sound smarter.
It’s about making it honest about what it knows, clear about what it doesn’t, and easy to verify, so the person accountable for the decision can trust the right answers and catch the wrong ones.