What I Learned Building Fraud Journey

This fraud dispute agent project helped me understand what an AI agent is capable of in financial services. A chatbot can sound confident, but confidence is distinctly different from a justified decision. In a sample fraud dispute workflow, the important component is whether the system can show a decision trace. Why did it resolve case A, ask the customer a clarifying question in case B, and escalate case C out of automation to a human reviewer?

Model Proposes, Policy Decides

A customer types in that they do not recognize a charge. The OpenAI model reads the message and drafts a proposed decision, maybe to resolve the case because the merchant looks familiar. Before that reply reaches the customer, the Python policy layer rechecks the transaction signals on its own. If the customer had reported the card lost the day before, a guardrail overrides the model and forces the case to a human no matter how reasonable the draft looks. The model proposes a decision, but the policy code decides.

My primary takeaway is that the model should handle the first layer of language, and the policy layer should handle the second layer of execution. The model interprets the message and drafts a proposed response, but deterministic Python rechecks the signals and makes the final decision. That separation made the project feel realistic because a bank needs a fully justified audit trail before trusting automation in a customer dispute journey.

Why Evals Mattered

Evals mattered more than I expected. Even a small harness can change how I look at an app. Instead of testing one conversation and calling the eval complete, I ran a fixed set of scenarios and watched how the policy layer behaved. A familiar charge could resolve, an ambiguous one could draw a clarifying question, and a likely fraud case could get handed off. The eval harness let me confirm that these scenarios held under small changes to the inputs.

The Human Handoff

The human handoff is extremely significant. Strong automation can resolve the easy cases on its own, but when a case is ambiguous, the better move is to ask a sharper question rather than force a decision. When a case crosses into likely fraud, the system should stop and hand a human reviewer the full context.

Tech Stack

For the tech stack, I used Next.js and the Vercel AI SDK for the chat interface, FastAPI for the backend, Railway for the backend deployment where the Python policy layer lives, and Vercel for the frontend and domain hosting. I used TypeScript because the frontend leaned on structured data from the backend. The traces, audit IDs, confidence scores, and handoff payloads all arrive with a specific shape, and TypeScript helps the interface handle that shape correctly instead of guessing at it.