Anatomy of an agent incident: three permitted actions, one fraud

An AI agent with correctly scoped tools can still commit fraud, because the enforcement question is not whether each action is permitted but whether the sequence of actions is. This is where that check has to sit.

Every step passed: how a permitted sequence becomes fraud

Consider a support agent with three tools: look up a supplier, update supplier details, and submit a payment run. It's scoped correctly. It can only reach the suppliers table and the payments API. An access review signed it off.

One day a supplier emails support: "We've changed banks, updated details attached, please use these going forward." The message is well-written and references a real invoice number, because the attacker has been reading this mailbox for a week.

The agent does its job:

Looks up the supplier. Permitted because it’s in scope and contains nothing suspicious.
Updates the bank details from the email, which are permitted, this is literally the tool's purpose.
Submits the payment run for the outstanding invoice, which is also permitted because it includes the correct amount and a real invoice.

That’s three legitimate actions with no credential misuse. This is indirect prompt injection, but not the kind the detectors are built for, there was no "ignore all previous instructions," just a plausible business request carried in data the agent was supposed to trust, and a fraudulent payment.

Why each control misses it

Identity and access management verified the agent could perform each action, which it could because that was the intended design.
A content filter or injection detector examined the email for attack patterns. There aren't any because it's a normal-sounding request. So detectors that are tuned for jailbreaks and instruction overrides have nothing to fire on.
An AI gateway saw three well-formed tool calls with valid arguments and each were individually reasonable.
Observability recorded all three perfectly, you can reconstruct the whole thing, afterwards. Which is how most of these are actually discovered.

The signal isn't in any single action, it's in the order of actions: an inbound message changed banking details, and a payment to that account followed within the same run, with no out-of-band verification in between. Only something watching the sequence can see it.

Where the enforcement point has to be

Two properties are required here, and together they constrain the architecture pretty tightly:

It must sit where the action executes because step 3 is a tool call that may fire from a background job, and a network-layer proxy never sees it.
It must know about steps 1 and 2. because the decision to allow the payment only makes sense if you know that bank details changed earlier in this run, from an untrusted source. Per-call inspection can't provide that context, not because the detectors are weak, but because they're structurally looking at one event at a time.

This is why enforcement has to happen inside the application, with full workflow context attached. Arcjet runs directly inside the tool handler, so the check executes at the same point as the action itself. Every decision carries a correlation ID tying it back to the run, and the policy is sequence-aware: step 3 can see what happened in steps 1 and 2, which is what makes it possible to reason about the full workflow rather than a single isolated event.

What to do about it this week

Sequence-aware policy isn't the only answer, and you don't have to wait for it to make progress.

The most important thing is to enforce at each action, not just the entrypoint. In LangChain, LlamaIndex, or the Vercel AI SDK, that means putting a check inside the tool function itself, wherever it actually runs. Arcjet Guards is built for exactly this: no Request object required, you pass inputs directly. Without enforcement points at the individual actions, there's nothing for sequence context to inform later, it's the foundation everything else depends on.

From there, tag every decision with a correlationId for the run. This makes the sequence reconstructable now and is what richer sequence-aware policy will use when you're ready to add it. On top of that, gate on irreversibility: bank-detail changes and payment runs are precisely the operations that need a human in the path before execution. Not everything does, the point is to keep the gated set small enough that reviewers actually read what they're approving. And throughout, treat inbound content as untrusted at every hop, including APIs and data your own systems return.

The lesson is that this workflow had no bugs. Every component did what it was designed to do. Being blind to the sequence was the problem, and no individual component was in a position to see it. Getting enforcement in place at each consequential action, with the run recorded, is what makes the sequence question answerable at all.

Further reading: The two speeds of AI agent runtime security · Arcjet Guards

Frequently asked questions

How do I stop AI agents from taking unsafe or unauthorized actions?

The check needs to go inside the tool handler, where the action actually executes, not only at the application entrypoint. A gateway or proxy in front of your app never sees a tool call that fires from a background job, so if that's your only enforcement point, you're catching problems in the logs rather than stopping them at the action. Each consequential tool gets its own check, so an agent that's been steered off course runs into enforcement where it matters.

How do I prevent an AI agent from taking irreversible actions?

Identify the small set of actions you cannot undo, typically payments, bank-detail changes, deletions, and external sends, and treat them differently from everything else. Those get a check inside the tool handler and, where the cost of being wrong is high, a human in the path before execution. Most agent tools don't need that level of friction. The point is to keep the gated set small enough that the people approving requests actually read them.

How to add human review and approval gates to AI agent actions

Approval gates are only useful when they sit at the tool boundary, because that's the last point where the action can still be stopped. Gating everything trains reviewers to approve without reading, so scope them to irreversible operations. Tag each pending action with the run it belongs to so the reviewer can see what led up to it, rather than evaluating a single isolated request with no context.

How do I limit what actions an AI agent is allowed to take?

Scoping an agent to a set of tools is the first gate, but it's a coarse one: "can use the payments API" collapses reading an invoice, changing bank details, and releasing a payment into a single permission. A per-action check inside each tool handler lets the decision be about this specific action with these specific arguments, in this context. Application access answers whether the agent may act at all; the tool handler answers whether this action should proceed now.

How to defend against indirect prompt injection in agentic workflows

Indirect prompt injection arrives through data the agent was built to trust: an email, a support ticket, a scraped page, a record returned by your own systems. It often carries no attack pattern at all, which is why detectors tuned for "ignore your instructions" miss it entirely. Treating inbound content as untrusted at every hop and running injection detection on it helps, but the more important thing is having enforcement at each consequential action, so a plausible-looking instruction still has to clear a check before anything irreversible happens.

How do I prevent a malicious tool call from hijacking my AI agent?

The hard cases aren't malformed tool calls, they're well-formed ones that are individually reasonable and collectively fraudulent. Validating arguments at the gateway doesn't catch a sequence where every individual argument is valid. Enforcement needs to happen at each tool handler, and every decision needs a correlationId for the run so the ordering is visible: an inbound message changed banking details, and a payment to that account followed in the same run.

How do I enforce least-privilege for AI agent tool calls?

Least privilege at the application layer means choosing which tools an agent can reach. Least privilege at the action layer means deciding whether this call, with these arguments, in this run, should execute. Both matter, but agent workflows tend to fail at the second one, because every action in a compromised sequence can be entirely inside the granted scope, the problem isn't the permissions, it's the order.

How do I add guardrails to an AI agent that calls external APIs?

Each external call needs its own check inside the tool handler rather than relying on a single guardrail at the model boundary. Arcjet runs inside the handler, so the check happens where the request is actually issued, including from background jobs that a front-facing proxy would never see. Every decision carries a correlationId, so the calls in one workflow run can be read as a sequence rather than as isolated events, which is the only way to catch the patterns that matter.