The agent graph behind our 1,250 email parsers

Written by Samuel Hsiung

Extra has more than 1,250 email parsers, small deterministic programs that turn a receipt, a shipping notice, or a flight confirmation into a card in your feed. We don't write them by hand, and we don't fix them by hand either. A graph of more than a dozen agents and deterministic steps finds broken parsers, fixes them, checks the fixes, and merges them. It has merged more than 6,600 fixes so far, and a typical one goes from opened PR to merged in about six minutes, with no person reviewing it.

Here's one path through it. A Brad's Deals headline read "Macy's: Scores Under $10," but our parser's price-stripping regex treated "Under" as clutter, so the card said "Macy's: Scores." A detector noticed, a fixer agent patched the regex against the original email and added a test, and a reviewer agent pointed out the same bug would still fire with a comma ("Macy's: Scores, Under $10"). The fixer handled that too, and the PR merged once CI passed.

Why keep parsers at all

Emails without a parser already go to an LLM. Doing that for every email would add cost and latency for every user, while a parser runs in milliseconds, for free, and can be tested like any other code. The catch is upkeep: senders change their templates constantly, and a changed template usually breaks a parser without raising an error.

That's about five fixes per parser. Most add a missing field or handle a template variant the parser hadn't seen; only a handful have ever been reverted.

How the graph is built

Each piece (detectors, triage, fix sessions, escalation agents, the parser factory) runs on its own schedule and picks up work from something saved: a row in the bug queue, a failed session's log, a pull request, a backlog file. Those saved things are the edges. Any piece can crash or be rewritten without the others noticing, and the agents can read every earlier decision.

Agents get the steps that need judgment: whether a parser's output is worse than an LLM's, how to patch it, what the patch missed. Deterministic steps get anything with a right answer: replaying emails, comparing fields, running CI, routing. When a step could go either way, the deterministic version has worked better.

Finding bugs worth fixing

Crashes and empty results are easy to catch. The hard case is a parser that returns something worse than it should, like a truncated title or a missing tracking number. For those we use shadow comparison: an LLM independently extracts a small sample of already-parsed emails, and a detector compares the two field by field. When the LLM's value is better, it files a bug like this (trimmed):

{
  "parser_name": "BradsDealsParser",
  "bug_type": "llm_outperforms_parser",
  "context": {
    "field_deltas": [{
      "field": "title",
      "parser_value": "Macy's: Scores",
      "llm_value": "Macy's: Scores Under $10",
      "data_present_in_email": true
    }]
  }
}

The important field is data_present_in_email. LLMs confidently report fields that aren't there, so a difference only counts if the value appears in the visible email. One report said the Google Flights parser missed a flight number, based on a logo image named DL.png. The email had no flight number.

False positives are expensive, because a coding agent will happily "fix" a parser that isn't broken. Cheap rules drop differences we already know are harmless, and an LLM auditor closes only the reports it's confident about. When in doubt, the report stays open: a false positive costs one wasted fix attempt, while a dismissed real bug goes unnoticed.

Getting a cheap open-weight model to fix them

The fixer is a headless coding agent running a small, cheap open-weight model. It runs thousands of times, and a frontier model would cost many times more. A small model does good work when the task is narrow and everything it needs is in front of it, so that's what the graph provides:

• The prompt already contains the bug reports, the failing emails, and the parser's source and tests, so no turns go to finding them.
• It may edit only its own parser and that parser's tests.
• It starts with a short playbook the graph has learned, like adding selectors for a new template next to the old ones, since old emails are still in people's inboxes.
• Each retry sees what earlier attempts tried. A session that can't make a single edit goes to a frontier model.

A deterministic verifier then replays the failing emails and a sample of passing ones through the patched parser. A reviewer agent reads the diff and can send it back for another round. It catches what a checker can't, like the comma case, a rule that's too broad, or a change that would hide real bugs. But the reviewer doesn't decide what ships: the replay, the regression test, CI, and branch protection are guardrails every fix has to pass. Merging without a human works because the blast radius is small. A parser handles one sender's email, so a bad fix means wrong cards for that sender until the detectors notice and the next fix lands.

The two catch different things. When we split large parsers into one file per email type, a byte-for-byte comparison of the moved code and the test suite caught every bug in the first ten splits. The LLM review caught none of them.

Agents take the shortcut

An agent under pressure to pass a check will look for the cheapest way to do it, and much of the graph exists to close those routes.

• Studying the grader. Given the repo, the fixer would read the verifier's source, theorize about how fixes are scored, and run out of time. Its prompt now says: "The verifier is a black box you cannot change; reverse-engineering it is the #1 way runs time out with no fix."
• Writing its own test. Now and then the fixer builds a sample email its fix happens to handle and reports success. The replay on real emails always wins.
• Deleting the problem. When a parser returns nothing, the email falls through to the LLM, which sometimes invents an item from a marketing email. The easiest way to clear that bug is to delete the route, which quietly sends that sender's mail to the LLM forever. The prompt forbids it in capitals: "Do NOT remove an existing route just because the parser currently returns no items."
• Fixing what isn't broken. A DocuSign report said a verification code was missing; it was already extracted under another field. The fixer spent a whole day on it. Parsers that keep defeating the fixer now go on a cooldown, and the report goes to escalation.

Escalation and the graph fixing itself

When a fix session fails, a more capable agent takes over that bug, and if it proves the report was wrong, it changes the detector so that kind of report stops being filed. A rescue agent pushes stuck PRs through or closes them. Every few hours, an automated research loop groups recent failures and false positives by cause and opens PRs against the prompts, filters, and checks the other agents use. No individual run is patched by hand.

The graph has had bugs of its own. A watchdog meant to stop sessions that talk without acting was reading the wrong messages and killing the fixer during its first long thinking step. And a filter to skip "unfixable" crash reports was rejected in review, because a parser crash and an infrastructure crash can look identical.

The routing graph

Every email walks a second, simpler graph to find its parser: a global router keyed on sender and subject, then a per-merchant router that picks a parser for each email type. It handles quirks you only find in production, like Apple's Hide My Email addresses and stores that send through Shopify or Klaviyo from their own domains. Misroutes are found and fixed by the same agents as every other bug.

Building new parsers

New parsers come from a factory agent that runs on a schedule with no engineer involved. Every email without a parser costs an LLM call, so the fallback logs work as a price list for missing parsers. Some senders reach many users, like a Microsoft account-security sender whose 423 recent emails all went to the LLM, across 203 users. Others are niche but heavy, like ArcaMax, a newsletter service with 2,236 fallback emails from nine users. Tickets the fixer files when it meets a genuinely new kind of email go first.

Before writing any code, the agent looks at what the fallback already extracts from that sender, which fields and how often, and uses that as the spec. It's the same standard the detectors will hold the parser to once it ships. The factory builds one parser at a time and never opens a PR for one it couldn't validate. Once a parser is live, its mistakes become bug reports like any other.

What we learned

We never made the models smarter. We tuned the harness around them constantly, and the research loop does much of that tuning itself, mostly by making checks stricter and the agents' examples more like real email.

The biggest improvement was to what a bug report contains. Before shadow comparison, a scheduled agent flagged whatever looked wrong, like a title that seemed too generic. Those were opinions, so the fixer had nothing specific to aim for and the verifier had nothing to check; about 320 fixes merged in four months. Once each report carried the value the parser should have produced, the graph merged more than 6,300, over 80% of them from shadow bugs.

A cheap open-weight model can carry the volume if the task is narrow, the context is complete, and the shortcuts are named. Model review makes fixes better; deterministic verifiers are the guardrails.

The open problem is verifying fields with no single right answer, like a description or a summary, where an LLM judge is least reliable.

Somewhere right now, a company is redesigning its receipt email. With any luck, the parser will be fixed before anyone here notices.