Two Models Agreed. Both Were Wrong: An Agentic-Graph Driven Migration

In my last article, I argued that the model was not the breakthrough. The graph was: the shared context, tools, quality gates, and human decision points that turn a capable model into reliable engineering output. I ended with a goal: make the graph more observable.

Since then, we used that graph for something with real consequences. We moved production business logic out of our workflow automation layer and into typed tasks on our cloud platform. Several partner API integrations depend on that logic, and every call our product handles passes through part of it.

I expected the hard part to be the code. It was not.

The bottleneck moved

With the graph in place, writing the migration became the fast part. An agent could port a workflow branch into a typed task, write the tests, and open a pull request within hours. The new path was ready long before it was trusted.

What we waited for was evidence. The question was never "does it compile?" or "are the tests green?" It was "have we seen enough real behavior to let this path decide?"

For high-volume partner API integrations, the answer arrived quickly. A shadow running beside the live path collected enough real executions to cover every variant we cared about. For low-volume integrations, live traffic could take weeks to show the rare cases, so we did not wait for production to produce them. We replayed historical executions as test data instead.

That is when I realized the bottleneck had moved. It was no longer writing code. It was deciding, with evidence, when the new path was allowed to take over production.

We iterate fast as a team, so we also put a limit on the waiting. For components on the critical path, a shadow runs for at most three days. In practice, that means between 24 and 72 hours, depending on how critical the component is. If the evidence is not there by then, the problem is usually the evidence plan, not the calendar.

The migration as a graph

Every migrated step walks the same graph:

1. Baseline. Pin the currently published workflow, read-only.
2. Contract. Write down inputs, outputs, and how each route behaves.
3. Implement. An agent builds a typed task with tests.
4. Review gate. CI must be green, and a second model reviews the change independently.
5. Dormant deploy. CI deploys the task to the cloud platform. Nothing calls it yet.
6. Shadow. The new path runs beside the old one and observes. The old path still decides.
7. Observation gate. Every execution is compared until all variants have been seen with no unexplained mismatch.
8. Cutover. A human switches authority, and records the rollback version first.
9. Monitor. Live traffic is watched, with the rollback ladder ready.

Each gate names its evidence and who may pass it. Some gates collect their evidence automatically, but a human still decides. The cutover gate is human only, and that is written into the graph itself rather than into a policy document that someone has to remember.

Shadow before authority

The most reusable idea from this work is simple: a new path earns authority by watching first.

During the shadow phase, the new task receives the same input as the live path. It computes its answer, logs one structured fact, and changes nothing. Every execution produces two answers to compare: what the live path actually did, and what the new path would have done.

When the two disagree, we classify the mismatch before anyone reacts. It is one of three things:

• Model variance: a language model upstream gave a slightly different answer. This is expected and not a defect.
• A contract or parity defect: the new path really behaves differently. This sends the step back to implementation.
• An infrastructure error: a timeout or a missing permission. This is fixed outside the business logic.

Historical executions get the same comparison. We sanitize them down to shapes and labels, with no identifiers, and compare the new answer with what the old path decided. That is how low-volume integrations get their rare cases covered, and it is also how we checked logic before any shadow existed: the backtest for the post-call branch selection replayed historical executions and matched every decision the old path had made.

Once a path is live, problems follow a short escalation ladder. If the outcome is an expected business result, we keep it. If one partner integration misbehaves, we switch that one off. Otherwise, we hotfix if we can, and roll back if we cannot. The rollback is never improvised: its version was recorded before the cutover.

After the first cutover, every reservation that reached the new router in the first hours completed, including each room in multi-room bookings.

Two models agreed. Both were wrong.

One step needed to port the logic that decides which actions run after a call. In the workflow automation layer, that logic was a set of expressions evaluated by the engine's filter rules. Claude ported it into a typed task.

To prove the port was faithful, Claude also built an oracle: a script that ran the original expressions over a large cross-product of edge cases and recorded the expected answers. The port matched the oracle exactly. Every golden case passed.

Then Codex reviewed the change, read-only, as the graph requires. In the second of four review rounds, it found a problem that neither the tests nor the oracle could see.

The oracle modelled the programming language, not the workflow engine. For one helper that checks whether a value is empty, both the port and the oracle assumed that a missing value would raise an error. The engine does not do that. It rewrites the call and quietly returns a result. The port and the oracle agreed with each other perfectly, and both were wrong in the same way.

This is the failure that worries me most in AI-assisted engineering. It is not a model making an obvious mistake. It is a model building both the answer and the test from the same wrong belief. Every check passes, because every check shares the assumption.

Two habits came out of that loop:

• Verify findings against the primary source. We did not accept Codex's finding because a second model said so. We read the engine's source code, confirmed the behavior, and only then fixed it. A reviewer is only independent if it checks something neither author wrote.
• Ask whether the evidence discriminates. At one point the agent wrote that "production confirms" a behavior. The reviewer asked whether that evidence would have looked any different if the opposite were true. It would not have. Evidence that fits both hypotheses proves nothing.

This was one of the moments I realized how important it is to keep a human in the loop. Many people say coding is dead and there is no longer any need to learn programming. I would put it differently: coding may have become a commodity, an increasingly automated tool, but systems thinking remains the most important skill today. Even with state-of-the-art models, like the ones we used in this migration, a shared wrong assumption can still deliver spectacularly wrong results. Understanding software design patterns, APIs, data security, algorithms, and data structures is what lets us properly evaluate the work AI delivers.

Failure modes the graph caught

The shared assumption was the most interesting failure, but not the only one. Each became a gate or a guard, so the next migration starts safer.

Failure mode	What happened	What it became
Shared wrong assumption	The oracle and the implementation agreed on wrong engine behavior	Review against the primary source
Over-claimed evidence	"Production confirms" fit both hypotheses	Ask whether the evidence discriminates
Done too early	A ticket was closed at merge while cutover was still pending	Status tied to cutover evidence, not merge
Session-only automation	A watch job died when the agent session ended	Persistent, app-owned watch state
Merge order	Stacked pull requests merged into their base branches instead of main	A merge-order rule: one at a time, retarget first
Invisible syntax trap	A delimiter inside a code comment broke an expression in production	A static guard test
Missing registration point	A new result type was missing from one union	A checklist gate for new tasks

None of these were exotic. They are the ordinary ways engineering goes wrong, made faster by agents. The graph does not prevent them by being clever. It prevents them by making each one a named step with named evidence.

Making the graph visible

The graph lived in documents and in the agents' instructions. That worked, but nobody could see it while an agent was working. So we built a small control plane for it.

It renders the migration graph and shows where each agent is. A hook streams every tool call the agent makes, and each call is mapped onto a node: reading the baseline, editing the task, requesting review, checking the deployed version. Loops are visible. When a review sends the agent back to implementation, you see it travel the edge and the visit count rise.

Gates are where it becomes control rather than observation. When an agent reaches a gate, the gate turns pending and holds the agent. Evidence attaches to it automatically: CI results, the reviewer's verdict, the deployed version, the shadow comparison. A person approves or rejects it with a comment, and the decision flows back to the agent as its next instruction.

Some calls should never wait for a gate, so the control plane also blocks them before they run:

• changes to the workflow automation layer, which only humans edit;
• pushes to a deploy branch;
• production deploys from an agent;
• edits while the agent waits for a human decision.

We now use it to visualize the agent graph for complex migration steps. It is still a prototype, and it observes and gates the work; it does not prove a business outcome on its own. But it changed the conversation. Instead of asking an agent where it is, we look at the graph.

What stays human

Agents traverse the graph. People own these decisions:

• editing and publishing the workflow automation layer;
• cutover and rollback;
• accepting risk;
• defining what "done" means;
• deciding when the evidence is enough.

The last one is the decision this whole article is about.

Limits

A shadow shows only the variants that happened while it ran. Rare cases still need historical replays or deliberately constructed data.

Two models can share a blind spot. The tie-breaker is the primary source, not a third opinion.

The graph can automate a bad process as efficiently as a good one. It is only as good as the gates we choose and the evidence we demand.

And automation fails in mundane ways. Our own watch server stopped when it hit a session time limit: the same failure mode the graph was built to catch.

Verification is a property of the graph

Models are getting better quickly, and that is good news. But a better model does not remove the need to check its work. It changes what checking means.

In this migration, the checks that mattered were not inside any single model. They were in the graph: a second model looking at the same change, the primary source settling disagreements, shadow traffic standing in for trust, and a person deciding at the edge where authority changes.

Two models agreed, and both were wrong. The graph is how we found out before production did.

Sources

1. John Torres, “The Model Wasn’t the Breakthrough. The Graph Was.” — the graph-engineering foundation this article builds on.
2. Anthropic, “Building effective agents” — evaluator-optimizer and routing patterns.