Compiling a world model to code: Generalizing Jeremy Berman's ARC-AGI result

Jeremy Berman's agent scored 96% on ARC-AGI-3 by compiling its reasoning into executable code. It's a powerful approach—but real-world problems like business are a lot harder to hill-climb than ARC puzzles.

Generalizing Jeremy Berman’s ARC-AGI result

AI researcher Jeremy Berman has just announced that he used Claude Opus 5 in Claude Code with a custom prompt and a couple of custom commands and achieved over 96% on ARC-AGI-3 in a single pass. ARC-AGI-3 is considered one of the world’s hardest AI capabilities benchmarks (you can play the ARC-AGI-3 games yourself here). Without Berman’s custom prompt and commands, Opus 5 scores just over 30%.

Berman described on Twitter how it works:

In one pass Opus wrote 269 programs (~12,700 lines). It built parsers for all 25 games, searching functions for 23, and game simulators for 9. … Code execution makes this scalable and cheaper, which is why using this harness is cheaper than asking Opus to solve each game directly. With code, the model can compile its reasoning into a function, run that function thousands of times, and execute whole action sequences.

ARC-AGI-3 mastermind François Chollet commented on Jeremy’s work,

Jeremy’s excellent work here is a great illustration of a very powerful type of approach: LLM-guided on-the-fly synthesis of a symbolic world model, i.e. making sense of the world by writing executable code that encodes your understanding of the causal mechanics of the world. So far, all of the top-performing harnesses on ARC-AGI-3 use this style of approach. This is not an approach that will only shine on ARC 3. This is where most of AI is headed in the long run.

I agree with Chollet that AI is headed in something like this direction, but I want to explore the limitations of that, because I think the real world’s a lot harder than ARC-AGI in specific ways that make the approach hard to generalize.

A quick crash course on Jeremy’s approach

ARC-AGI presents the agent with a gameboard. The agent takes actions to manipulate the board state. It has a budget of 2,500 actions. The board resets and starts the next level only if it reaches the “goal” state (meaning you “won”) or you forfeit or exhaust your action budget (meaning you “lost”).

The most important instruction Jeremy gave the model was to “diff” successive board states with a mechanical difference-checking tool. This is basically a correction to the model’s epistemology. Opus has poor “vision” capabilities and also isn’t great at parsing ASCII representations of game boards. But it doesn’t really “know” this about itself, so it’s helpful to point it out and to give the model a strategy for working around this limitation. This is a super useful insight if you’re doing visual work with agents, like frontend web design, but I don’t know that it generalizes much beyond that.

Jeremy also gave his agents an /act command for taking actions and /status and /board commands for checking game state. Actions and resulting board states got auto-logged to the file system in logs.txt. Honestly, the motivation here was probably as much to support human observability as anything else.

As for methodology, Jeremy told the agent to:

• Log durable findings (confirmed mechanics, level solutions, disconfirmed hypotheses) in notes.md.
• Build reusable parser and simulator software tools.
• Form a hypothesis, then run the smallest/cheapest possible experiment to verify, then compare predicted with observed board state.
• Nail down game mechanics first, and game strategies only afterward.
• Batch actions to conserve tokens.

He also included instructions to prevent the model from giving up early or stopping to ask questions.

Jeremy’s Opus 5 agent completed 24 of the 25 ARC-AGI-3 games, and it averaged just ~350 action steps per game (~86% under the 2,500-step budget). Which is really, really good.

What generalizes

Working around model epistemology limitations

I think the most persuasive takeaway from Jeremy’s implementation is that to get the best results from AI models, we’re going to need to systematically study AI epistemology and build tools to help models work around their epistemological limitations. Jeremy’s Opus agent got a lot of leverage out of a simple diff tool; you can imagine that similar visual diff tools might be really valuable in other contexts, such as when using Opus for frontend web development.

What only partly generalizes

The rest of the methodology here only partially transfers to other domains, I think.

Say you’re trying to use Jeremy’s method to solve business problems. You can definitely use something like Jeremy’s ARC-AGI approach! But the structure of real-world problems is pretty different, in ways that really confound the strategy.

The scientific method is good, but real science is less deterministic than ARC

Pre-registering hypotheses, testing them, comparing them to observed outcomes, then capturing learnings is basically the correct approach for any problem space. But if you’ve used this strategy with AI models trained in game environments, then you’ve encountered the failure modes:

1. The model tries one experiment, draws a hard conclusion from it, then never revisits that conclusion, so it ends up “learning” things that are simply incorrect.
2. The model over-uses its learning log, recording a lot of really basic world-knowledge stuff that isn’t actually of any use.

Models have bad epistemological hygiene, if you will. And to some extent, that’s because the game worlds the models are trained in are very predictable/deterministic, and an N-of-1 experiment is usually sufficient to learn a new game mechanic. But in the real world, you’re sampling experimental outcomes from a non-stationary probability distribution, and you need to have a real sense of humility about the things you “learn” from experiments.

Real AI-powered learning over real-world problem spaces is going to require large numbers of humans and AI agents coordinating to run many experiments, in much the same way that large institutions already do.

The opportunity cost of experimenting is often large

In the real world, running experiments has an opportunity cost. If I spend my business budget experimenting with SEO, I can’t spend it on ads. If I spend it experimenting with ads, I can’t spend it on events. If I spend it experimenting with events, I can’t spend it on salespeople. And so on.

And for any given experiment, the minimum bet size is often large, and the consequence of running out of budget may be existential.

ARC-AGI-3 does have a budgeting mechanic (in that it limits the model to 2,500 steps), but I’m not sure it simulates tradeoffs in quite the same way. Experiments in ARC-AGI are relatively cheap, and it’s not typically the case that exploring one possible solution forecloses exploring another. Just because the models successfully use experimentation to hill-climb this game environment doesn’t mean they’ll be able to do the same for maximizing real-world business revenue.

Real-world “board state” and “win condition” aren’t always clear

In ARC-AGI-3, the model always has perfect information about the board state and can clearly causally attribute state changes to actions taken. The games also have a straightforward “win” signal: the board resets, the level clears. All of this happens pretty quickly, so you get a fast and clear reward signal.

In contrast, in the real world, you often have only limited knowledge of the board state and struggle to causally attribute wins or losses to any action you took. Reward signals may appear only after unpredictable lag times. You may have a general idea of your goals, but you almost certainly haven’t quantified and probably can’t measure or verify everything you care about, so it’s hard to say if you’ve “won”.

In business, for instance, you find out once per quarter what the revenue and profit numbers were. But what timeframe are we optimizing revenue over? Do we only care about magnitude, or also variance? What else do we care about besides these headline numbers? Operational continuity? Sustainability? Employee satisfaction? Workplace safety? Human rights? How much do we weight each of these, and how can we even measure them?

Businesses are going to start using AI to hill-climb their business goals. Chollet is right about that. But they’re going to need to be very careful not to under-specify what success looks like.

Triaging the limitations of the method

A few modest proposals for overcoming the limitations of the Berman method for real-world work:

1. Every person and every business should start curating their own reward function. This reward function should quantify everything you care about, with weights, and elasticities, and variances. “Value function engineer” is the new job title every company should be hiring for.
2. Companies should be figuring out how to make the reward signals for the outcomes they care about fast and fully observable. The timeless wisdom that you should decompose long-term projects into smaller, bite-sized pieces remains true in the AI era. The same goes for big-picture goals. For example, you can decompose revenue goals into smaller goals for clicks and conversion rates.
3. Obtain as much historical data as possible for your problem domain. Jeremy’s agents achieved what they did by building “custom parsers” for measuring board state and “custom simulators” for predicting outcomes. Experiments are expensive, but simulation based on someone else’s past experiments is cheap. The more such data you can gather, the better you can understand the “game mechanics” and “board state” before you start burning budget.

Broadly, here’s the transferable methodology:

• Parse the historical record with code.
• Keep a calibrated hypothesis ledger.
• Synthesize a world model and compile it as an executable simulation.
• Generate and pre-register predictions before acting.
• Monitor the difference between the model and reality.
• Exercise intellectual humility and reason about outcomes probabilistically.
• Re-derive from scratch when stuck.
• Keep everything human-observable so we can identify when automated hill-climbing has lost the plot.

If you can do all that, then you can use the Jeremy Berman method to solve your real-world human or institutional problems with an automated learning system.