Planning eval · the compare
Two questions, two harnesses.
The compare has two parts. First, inside each harness: does more planning make a better app? Then, across harnesses: what changes when the builder changes?
The two runs built different apps from different briefs, and were graded by different exams and judges. A Codex 95% and a Claude 85% are not the same kind of number.
Part 1. Plan depth, inside each harness
Read each column top to bottom. Scores are the mean of five builds; the range is worst to best build.
| Plan, least to most | Claude Code exam | Codex checklist | Build time, Claude / Codex | Lines of code, Claude / Codex |
|---|---|---|---|---|
| Goal only | 84% (75% to 94%) | 90% (83% to 96%) | 19 / 35 min | 4,533 / 4,471 |
| Epics | 83% (81% to 87%) | 92% (88% to 100%) | 19 / 35 min | 4,392 / 3,067 |
| Stories | 82% (67% to 90%) | 95% (86% to 100%) | 13 / 34 min | 3,583 / 3,088 |
| Epics, features, stories | 86% (85% to 88%) | 94% (89% to 100%) | 16 / 40 min | 3,838 / 4,317 |
| Sprint | 67% (52% to 92%) | 96% (94% to 100%) | 14 / 39 min | 3,499 / 4,520 |
Claude Code: flat, then a drop
Four plans scored 82% to 86%. The sprint plan fell to 67%: it committed to two weeks of work, and the builds left out what the plan put in "not this sprint", including budgets in 3 of 5 builds.
Codex: a small climb
Scores rose from 90% (goal only) to 96% (sprint). But 37% of the checks could not be judged from screenshots, and each build was graded on a different subset.
Part 2. Harness against harness
| Claude Code run | Codex run | |
|---|---|---|
| Builder | Claude Opus 5.5, effort high | gpt-6-astra, reasoning high |
| Product | Personal budget tracker (242-word brief) | Daily spending report (158-word brief) |
| Sprint plan | One two-week commitment, the rest deferred | Four sprints covering everything |
| Time limit | 90 minutes | 60 minutes |
| Mean build time | 13 to 19 min per plan | 34 to 40 min per plan |
| Builds that booted | 25 of 25 | 25 of 25 |
| Graded by | Codex: 20 number questions read by scripts, 6 screen questions | Claude: 25 checks judged from screenshots, unseen checks excluded |
| Grader problem found | Codex-written test scripts did the apps' work; 29 passes removed | 37% of check slots ungraded |
Same in both: five plan depths from one brief, five builds per plan, one headless session per build, a FastAPI + React + SQLite scaffold, invented bank files, and the other company's AI as the grader.
Conclusions
- Planning is not what gets an agent to a running app. 50 of 50 builds booted, from a one-paragraph goal to a full sprint plan, in both harnesses.
- What the plan leaves out matters more than how deep it goes. Claude's sprint plan deferred budgets, and the builds skipped them. Codex's sprint plan covered everything in four sprints, and scored highest.
- The grader decides what "better" means. Number-reading scripts and screenshot judges measure different things, and both needed checking: one did the apps' work, the other could not see over a third of the checks.
- Codex took about twice as long per build on a shorter brief, with similar code size. Speed is the clearest harness difference these runs show.
- No exact conclusion yet. Plan in detail where you need certainty, vibe code the rest and regenerate, since code is almost free. Planning is a key part we have not cracked yet, and we will keep investigating.