Planning eval · 50 AI builds · October 2026

Kanban, Stories, Features, Sprint? Which Planning Method Works Best With AI?

How much planning should you hand an AI coding agent? We wrote five plans from the same brief, from just the goal up to a full sprint plan, and had two agents build each plan five times. Then the other company's AI graded every app.

Harness 1

Claude Code

Personal budget tracker

82% to 86%

Four plans within a question of each other on a 26-question exam; the sprint plan scored 67%. Built by Claude Opus 5.5, graded by Codex.

How much you plan barely changes whether the app works. It changes what gets built.

Which plan wonResultsThe plansAll 25 builds
Open Claude Code →

Harness 2

Codex

Daily spending report

90% to 96%

On a screenshot checklist, rising slightly with more planning; 37% of checks could not be judged. Built by gpt-6-astra, graded by Claude.

More detail scored a little higher on screenshots. No proven winning plan.

SummaryResultsThe plansAll 25 builds
Open Codex →
Two experiments, one question.

Different apps, briefs and exams, so compare the direction of each result, not the percentages. Read the full compare: plan depth inside each harness, then harness against harness.

What we learned

No exact conclusion yet.

No plan won outright in either harness. Planning is a key part of building with AI, and we have not cracked it yet. We will keep investigating.

  1. Every plan produced a running app. 50 of 50 builds booted, from a one-paragraph goal to a full sprint plan.
  2. Plan in detail where you need certainty. Say what must never go wrong and give exact checks: $100 plus $50 is $150, and importing the same file twice must not make it $300.
  3. Vibe code the rest, and regenerate. Code is almost free now: a whole app took 13 to 40 minutes on average. Rebuilding from a better prompt can beat patching.
  4. Plan content beats plan depth. Whatever a plan pushes to "later", the agent does not build. Claude's two-week sprint plan left budgets out of 3 of 5 budget apps.
  5. Check the checker. AI-written test scripts quietly did the apps' work (29 fake passes), and screenshot judges could not see over a third of the checks.
  6. Rethink planning. Sprints, story points and deep backlogs were built for human teams. With agents, the plan's job changes, and the right shape is still open.

All code, plans and raw results come from the planning-eval harness. All bank data is invented.