Build time
19 min
average
One brief for a personal budget app, planned five ways. Claude Opus 5.5 built each plan five times, and another AI tested every build from the outside.
A personal budget tracker, described in one paragraph.
An AI turned that paragraph into five plans, from “just the goal” up to “a full sprint plan”.
Claude Opus 5.5 built each plan five times, alone, in one session per app.
Codex, a different AI, loaded the same bank files into every app and asked 26 questions, like “how much did I spend in January?” 20 have exact answers and 6 are judged from screenshots.
An app's score is how many of the 26 it got right. A plan's score is the average of its five apps. We also logged build time, cost and code size.
Some of Codex's test scripts did the app's work themselves. We blocked that and scored everything again.
From least to most planning. Same brief, same model, same limits. Every number is the average of the plan's five apps.
Each white dot is one app's exam score on a scale from 50% to 100%, and the orange line is the plan's average. Orange numbers are the best in their column.
Four of the five plans scored about the same, 82% to 86%.
With less planning, the AI added more on its own, and builds took 19 minutes and cost $5.04 to $5.51. With more planning, apps came out smaller, in 13 to 16 minutes for under $4.
It dropped budgets to fit “two weeks”, but the AI only needed 14 minutes. 3 of its 5 apps have no budgets.
Epics, features, stories wrote out how to calculate “safe to spend today”, and all 5 of its apps got it right. Epics only named it and got 0 of 5: all five guessed the same calculation, and the test marked it wrong.
Before we checked Codex's tests, 29 of their passes were the tests doing the app's work.
One product, one model and five apps per plan, so read this as a strong signal, not proof.
How many of each plan's five apps passed each question (a partial counts half). Highlighted rows are the questions where the plans differ by two apps or more. One judged question, an on-screen over-budget warning, failed for every app because of a gap in the test, so it costs every plan the same.
| Check | Goal only | Epics | Stories | Epics, features, stories | Sprint | |
|---|---|---|---|---|---|---|
| C01 | Transaction Countexact | 4/5 | 5/5 | 5/5 | 5/5 | 1/5 |
| C02 | Detected Accountsexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C03 | January Spendexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C04 | January Incomeexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C05 | February Refundexact | 5/5 | 5/5 | 5/5 | 5/5 | 2/5 |
| C06 | February Incomeexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C07 | March Splurgeexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C08 | March Boundary Dateexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C09 | March Totalexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C10 | April Duplicateexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C11 | April Totalexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C12 | May Alert Spendexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C13 | May Alertexact | 3/5 | 3/5 | 3/5 | 0/5 | 2/5 |
| C14 | June Merchant Fixexact | 5/5 | 5/5 | 5/5 | 5/5 | 2/5 |
| C15 | June Incomeexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C16 | July Partial Monthexact | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C17 | Future Merchant Ruleexact | 5/5 | 5/5 | 2/5 | 5/5 | 2/5 |
| C18 | Daily Dateexact | 4/5 | 5/5 | 4/5 | 5/5 | 2/5 |
| C19 | Dining Safe To Spendexact | 2/5 | 0/5 | 4/5 | 5/5 | 2/5 |
| C20 | Total Safe To Spendexact | 1/5 | 0/5 | 4/5 | 5/5 | 2/5 |
| C21 | Dashboard Homejudged | 4.5/5 | 5/5 | 3/5 | 3/5 | 3/5 |
| C22 | Budget Warningjudged | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
| C23 | Navigationjudged | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| C24 | Savings Tipsjudged | 5/5 | 5/5 | 5/5 | 5/5 | 2/5 |
| C25 | Income Sourcesjudged | 5/5 | 2/5 | 0/5 | 0/5 | 0/5 |
| C26 | Sync Changesjudged | 1/5 | 3.5/5 | 1.5/5 | 3.5/5 | 1.5/5 |