Visible checks
A small lead, not a verdict
The sprint plan has the highest visible-check mean. Uneven coverage prevents a reliable winner claim.
Completed experiment 25 builds finished
More planning.
Better apps?
Five ways to ask for the same money-tracking app. Five independent builds each. Here is what the recorded results show, and what they do not.
Automated results. Human review pending.25/25Builds finished normally
25/25Apps booted initially
62.6%Checks actually graded
0/25Human app reviews
Visual clarity
Stories score highest visually
The stories builds average 3.2/5. Screenshots do not prove that a person can complete the money-control tasks.
Real money control
The important gaps remain
No completed interaction task or reliable saved-state behavior was verified by the automated collector. That is an evaluation gap, not proof every app failed.
A high score is not full app acceptance
234 of 625 checks were excluded (37.4%). Only 391 checks received a grade. Partial verdicts get half credit; excluded checks receive no grade. Each build has a different denominator, from 9 to 21 checks. The percentages below are not financial accuracy or requirements-completion rates.
Five builds per approach
Compare the approaches
Means use the five individual build scores. Each bar runs from 0% to 100%.
| Planning input | Visible-check score | Checks graded | UI, 1-5 | Mean build time |
|---|
Build time is wall-clock time per app, not total project effort. Planning preparation and evaluation time are not included. Shared resources and the serial-to-parallel change affect timing.
Look behind the percentages
What was actually graded?
339 pass52 partial234 excluded
There are zero fail verdicts in the screenshot exam. This does not mean every check passed: partials receive half credit, and excluded checks receive no grade. Of the passes, 25 are the HTTP health check.
How the score works
(Pass + half of partial) / graded checks
Excluded checks leave the denominator. We average each build's score, not all checks pooled together. Always read score and coverage together.
The least observed requirements
These are missing observations, not confirmed missing app features.
Beyond how the screens look
Could a human use the app?
The collector tried the same ten intents per app. Finding navigation is not completing the task.
| Human intent | Navigation observed | Completion verified |
|---|
Most interactive behavior is still unverified
One budget edit was attempted. Its reload and restart readings remained unverified. No note or correction edit was verified. Refresh was attempted, but next-day advancement and duplicate-safe syncing remained unverified.
Three collections recorded later connection errors, despite successful initial boots.
Source amounts: limited evidence
Only identifiable visible amount magnitudes are compared. Totals, transfer direction, classification, duplicates and daily advancement remain unverified.
13 mismatches need interpretation
All 13 are in one sprint-plan build, on transfer rows labelled "Fee only". The checker compared displayed fees or exchange loss with gross transfer amounts. This may be a scorer interpretation mismatch, not a proven financial error.
The raw evidence is preserved. Do not rank approaches by this source-matching count.
Every outcome stays in the sample
Explore all 25 builds
Open a build for its denominator, check verdicts and observed tasks. No app was repaired for these results.
Screenshots and app identities are not attached to these arm-labelled cards before blind human review. Reviewers should not see approach labels or machine scores before giving their verdicts.
Keep the claim smaller than the evidence
Method and limits
What stayed the same
- One short human brief, with five approved planning treatments.
- Codex CLI,
gpt-6-astra, high reasoning. - One empty React, FastAPI and SQLite scaffold.
- The same synthetic bank records and supporting files.
- Five independent builds per treatment, in seeded randomized order.
- A 60-minute wall-time cap per main build. No feedback between builds.
- A different-provider Claude judge and the same frozen exam.
What this experiment cannot settle
- Detailed plans add implementation choices as well as structure. This is not a structure-only comparison.
- Five builds per approach, on one product, are not proof of a general or statistically significant effect.
- Screenshot checks cannot prove correct accounting, daily sync or saved-state reliability.
- Human usability review remains pending. Nobody has accepted these apps for real money.
- The run started serially and moved to two concurrent builds. The change is recorded; no finished build was rerun.
- Dollar costs and planning preparation time were not measured. No cost estimate is invented.
Recorded source files and reproducibility
This is a read-only snapshot of one completed batch. The earlier 30-minute attempts and infrastructure pilot are not in its 25-build sample. Rendering this page does not launch builds, alter evidence or publish anything.