External review· done on the public product, aistudio.google.com/apps — no internal tools or access ·default model (Gemini 3.5 Flash)·June 2026
The question: can AI Studio Build's default code-generation be improved with a system-instruction — and where does that not work? What we did: generated apps on the public product, audited the downloads, and A/B-tested fixes across 14 rounds of probes (~157 apps). The finding:
AI Studio Build's default output is strong on what you ask, weak on what you don't.
Asked explicitly for hard things — end-to-end encryption, server-side validation, optimistic rollback — Build implements them correctly. The same apps ship with no error boundary, no reduced-motion floor, and no mobile-safe viewport. The polish wins a first glance; the engineering fails the first real session.
The gaps are defaults, not capability — Build builds what you ask and omits the engineering you didn't. Each defect class has a primary owner among four: instruction, lint, template, pipeline.
The test: we added one system-instruction block, then generated the same 10 apps with it off, then on. Four engineering defaults that were absent in every app appear in every app:
0/10 → 10/10 on every paired app, same session (McNemar paired test, p≈0.002).
The instruction is layer one. Three other defect classes need a different owner — the honest part of the result:
- Lint — a prohibition prose won't enforce.
transition-all(a paint-heavy CSS shortcut) barely moved (113→88) and one run got worse. Needs a deterministic rule. - Template — config the model won't author from a prompt: a strict
tsconfig, an.eslintrc, a sharedsrc/utils. Ship it in the scaffold. - Pipeline — the downloaded code disagreeing with the live app (a stale export, or an edit reported done but never written). Downstream of the model; no prompt reaches it.
~157 apps generated across 14 rounds. Three numbers, one map: the cause-and-effect test runs on 10 matched off/on pairs; broad defect rates come from a separate 55-app default sample; whole-corpus counts are over 101 apps — we never mix them. · every prompt
- 1Write a representative one-shot prompt
- 2Run ONE shot in Build — no follow-ups
- 3Download the .zip artifact
- 4Audit · 12 dims · Lighthouse · axe · render
- ↻if defect, A/B the lever (same prompt ± one rule)
- 5Route defect → owner layer, then re-measure
- ↻repeats & deepens across 14 rounds
The constraint — no internal access, public product only — is the method: treat Build as a black box, probe with the artifact a user can download. Each lap sharpens the diagnosis; the dashed branch isolates the lever (same prompt ± one rule) before routing the defect to its owner.
Add the block to your system-prompt as a candidate change and run it against your next eval batch — it flips the concrete defaults 0/10→10/10, same-session paired. Low cost, high leverage, already validated.
Ship the deterministic fixes in the pipeline — a one-line ESLint config + a template stylesheet & src/utils. Reaches 96/101 apps at near-zero marginal cost.
Add an integrity check to the export — does the downloaded .zip contradict itself? (the download lags the live app; an edit reported "done" but never written; a wrong .env key). Catchable from the file alone.
Fold this loop in as a held-out eval slice — contamination-free, outside-in signal on the axes preference arenas reward least; the signal your own telemetry can't generate.
What we found
Default gaps, not capability gaps.
Every recurring defect has a counter-example in the same fleet — same model, same batch. The model can; it just doesn't by default:
One notes app pipes user markdown straight into dangerouslySetInnerHTML — stored XSS.
Its sibling renders markdown to JSX and blocks javascript: links — safe by construction.
A within-fleet control: the capability exists; the default doesn't.
And the fix is cheap because of where each defect class lives. The old "two exceptions" was too small: the data closes four layers, each proven by a negative experiment — a fix at the wrong layer provably fails.
transition-all, a paired run made it worse (9→18, 18→29 under ON); key={index} (155×) and as any (68×) ride the same one-line ESLint config. .eslintrc + tsconfig strict, and an src/utils for cents + crypto.randomUUID. It's one cause, not a list: Build is tuned to win the five-second preview vote, not survive the first real user. Measurable — Lighthouse a11y averages 83 ("looks fine") while axe finds ~14 serious/app; Flash→Pro lifts the surface score, not the structural defaults.
Twelve dimensions, graded
Where Build stands on every engineering axis — and which layer fixes each.
One row per dimension across all ~157 apps. The grade is Build's untreated default; the OFF→ON column shows what one pasted instruction block does — and that column alone tells you how each defect gets fixed:
↗ steep = the instruction fixes it · → shallow = prose can't, needs a lint · — flat = pipeline, no prompt reaches it · ● dot = already a strength
strict 0/101; non-null ! 195× vs unknown 2×templateuseReducer 2/157; updater-form <10% of setStatetemplatemin-w-0 missing on ~80% of flex children; dvh 0 by defaultinstructiontransition-all 1361×; resists prose (113→88; one round even got worse)lintGrades are Build's untreated default; the slope is the same off→on paired flip; the left edge is the owner layer.
Versus the field, the grades sort cleanly: Build trails every shadcn-based rival (v0, Lovable, Bolt, Cursor) on accessibility by construction — they inherit Radix, Build hand-rolls — yet leads on artifact integrity and security-when-asked, the two things no rival can claim.
Why the fix is cheap
The defects share legs — so a few changes reach almost everything.
The worst failures aren't single bugs; they're compounds — individually-medium defects that co-occur into something worse than their sum. And the most prevalent defect in the corpus is the quietest: 74/101 apps swallow errors to the console only — a failed save looks like success. Because the legs are shared and template-level, the marginal cost per app is near zero.
Unguarded JSON.parse(localStorage) in a root initializer + no ErrorBoundary → blanks on first render, and re-poisons every reload, so the white screen survives refresh. Confirmed at runtime — 3/3 apps blanked on a corrupt key, byte-identical across reloads.
confirm() delete + write + a catch that only warns + no boundary → a failed save reads as success; reload shows stale data, swallowed again.
No reduced-motion + no dvh + a viewport trap → on a phone: clipped under the URL bar, a fixed element forcing horizontal scroll, transition-all janking through it — at once.
Float money + key={index} on editable rows + alert-save → an amount can bind to the wrong record while float drift hides under a clean $X.XX.
One change at the right layer breaks several compounds at once: a template <ErrorBoundary> reaches 76 apps, the reduced-motion/dvh stylesheet 84, killing native alert 49 — the full per-intervention table is one click away.
Counts are over the full 101-app corpus (conservative — they include instruction-treated apps that already lack the defect). ErrorBoundary and the alert-fix together reach 91/101 apps; all four template fixes together reach 96/101. "Reaches" = removes a defect leg, not fixes every app — but at template level there's no per-app cost.
Is the sample even representative?
We tested the genres Build is actually for — and gave credit where due.
- In the signature genres (image, chat, maps), the AI is real and securely wired — a server-side proxy, the key never in the browser. We went looking for the "it fakes the AI" dunk and found the opposite. Credit to Build.
- It builds what you ask — even the hard things. Easy capability claims: 5/7 honored. The hardest, most-faked claims — end-to-end (ECDH) encryption, real server-side validation, optimistic rollback, unbounded undo/redo — 7/7 built for real. Over-claims sit only at the margins (a hardcoded "60 FPS" badge, ARIA promised but unwired). So the gaps are defaults, not capability — Build skips the engineering you didn't think to ask for.
- Terse prompts still produce feature-dense apps: a three-word "a kanban board" → 1655 LOC with tags, filtering, search, drag-and-drop. So our detailed probes represent real outcomes, and the defects aren't prompt artifacts.
- The defect signature is genre-independent — a Snake game and a maps explorer ship the identical a11y/robustness gaps, 9/9. The defects travel with the template, not the genre.
The findings cut both ways: a clean concession that strengthens every critical claim.
We pulled the lever — reproducibly
The decisive test isn't a better prompt. We pasted the whole block into Build's Instructions → "Write my own instructions" panel once, then ran plain prompts that mention none of it. You can reproduce it end-to-end: every prompt across all 14 rounds is published — paste the block, run them, score the downloads.