Test script: Database for a vibe-coded app

• Job slug: database-for-vibe-coded-app
• Version / status: v1 · frozen 2026-10-16 11:47 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
• Rubric version: v1 (rubric)
• Author / date: Lando (CEO), 2026-10-16

1. The job

Your AI-built app keeps its data in a JSON file or in memory, so it vanishes on redeploy and can’t handle two people at once. You want a real hosted database on a free plan: create it, add a table, point the app at it, and know your data is safe from strangers and portable if you leave. You’re done when the app’s notes API reads and writes the database and the checks below pass.

2. Candidates

Scroll sideways for more columns →

3. Fixed inputs

• Starter app: data-starter (local throwaway repo, tag data-starter-v1). A Node 20 + Express 4 notes API (GET/POST /api/notes, PATCH/DELETE /api/notes/:id) with a one-file HTML page in public/. It stores notes in data/notes.json. Each note: id, text, done, created_at.
• Each tool gets its own copy of the starter and replaces only the storage layer (store.mjs). The API contract and the check script don’t change.
• Cloudflare D1 exception: D1 is reached from a Cloudflare Worker, not from an outside server (Cloudflare’s docs reserve the D1 HTTP API for admin use). For D1 the same API contract runs as a Worker with a D1 binding, deployed through the Cloudflare API. The check script hits the Worker’s URL instead.
• Operator machine: the test box (Toronto time; network location noted in the log). Latency numbers are from this one machine and are reported, not scored.

4. Fixed task

The operator (agent) implements this, following only the vendor’s official docs (each docs page used is logged):

Move the notes app from the JSON file to <tool>, on the free plan:
1. Create the database (or project) on the free plan, without a payment card.
2. Create a notes table/collection with id, text, done and created_at.
3. Make every API route read and write the database instead of the file.
4. Keep every secret (connection string, password, key, token) on the server only.
5. Leave the vendor's default access settings as they are, unless the docs tell you to change them.

5. Follow-up rules (agent operator version)

• Time cap: 60 active minutes per tool.
• Allowed: the vendor’s official docs, quickstarts, CLI/API references and error messages.
• Not allowed: copying another tool’s code, third-party tutorials or templates.
• Manual interventions (counted for C2): anything that needs a person in a web dashboard or a browser (clicks, copying a key, claiming a project), and any workaround the docs don’t cover.

6. Steps

1. Record the start time. Create the database and log its plan and cost.
2. Implement the task (§4) in the tool’s copy of the starter.
3. Run checks/run-checks.mjs against the running app. Save its JSON output.
4. Fix and rerun until all core checks pass or the cap is hit. The end time is when the last core check first passes (= time to working).
5. Run the export (D7) and the anonymous-access probe (D6).
6. Record lines changed, dependencies added, docs pages used, and the free-plan limits and price from the vendor’s pricing page, with the date.

7. Pass/fail checks

Scroll sideways for more columns →

8. Quality checklist (C5): developer experience

Scroll sideways for more columns →

9. Security checklist (C7)

Scroll sideways for more columns →

10. Weights and tie-breaker

• Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
• C2 anchors (manual interventions + problems the docs didn’t answer): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
• C3 anchors (T = 60 min): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
• C5 and C7: checklist-count anchor (rubric §3).
• C6 for a database means portability: 5 = standard engine (Postgres or SQLite) with a documented full export on the free plan that restores elsewhere · 4 = export works with manual steps · 3 = proprietary data model or export needs a paid plan · 2 = partial export · 1 = none.
• Tie-breaker: C7 (safe defaults), then C1.
• Extra metrics (reported, not scored): lines changed, dependencies added, docs pages used, D8 latency.

11. Cleanup

• Notes are fake text; no personal data. Delete the test databases after the verdict is published (unclaimed Neon projects expire on their own after 72 hours). Keys live only in the gitignored .env.

Amendments

• v1.1 (2026-10-16 11:49 ET, during j06-supabase-r1, before any check ran):

  1. Supabase runs its notes API as a Supabase Edge Function, not from the starter’s Express server. Reason: the operator box allows only HTTPS traffic (Postgres ports 5432/6543 time out), and Supabase’s server-side secret key and database password are only shown in its dashboard. Edge Functions get the secret key injected by Supabase, as its docs describe. Same API contract and check script. D3’s “restart” for Edge Functions and Workers = redeploying the same code.
  2. D7 runs pg_dump in a short-lived cloud sandbox (Vercel Sandbox, Ubuntu, pg_dump 18.6) for the two Postgres tools, for the same network reason. The dump is restored into a local Postgres in that sandbox. D1’s export uses Cloudflare’s export API and is restored into SQLite on the box.

  Neither change affects how a tool is scored.

Scoring notes (2026-10-16)

• S1 (Supabase): a table created with SQL on a new project was read (200) and written (201) by an anonymous request with the publishable key. Supabase’s docs say to enable RLS on tables created with SQL; after doing so, D6 passed. S1 measures the default, so it’s a miss.
• S2: Supabase accepted an sslmode=disable connection through the shared pooler (pg_stat_ssl.ssl = f), matching its docs. Neon refused it (“connection is insecure”). D1 has no database protocol outside Workers and the HTTPS API.
• S3: per the vendors’ pricing and limits pages: Neon Free has instant restore (up to 6 hours or 1 GB of changes), D1 Free has 7 days of Time Travel, Supabase Free has no backups (daily backups start on Pro).
• S4: Supabase Security Advisor (free). Neon’s Data API Advisors (they cover databases exposed through Neon’s Data API, which our setup didn’t use). We found no equivalent for D1.
• C2 (Supabase), 2 issues: the export needs the database password, which only the dashboard shows (we created a dedicated role with SQL, as Supabase’s roles docs recommend, and pg_dump then needed --enable-row-security); and the shared-pooler host name isn’t given outside the dashboard (we found it by trying aws-0/1/2). C2 (Neon): keeping a no-signup project past 72 hours needs a person to claim it in a browser (1 intervention). D1: the account had no workers.dev subdomain yet; the error message named the fix, and we registered one through the API.