Test script: Add login to a vibe-coded app
The script we wrote before testing, with every amendment and its timestamp. The verdict it produced: Clerk vs Supabase Auth vs Firebase Auth for vibe-coded apps.
Published as written, with private emails and internal file paths removed. Times are Eastern. How we score: the rubric.
- Job slug:
add-login-to-app - Version / status: v1.1 · v1 frozen 2026-10-15 23:40 ET (before the first run); v1.1 amended 2026-10-15 23:48 ET during run j02-supabase-r1. Changes are listed under “Amendments” at the end, and every vendor run uses v1.1.
- Rubric version: v1 (rubric)
- Author / date: Lando (CEO), 2026-10-15
1. The job
You have a small app (made with AI, no login yet) where everyone sees everyone’s data. You want real accounts: sign up, sign in, sign out, password reset, optional “Sign in with GitHub”, and each user sees only their own data, enforced on the server.
2. Candidates
| Tool | Plan | How the operator signs in to the vendor | ToS check (2026-10-15) | Run mode |
|---|---|---|---|---|
| Supabase Auth | Free org (“Acrophyl”), new free project (MCP get_cost = $0/month) | User’s existing account, through the official Supabase MCP | AUP bans automated or bulk creation of Supabase accounts and probing third parties. It doesn’t cover end-user test accounts in our own project or tests of our own app. Terms keyword scan: no automation or benchmark clause. | Agent-driven |
| Clerk | Hobby (free) | User’s account (GitHub login), created by the user | Terms keyword scan: no automation, bot, scraping or benchmark clause found | Agent-driven. The user copies the two API keys from the dashboard into secret env vars |
| Firebase Auth | Spark (free), project buildwiththis | User’s Google account; the user enabled Email/Password and registered the web app; the service-account JSON is in a box secret env var | Terms keyword scan (2026-10-16): no automation, bot, scraping or benchmark clause found | Agent-driven (run j02-firebase-r1) |
3. Fixed inputs
- Starter app: private repo
pathakarchit/bwt-auth-starter, tagstarter-v1(commit8538483). Vite 5 + React 18 notes UI and an Express 4 notes API storingdata/notes.json. No auth: every visitor sees every note. - Each tool works on its own branch from
starter-v1(supabase,clerk,firebase). - Notes stay in the starter’s own API for every tool. We test the auth provider, not its database. Isolation must be enforced by the Express API verifying the provider’s token.
- Test users: plus-addresses on the operator’s own inbox,
+bwt-aand+bwt-b. Each gets a random 20-character password generated per run, kept in.env.testand never committed. (v1 usedexample.com, which Supabase rejected as invalid.) No third party gets mail. - Email quota: sign-up is the first action on a fresh project, because some providers’ built-in mailers allow only a few emails per hour. Any email that can’t be delivered is logged, not worked around silently.
- Browser: Playwright Chromium (headless), 1280×800.
4. Fixed task (the “prompt”)
The operator (agent) implements this, following only the vendor’s official docs (each docs page used is logged):
Add authentication to the notes app:
1. Email + password sign-up, sign-in, and sign-out.
2. "Sign in with GitHub" if the free tier allows it.
3. The notes screen is only for signed-in users; signed-out visitors see the sign-in screen.
4. Each note belongs to the user who created it. The API must verify the user's token on every
request and only return, create, or delete that user's notes. Requests without a valid token get 401.
5. The session survives a page reload.
6. A "Forgot password?" flow: request a reset email, and a page to set a new password.
7. No secret keys in the browser bundle.5. Follow-up rules (agent operator version)
- Time cap: 120 active minutes per tool.
- Allowed: reading the vendor’s official docs, quickstarts and SDK references. Errors are fixed using those docs or the error message. Each docs page used is logged.
- Not allowed: copying another tool’s branch code, third-party tutorials, or community templates.
- Manual interventions (counted for C2): any change in the vendor dashboard, any SQL or admin action outside the SDKs, and anything a user would have to do by hand.
6. Steps
- Record the start time. Create or choose the vendor project and log its plan and cost.
git checkout -b <tool> starter-v1. Implement the task (§4).- Start the app (
npm run dev). Runchecks/run-checks.mjs(Playwright + direct API calls). Save its JSON output and screenshots. - Fix and rerun until all core checks pass or the cap is hit. Record the end time when the last core check first passes (= time to working).
- Record lines changed (
git diff --shortstat starter-v1 -- . ':!package-lock.json'), dependencies added, and docs pages used. - Record free-tier limits and pricing from the vendor’s pricing page, with the date.
7. Pass/fail checks
| ID | Check | How to test | Pass if | Evidence |
|---|---|---|---|---|
| A1 [core] | Sign-up | UI: sign up user A and user B | Accounts created; a confirmation step, if any, is reported | Screenshot, provider user list |
| A2 [core] | Sign-in | UI: sign in as A; also try a wrong password | Correct password works; wrong password is rejected with a message | Screenshots |
| A3 [core] | Sign-out | UI: sign out | Back to the sign-in screen; reload stays signed out | Screenshot |
| A4 [core] | Protected screen + API | Load the app signed out; call GET /api/notes with no token | Sign-in screen shown, no notes rendered; API returns 401 | Screenshot, HTTP log |
| A5 [core] | Per-user isolation (direct API) | A creates a note. With B’s token: list, GET A’s note by id, DELETE A’s note. Also try a forged token and a tampered token | B never sees or deletes A’s note (404/403/401); forged and tampered tokens get 401 | HTTP log |
| A6 [core] | Session persists | Sign in, reload | Still signed in, notes visible | Screenshot |
| A7 | Password reset exists | UI: “Forgot password?” → submit A’s email; open the set-new-password page | The request is accepted by the provider and the set-password UI exists. (Email delivery is logged separately) | Screenshots, HTTP log |
| A8 | GitHub login | UI: the button starts the provider’s GitHub OAuth flow | Redirects to github.com’s authorize page with the provider’s client. A full round-trip needs a human GitHub login and is logged as “not run by agent” | Screenshot |
| A9 [core / G2] | No secrets in client | Search dist/ for key-shaped secret values (sb_secret_…, sk_live_…, sk_test_… followed by key characters, private keys, JWTs whose role is service_role). Bare prefixes inside SDK code don’t count, but they are listed in the evidence. | None found | grep output |
| A10 [G2] | Server trusts tokens, not client-sent ids | Send POST /api/notes with B’s token plus a body field ownerId = A’s id | The note is owned by B | HTTP log |
8. Quality checklist (C5): developer experience and docs
| ID | Item | Pass if |
|---|---|---|
| Q1 | Quickstart for React (Vite) exists in the official docs | Yes, and it works as written |
| Q2 | Server-side token verification for Node/Express is documented | Yes, with a code sample |
| Q3 | Prebuilt UI or a clear headless pattern | One of the two is available |
| Q4 | Errors hit during the run were explained by the docs or the error message | Yes |
| Q5 | Docs state free-tier limits clearly | Yes, on the pricing page or in the docs |
9. Security checklist (C7)
| ID | Item | Pass if |
|---|---|---|
| S1 | Default password policy rejects weak passwords (try 123) | Rejected |
| S2 | Email confirmation is on by default | On (or the docs clearly warn) |
| S3 | Tokens expire (short-lived access token) | Expiry ≤ 1 hour, or documented rotation |
| S4 | Security advisor / warnings available | The vendor surfaces misconfigurations |
10. Weights and tie-breaker
- Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
- C2 anchors (manual interventions + docs-unanswered problems): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
- C3 anchors (T = 120 min): 5 = ≤30 · 4 = ≤60 · 3 = ≤90 · 2 = ≤120 · 1 = cap hit.
- C6 for auth means lock-in: can you export users (with password hashes) and leave? 5 = documented export including hashes; 3 = export without hashes; 1 = none.
- Tie-breaker: C1, then C2.
- Extra metrics (reported, not scored): lines changed, dependencies added, number of docs pages used.
11. Cleanup
- Test users use the operator’s own inbox plus-addresses. Delete the vendor projects/apps 30 days after the verdict is published. Never commit keys;
.env*is gitignored.
Amendments
-
v1.1 (2026-10-15 23:48 ET, during j02-supabase-r1):
- Test users moved off
example.com, because Supabase returned400 email_address_invalid. - Sign-up now has to be the first email-sending step.
- A9 now matches key-shaped values. The literal
"sb_secret_"inside supabase-js triggered a false positive.
None of these changes affects how a tool is scored.
- Test users moved off