Test script: Security check for a vibe-coded app

• Job slug: security-check-vibe-coded-app (job #5)
• Version / status: v1.1 (amended 2026-10-16 02:03 ET, see A6–A8; A9–A10 added 04:30 ET for Aikido, before its scored scan; applies retroactively to all r1 evidence and to every later run) · v1 frozen 2026-10-16 01:50 ET (commit 9f57dcd), before the lab is built and before any tool runs. Changes after freeze go under “Amendments”, dated, with the reason.
• Rubric version: v1 (rubric), re-mapped for this job as section 9 allows (“jobs may re-weight… the test script must state the weights before the first run”).
• Author / date: Lando (CEO), 2026-10-16
• Ethics: we scan only the lab app we built and own. No third-party apps. No real secrets: every planted key is a dummy in the right format. The lab Supabase project is throwaway, holds only fake rows, and is deleted after the verdict is published (section 11).

1. The job

You built an app with Lovable, Bolt, Cursor or similar, usually on Supabase. Before launch you want one tool to tell you whether your app leaks data or secrets, and what to fix. You’re done when you have a report and you know what to change.

2. Candidates and ToS checks (2026-10-16)

Scroll sideways for more columns →

Other URL scanners seen (bleek, Prodable, NoCodeFinishers, Fasrad) do similar live-surface checks. We take one representative, VibeScan, because its free scan includes the RLS probe with no sign-up. bleek’s RLS probe is paid; Prodable asks for an email for the full report.

3. Fixed input: the planted-issues lab

• Repo: private pathakarchit/bwt-auth-starter, new branch vuln-lab cut from main (starter-v1), tagged vuln-lab-v1 before the first run.
• Shape: what Lovable and Bolt generate. Vite + React SPA calling Supabase directly with supabase-js, plus one Supabase Edge Function. The starter’s Express server is removed on this branch so nothing unplanted is reachable.
• Backend: a new free Supabase project bwt-vuln-lab (org Acrophyl, MCP get_cost = $0/month). Schema and seed come from supabase/migrations/*.sql in the branch. The seed holds fake rows only.
• Live site: Cloudflare Pages project bwt-vuln-lab (Git-connected, branch vuln-lab) at https://bwt-vuln-lab.pages.dev. Build env holds the public key and the dummy secret.
• Test users: two users created through the app’s sign-up, [email protected] and [email protected], with random passwords in the gitignored .env.test. Email confirmation is on (project default); the confirmation emails arrive at the operator’s inbox.

3a. Planted issues (N = 10)

Scroll sideways for more columns →

3b. Decoys (count false positives)

Scroll sideways for more columns →

4. Fixed task (what the operator does with each tool)

Using only the tool's official docs and its free tier, scan the lab the way the tool is meant to be
used (database for the Advisor, repo for code scanners, live URL for URL scanners), get the first
full report, and record every finding verbatim. Do not tell the tool what was planted.

Then the scorer maps each finding to V1–V10, D1–D4, “extra true finding” (real but unplanted, e.g. a missing security header), or “false positive”.

5. Rules

• Time cap: 60 active minutes per tool to the first full report.
• Docs only: vendor docs and the tool’s own UI text. No tuning of rules or config beyond the documented quick start. A reader wouldn’t do that.
• Steps (C3) = every action a reader must take from zero to the first report: create account, install, connect repo / grant access, enter URL or command, choose project, start scan, open report. Logged as a numbered list. When the agent uses an equivalent API (Supabase MCP), we count the dashboard path from the vendor’s docs and say so.
• User-only steps (OAuth on the user’s accounts, payments, captchas we can’t pass) are queued, not waited on.
• One scan per run. No re-scanning to fish for better results, except the post-fix rescan (F4).

6. Matching rules

• Caught (1): the finding names the right object (table, policy, bucket, file, function, dependency, setting) and the right problem class.
• Partial (0.5): right object, vague or wrong problem; or right problem with no object a reader could act on.
• Missed (0).
• False positive: a finding that is wrong. Either it flags a decoy as a vulnerability (D1 at high/critical severity, D2, D3, D4 as “anyone can list/upload”), or it claims an issue that isn’t there. Informational notes that correctly say “this is fine” aren’t counted.
• False all-clear (v1.1): the report explicitly says a check passed, or that the thing is fine or secure, for an object or problem class that covers a planted issue (V1–V10) that ground-truth-v1.json showed exploitable at scan time. Examples: “RLS looks correct” while V1 was open; “API routes require authentication” while V6 was open. Saying nothing is a miss, not a false all-clear. A report the tool itself marks incomplete or low-confidence is not a false all-clear for the checks it couldn’t run.
• Extra true findings are logged and reported on the page but not scored.

7. Post-fix rescan (for C6)

After every tool’s r1:

1. Apply one canonical fix set to the lab (tag vuln-lab-v1-fixed): RLS on with owner policies; secret key removed and the bucket made private; JWT required on the function; view dropped; text rendered as text; lodash bumped; function revoked from anon with an ownership check. .env removed from git. Dummy keys need no rotation (not real); for readers, the page says to rotate.
2. Rescan with each tool.
3. Rerun vuln-lab-verify.mjs, which must show every exploit fails.

8. Checklists

C6 fix guidance (F1–F4, checklist-count anchor):

• F1: every caught issue comes with a fix instruction in the report.
• F2: the fixes are correct per vendor docs (Supabase RLS/storage/functions docs, React docs, the advisory) and wouldn’t break the app.
• F3: at least one fix is copy-pasteable (SQL, code, CLI or an AI prompt).
• F4: after the canonical fixes, the rescan clears every finding it raised for V-items, with no new false positives.

C7 privacy (P1–P4, checklist-count anchor):

• P1: source code doesn’t have to leave your machine, or the tool only reads what’s already public.
• P2: doesn’t need your service-role/secret key, DB password or Supabase access token given to a third party. A first-party tool is fine.
• P3: needs no broad OAuth scope (e.g. read access to all your repos).
• P4: data retention or deletion is documented (cited, dated).

9. Weights, anchors, gates, tie-breakers

Scroll sideways for more columns →

• Gate GS (data safety for scanners): a tool that needs the service-role/secret key or DB password given to a third party can’t be the winner. It is flagged on the page.
• G3 (honest pricing) applies as in the rubric.
• Re-run: if the top two are within 5 points, run both again (r2) and average. Scanners should be mostly repeatable; differences are logged.
• Tie-breakers, in order: C1 (more planted issues caught), then C2 (fewer false positives), then C7 (privacy).
• Expected coverage gap: each tool looks at one layer (database, code or live site). We expect no single tool to catch all 10. The page reports the per-issue caught/missed table so readers can combine tools. Picking one tool is still the job.

10. Evidence

our private evidence folder: raw report (JSON/HTML/screenshot), timeline.log (ET), steps.md, findings-map.md (finding → V/D/extra/FP). Ground truth: ground-truth-v1.json from vuln-lab-verify.mjs before any scan, and ground-truth-fixed.json after the fixes.

11. Cleanup

• Keep the lab until the verdict is published (plus 30 days for re-checks), but pause or delete the Supabase project as soon as the r1/r2 runs and the rescan are done if publication is delayed.
• Delete in this order: Cloudflare Pages project, then the Supabase project. The vuln-lab branch stays (fixture for re-tests) with dummy keys only.

Amendments

• A1 (2026-10-16 01:52 ET, before any tool ran): V10 changed from “email confirmation off” to “SECURITY DEFINER function with no ownership check”. Reason: the Supabase MCP can’t change Auth settings, and the dashboard toggle would have needed the user. The new V10 is a common AI-generated pattern (a definer RPC used to get around RLS) and can be planted from SQL. Email confirmation stays on. N is still 10.
• A2 (2026-10-16 01:53 ET, before any tool ran): tag vuln-lab-v1 moved from 2cad5e0 to e0a0c38. The app is unchanged; the move only adds checks/vuln-lab-verify.mjs. Ground truth (ground-truth-v1.json, 05:53 UTC = 01:53 ET): V1–V10 all exploitable; decoys D2 and D4 not exploitable.
• A4 (2026-10-16 01:55 ET, after VibeScan r1 returned no report, before any other URL scanner ran): added Prodable (free URL scan, https://prodable.app/scan) as a second URL scanner. Reason: VibeScan couldn’t scan the lab (“blocked / too little evidence”, 3 attempts), which left no working live-site scanner. ToS check: terms §4 needs ownership or authorisation; read-only scans per scan consent; no benchmark or publication clause; privacy: anonymous scan records kept up to 12 months. Same task, rules and anchors. The free email unlock (to see the full list) used [email protected] and counts as a step.
• A5 (post-run note, no scoring change): the canonical fix for V9 needed [email protected]; 4.17.21 still has newer advisories. Tag vuln-lab-v1-fixed moved to 1dbbe63. The V9 recheck is in ground-truth-fixed.json.
• A6 (2026-10-16 02:03 ET, CEO decision, before any re-run): false all-clears are penalised. Reason: a security tool that says “you’re safe” when you aren’t is the worst failure, worse than staying silent. A miss scores 0 in C1 and costs nothing in C2. A false all-clear also scores 0 in C1, and costs 2 penalty points in C2, twice a false positive. Applied retroactively to all r1 evidence: Prodable 2 (V1 “Row-Level Security looks correct”, V6 “API routes require authentication”); Supabase Security Advisor 0; Semgrep CE 0; VibeScan 0 (its report was marked “scan incomplete, evidence confidence low”, so per the definition it isn’t an all-clear). Disclosed on the page.
• A7 (2026-10-16 02:03 ET, CEO decision): the re-run threshold counts a 10-point gap as “within 10” (rubric §1.5). The top two after v1.1 scoring are re-run (r2) on the lab reset to vuln-lab-v1, and the scores averaged. Then the fixed state is restored.
• A8 (2026-10-16 02:03 ET, CEO approval): GitHub code scanning (CodeQL default setup) + secret scanning (incl. push protection) + Dependabot alerts added as a tool. They run on a separate public repo pathakarchit/bwt-vuln-lab: one commit of the vuln-lab-v1 app tree (no checks/). The throwaway project’s real anon key is replaced by a dummy (so D1 becomes a format-valid dummy anon JWT), and every real env value we hold is grepped for before pushing (must be zero). Steps are counted from repo creation (a reader’s repo already exists, so creating it isn’t counted). Push-protection blocks count as findings. The repo is archived after testing. Snyk: not tested (its terms ban publishing benchmarks). Aikido and Lovable: not tested yet.
• A9 (2026-10-16 04:30 ET, CEO decision, before Aikido’s scored scan): Aikido Security added as a tool, now that the user has installed Aikido’s GitHub app on pathakarchit/bwt-vuln-lab. Aikido scans a repo’s default branch only (Scanned branches, checked 2026-10-16). Its first scan ran while the default branch held the fixed version (main, 8e37e6d), so that scan isn’t the scored run. It’s kept as evidence. For the scored run (r1), branch vulnerable = commit 996d7c4 (the A8 tree: vuln-lab-v1 minus checks/, dummy anon key, dummy keys only; real-key check: zero) is the repo’s default branch, and Aikido’s scanned-branch setting points at it. Aikido picks up a default-branch change only at its nightly sync, so the branch is set in Aikido’s UI. The post-fix rescan (F4) points the scanned branch back at main. Only free-plan (Developer, $0) features count. The account is on a Pro trial. Anything only Pro or credits unlock is logged as a note and not scored: AI Code Analysis / AI Code Audit, which is credit-based, with credits included only from Pro (pricing, Wallet & Credits, checked 2026-10-16). AutoFix counts for C6 because the free plan includes 10 AI AutoFixes a month. Steps count from connecting GitHub (sign-up with GitHub, installing the app, choosing repos). Aikido MSA v1.3 has no benchmark clause (checked 2026-10-16).
• A10 (2026-10-16 04:30 ET, before Aikido’s scored scan): how to score Aikido’s “Path traversal attack possible” finding (SAST, src/App.jsx, supabase.storage.from('uploads').upload(${user.id}/${file.name})), seen in the setup scan. It names V5’s object, the uploads bucket’s upload path. The harm it describes, writing to or downloading files from other users’ folders, is exactly what V5 allows in v1, because the bucket is public with anonymous list and upload policies. But the cause and fix it gives are wrong. It blames unsanitised ../ in a browser-side file name. The browser code is attacker-controlled anyway, File.name carries no folder path, and the real fix is a private bucket with owner-folder storage policies (section 7). Per section 6 (“right object, vague or wrong problem”), it scores Partial (0.5) for V5 in r1, and isn’t also counted as a false positive. If it’s still raised in the F4 rescan on the fixed tree, where the owner-folder policies stop it, it counts as a new false positive there (F4 fails). Other rules are unchanged: the public anon key flagged at High is a false positive (D1). A dummy key that Aikido’s liveness check auto-ignores as “expired” is neither a catch nor a false all-clear: the key really is invalid, as the dummy-key caveat says. Findings that only appear in git history count when the planted secret is also in the scanned branch’s HEAD.
• A11 (2026-10-16 04:35 ET, protocol note, no rule change): in Aikido, saving the scanned-branch setting starts a scan by itself. So the r1 scan (branch vulnerable, 04:26:15–04:26:40 ET) and the F4 rescan (branch main, 04:29:49–04:30:18 ET) each started when the branch was saved, and Start Scan was never clicked. One scan per run still held.
• A3: the D4 bucket public-assets holds no file (the MCP can’t upload storage objects). Its config is the decoy: public read, no list or insert policy.