Test script: Take payments in your vibe-coded app

• Job slug: take-payments-in-app
• Version / status: v1 · frozen 2026-10-16 12:50 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
• Rubric version: v1 (rubric)
• Author / date: Lando (CEO), 2026-10-16

1. The job

Your AI-built app is ready to sell something: a template, a guide, lifetime access. You want to charge $24 once, send buyers to a checkout page you don’t have to build, find out reliably when someone has paid (a signed webhook your server can trust), and look the order up later. You’d rather not register for sales tax in every country yourself. You’re done when a test buyer has paid $24 with the vendor’s test card, your server has received and verified the signed “order paid” event, and the order shows up through the API.

2. Candidates

Scroll sideways for more columns →

3. Fixed inputs

• Product: “BuildWithThis Test Kit”, a one-time digital product priced $24.00 (2400 cents). If the store’s currency can’t be USD, the store currency at the closest round price, noted.
• Buyer: test email +payments-test on our own domain, billing country Canada, postal code M5V 2T6 (Toronto, Ontario: 13% HST applies to digital goods). Card: the vendor’s documented success test card (4242 4242 4242 4242, any future expiry, any CVC).
• Webhook receiver: one tiny endpoint for both tools that stores each request’s raw body and headers verbatim (a Supabase Edge Function on our free test project; the box can’t open inbound ports and a cloudflared quick tunnel couldn’t connect). Signatures are verified offline on the box with each vendor’s secret, using the vendor’s documented method, so the receiver never holds a secret.
• Check script: checks/payments-checks.mjs (Node 20, published without secrets), same steps for every tool. Purchase automation: Playwright (Chromium, headless) on the box.

4. Fixed task

The operator (agent) does this for each tool, following only the vendor’s official docs (each docs page used is logged):

Take a one-time $24 payment with <tool>, in test mode / sandbox:
1. Create a $24 one-time digital product (through the API if the tool has an endpoint for it;
   otherwise in the dashboard, counted as a manual intervention).
2. Generate a hosted checkout link for it through the API.
3. Complete a test purchase with the vendor's documented test card, headless if feasible.
4. Register a webhook endpoint with a signing secret, receive the order event, and verify its signature.
5. Fetch the order through the API.

5. Follow-up rules (agent operator version)

• Time cap: 60 active minutes per tool.
• Allowed: the vendor’s official docs, API reference, SDK references and error messages. Not allowed: third-party tutorials.
• Manual interventions (counted for C2): anything a person must do in a dashboard or browser (other than the test purchase itself), and any workaround the docs don’t cover. If a step needs a dashboard and our operator can’t reach one, the run is blocked, queued for a person, and the tool isn’t scored until it runs.

6. Steps

1. Record the start time and confirm the key is a test/sandbox key.
2. Run the check script: create the product, create the checkout link, register the webhook, run the headless purchase, wait up to 5 minutes for webhooks, verify, fetch the order, run the probes.
3. Fix and rerun until all core checks pass or the cap is hit. End time = when the last core check first passes.
4. Record fees, merchant-of-record status, payout requirements and test/live separation from the vendor’s official pages, with the date.

7. Pass/fail checks

Scroll sideways for more columns →

8. Quality checklist (C5): developer experience

Scroll sideways for more columns →

9. Security checklist (C7)

Scroll sideways for more columns →

10. Weights and tie-breaker

• Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
• C2 anchors (manual interventions + problems the docs didn’t answer): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
• C3 anchors (T = 60 min): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
• C4 cost of the test: 5 = $0 with no card on file · 4 = $0 but a card was required · 1 = we had to pay. Transaction fees are reported side by side but not scored here (both candidates are pay-per-sale with no monthly fee).
• C5 and C7: checklist-count anchor (rubric §3).
• C6 portability: 5 = orders and customers can be listed through the API and exported as CSV from the dashboard · 4 = one of the two · 1 = neither.
• Second run: if the top two are within 10 points, both run again on fresh products and webhooks.
• Tie-breaker: lower total fee on a $24 sale to a Canadian card, then C7.

11. Cleanup

• Archive or delete the test products, checkout links and webhook endpoints after the verdict is published. Drop the webhook inbox table and function. No real money moves; test/sandbox only.

Amendments

• v1.1 (2026-10-16, during the runs; none changes how a tool is scored):

  1. Check script, page load (12:51 ET, j09-polar-r1): the first purchase attempt waited for the checkout page to go “network idle”, which never happens because the payment form keeps a connection open. It now waits for the page load plus 8 seconds. No check had run. Our problem, not Polar’s.
  2. Check script, address dropdowns (12:53 ET, j09-polar-r1): Polar’s country and state pickers are custom dropdowns, not plain <select> elements, so our first attempt left the country on “United States” and the form asked for a state. The script now opens each dropdown and picks Canada, then Ontario. The test card was never charged on that attempt. Our problem, not Polar’s.
  3. Lemon Squeezy blocked at P1 (12:56 ET, j09-lemonsqueezy-r1): Lemon Squeezy’s API has no endpoint to create a product or a variant (POST /v1/products returns 405 “Supported methods: GET, HEAD”), so the product has to be made in the dashboard, and our operator can’t reach the dashboard right now. Under §5 the run is blocked, queued for a person, and Lemon Squeezy is not scored until it runs. Nothing else was created in the store.
  4. Public script: published at /downloads/payments-checks.mjs, with keys read from environment variables (updated with the v1.2 script fixes).

• v1.2 (2026-10-16 afternoon, Lemon Squeezy unblocked; none changes how a tool is scored):

  1. Lemon Squeezy product made in the dashboard (before 15:06 ET): a person created “BWT Test Kit” (product 1445998, one variant, 2257367 from the API) in the test store. Counted as 1 manual intervention for Lemon Squeezy (C2), as §4.1 says. It’s priced CA$24.00, not US$24: the store currency defaulted to CAD when Canada was chosen as the store’s country at signup. Under §3 (“the store currency at the closest round price, noted”) 2400 in the store currency passes P1. It is a finding: a Canadian seller who wants to charge in US dollars has to change the store currency on purpose.
  2. Check script, Lemon Squeezy product lookup (15:06 ET): the script looked for the product by our planned name, “BuildWithThis Test Kit”. It now takes the product ID (LS_PRODUCT_ID) and reads the real name back for P2.
  3. Check script, webhook secret length (15:06 ET): our first webhook request used a 48-character secret and got HTTP 422, “The secret field must not be greater than 40 characters.” Lemon Squeezy’s signing docs say secrets are “normally” 6–40 characters. The error message was clear, so Q4 passes. The script now uses 40 characters. Our problem, not Lemon Squeezy’s.
  4. Check script, Lemon Squeezy checkout form (15:07–15:10 ET): four tries failed in our script before a payment was attempted: hidden copies of inputs, Stripe’s “save my info” (Link) box, and a Canada form that asks only for a postal code (no street, city or province). The card was never charged on those tries. The Lemon Squeezy success state is a “Thanks for your order!” dialog on the same URL. The script waits up to 90 s for a URL change first, so about 90 s of each Lemon Squeezy run is our script waiting. Our problem.
  5. P8 for Lemon Squeezy: Lemon Squeezy has one API host for test and live, so there’s no “live API” to call. The equivalent we ran was asking for a live checkout (test_mode: false) with the test key. It wasn’t rejected: Lemon Squeezy silently returned a test-mode checkout. The key couldn’t reach live mode, so we scored P8 as a pass, but the request was neither rejected nor flagged.
  6. Second runs (gap of 10 or less, rubric §1.5): Polar 91 vs Lemon Squeezy 87 after the first runs, so both ran again (j09-polar-r2 on a fresh product, checkout link and webhook; j09-lemonsqueezy-r2 on a fresh checkout and webhook with the same dashboard-made product, since the API can’t create another).

Scoring notes (2026-10-16)

• Results, both runs identical: Polar 91 (r1) and 91 (r2), average 91. Lemon Squeezy 87 (r1) and 87 (r2), average 87. The gap is 4, under 5, so the tie-breaker applies (rubric §4.3). Fees: a $24 sale on a Canadian card costs the same with either tool (5% + 50¢ + 1.5% = $2.06), so that’s a tie too. C7 decides it: Polar 5, Lemon Squeezy 3. Polar wins on the tie-breaker. It’s close.
• P7 fails for both: neither tool charged our Ontario test buyer any tax. Polar: $0 tax in both runs. Lemon Squeezy: the order shows tax_name: HST, tax_rate: 0.00 and $0 tax. Neither vendor’s docs explain the zero. In an unscored probe (no purchases), Polar’s sandbox did charge VAT for Sweden ($4.80) and the UK ($4.00) inside the $24, and $2.13 New York sales tax on top.
• Polar fee measured: 170 cents on each $24 order (platform_fee_amount). Lemon Squeezy’s order object shows no fee field, so its fee is from its published pricing only.
• Polar: C1 4 (7/8) · C2 5 (0 manual steps) · C3 5 (4 min 16 s; 2 min 43 s) · C4 5 · C5 5 (5/5) · C6 4 · C7 5 (4/4) = 91. C6: orders and customers list through the API, and undocumented GET /v1/orders/export and /v1/customers/export endpoints returned CSV files. The 5 anchor asks for a CSV export from the dashboard, which we didn’t check and Polar’s docs don’t describe, so 4. S1: Standard Webhooks signs webhook-id.webhook-timestamp.body. The SDK’s validateEvent and our own check accepted every delivery (15 in r1, 9 in r2) and rejected a one-byte change and a wrong secret.
• Lemon Squeezy: C1 4 (7/8) · C2 5 (1 manual step: product made in the dashboard) · C3 5 (6 min 11 s; 2 min 9 s, including about 90 s of our script waiting each time) · C4 5 · C5 4 (4/5: Q1 misses, no API to create products or variants) · C6 5 (orders and customers list through the API; the docs describe an orders CSV export from the dashboard’s Orders page, emailed to the store owner) · C7 3 (2 of 4 missed) = 87. S1 misses: the X-Signature HMAC covers the body only, with no timestamp, so a captured request could be replayed. S3 misses: its docs describe no scopes or expiry for API keys. S2 passes on the P8 rule in amendment 5. S4 passes: the checkout is hosted.
• Lemon Squeezy webhooks: one order_created per purchase, about 3 seconds after the order. Its signature verified with Lemon Squeezy’s documented HMAC; a one-byte change and a wrong secret were rejected.