Test script: Send emails from your app

Job slug: send-emails-from-app
Version / status: v1 · frozen 2026-10-16 12:44 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
Rubric version: v1 (rubric)
Author / date: Lando (CEO), 2026-10-16

1. The job

Your AI-built app needs to send email: sign-up confirmations, password resets, receipts. You want it sent from your own domain, landing in the inbox, with the key kept on your server. You’re done when the app’s send endpoint delivers mail from your domain to a real inbox and the checks below pass.

2. Candidates

Tool	Plan	How the operator gets an account	ToS check (2026-10-16)	Run mode
Resend	Free	Browser signup by a person (hCaptcha), then an API key	Terms and acceptable use: no benchmark or automation clause; bans spam and cold outreach.	Agent-driven
Postmark	Free developer plan (new accounts start in test mode)	Browser signup by a person (reCAPTCHA), then the default server’s API token	Terms (effective 2024-12-10): access only through Postmark’s standard APIs; no benchmark clause. We use only the official API and SDK.	Agent-driven
Brevo	Free	Browser signup	Not checked	Not tested: signup asks for an organization street address before any API access. Our test identity has none. Onboarding finding, not a score.
Amazon SES	Free tier	AWS account	Not checked	Not tested: needs an AWS account with a payment card.

3. Fixed inputs

Test domain: mail-test.buildwiththis.com, a subdomain used only for this test. DNS records go only under it. The root domain’s mail, SPF, DKIM, DMARC and newsletter records are not touched. A DMARC record v=DMARC1; p=none is added for the subdomain.
Sender: noreply@mail-test.buildwiththis.com. Recipients: only our own inbox addresses at buildwiththis.com (+mailtest and plus-variants). No email goes to anyone else.
Message (same for every tool): subject BWT email test <run-id> <n>, an HTML part and a plain-text part, a Reply-To of +reply, and for E5 a fixed 20 KB PDF attachment (SHA-256 in the evidence).
App: one Node 20 server endpoint, POST /api/send (JSON {to, subject, html, text, replyTo, attachments?}) → 200 {id}, plus POST /api/send-batch for E6. It uses the vendor’s official Node SDK and reads the key from the server environment.
Check script: checks/email-checks.mjs, same for every tool. Inbox arrival is read from the receiving mailbox.

4. Fixed task

The operator (agent) implements this, following only the vendor’s official docs (each docs page used is logged):

Send email from the app with <tool>, on the free plan:
1. Add and verify the sending domain mail-test.buildwiththis.com using the DNS records the tool gives you.
2. Implement POST /api/send and POST /api/send-batch with the tool's official Node SDK.
3. Send from noreply@mail-test.buildwiththis.com. Keep the API key on the server.

5. Follow-up rules (agent operator version)

Time cap: 60 active minutes per tool. Waiting for DNS to propagate or for the vendor to verify the domain is logged separately and excluded from active time, up to 30 minutes.
Allowed: the vendor’s official docs, SDK references and error messages. Not allowed: another tool’s code, third-party tutorials.
Manual interventions (counted for C2): anything a person must do in a dashboard or browser after signup and key creation (including a vendor’s manual account review), and any workaround the docs don’t cover.

6. Steps

Record the start time. Log plan, limits and cost.
Add the domain, publish the records under mail-test, and wait for verification.
Implement the endpoints (§3, §4). Run the check script and read the inbox. Save the JSON output.
Fix and rerun until all core checks pass or the cap is hit. End time = when the last core check first passes.
Record lines of code, dependencies, docs pages used, and the free-plan limits and prices from the vendor’s pricing page, with the date.

7. Pass/fail checks

ID	Check	How to test	Pass if	Evidence
E1 [core]	Send through the API	`POST /api/send` with the fixed message	The vendor accepts it and returns a message id	JSON log
E2 [core]	Own domain verified	Vendor’s domain status after adding its records under `mail-test`	Verified, with DKIM, using only records under the subdomain	API/dashboard status, DNS export
E3 [core]	Lands in the inbox	Look for the message in the receiving mailbox	Arrives within 120 s, From = `noreply@mail-test.buildwiththis.com`, delivered (not quarantined)	Mailbox record
E4	Both parts and Reply-To	Read the received message	Text body readable and Reply-To kept (as far as the mailbox shows)	Mailbox record
E5	Attachment	Send the fixed PDF	Arrives and the SHA-256 matches	Mailbox record, hash
E6	Batch send	`POST /api/send-batch` with 5 messages to 5 plus-addresses	All 5 accepted in one API call and all 5 arrive	JSON log, mailbox
E7 [G2]	Key stays on the server	Search everything the app serves for key values and key-shaped strings	None found	grep output

8. Quality checklist (C5): developer experience

ID	Item	Pass if
Q1	The official Node quickstart works as written	Yes
Q2	Domain setup lists the exact DNS records and verification finishes within 30 minutes	Yes
Q3	A safe way to test without emailing real people is documented (test mode, test addresses or a sandbox token)	Documented
Q4	Errors hit during the run were explained by the docs or the error message	Yes
Q5	The pricing page states free-plan limits and what happens at the limit	Yes

9. Security checklist (C7)

ID	Item	Pass if
S1	Scoped keys	You can create a key that can only send (not manage the account or domains)
S2	Domain authentication guided	The tool gives DKIM plus SPF or return-path records for your domain, and its docs recommend DMARC
S3	Tracking off unless you turn it on	Open and click tracking are off by default
S4	Abuse guard for new accounts	A new account can’t email strangers until the domain is verified or the account is reviewed

10. Weights and tie-breaker

Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
C2 anchors (manual interventions + problems the docs didn’t answer): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
C3 anchors (T = 60 active min): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
C4: rubric anchors. “Free tier enough” means the free plan can send the run’s emails and the plan lets you email your real users without paying.
C5 and C7: checklist-count anchor (rubric §3).
C6 portability: 5 = standard SMTP relay on the free plan and the domain records stay yours (switching = new DNS records) · 4 = SMTP only on paid plans · 3 = API only · 1 = none.
Tie-breaker: C1, then C2 (getting to your first real user with the least friction).

11. Cleanup

Remove the vendor domains and the mail-test DNS records after the verdict is published. Keys stay in each vendor’s account and our private test environment.

Amendments

v1.1 (2026-10-16, during the runs; none changes how a tool is scored):
1. Keys (12:45 ET, before either run): each vendor’s send key can’t add a domain (Resend “Sending access” key: API 401 restricted_api_key; Postmark server token: 401, needs an Account token). The operator got a Resend Full-access key and the Postmark Account token, used them only to add and verify the domain, and ran the app with the send-only key / server token. Creating keys is part of signup for both, so it isn’t counted for C2.
2. Postmark test mode (12:49 ET, j08-postmark-r1): every send was rejected with error 412: while the account is pending approval, recipients must share the From address’s domain (mail-test.buildwiththis.com). Our recipients are on buildwiththis.com, as a real app’s users would be on other domains. Postmark’s staff review is a manual step a person has to request. We counted it as one manual intervention and, as instructed, didn’t wait for it. The checks it blocks are marked failed for this run.
3. Supplementary, unscored (12:49 ET): to show what works in test mode, we re-ran Postmark’s checks with the confirmed root-domain sender signature as From (allowed in test mode because the recipients share its domain). It isn’t scored because the frozen sender is noreply@mail-test.buildwiththis.com.
4. Second runs: not needed. The scores are 44 points apart (rule: rerun the top two if within 10).

Scoring notes (2026-10-16)

Resend (j08-resend-r1), 100: domain verified 2 min 22 s after we asked (waiting excluded from active time). All 7 checks passed on the first try with the send-only key; every message was in the inbox in the same second Resend logged it delivered. Q5: the pricing page lists the free limits (3,000 a month, 100 a day) and says what happens if you go over (you’re notified and offered an upgrade). S3: the domain object shows open and click tracking off, as Resend’s tracking docs say.
Postmark (j08-postmark-r1), 56, fails G1: DKIM and return-path verified 21 s after we added the records (E2 ✔), and no key reached the app’s output (E7 ✔). Every send was rejected with 412 (pending approval), so E1 and E3–E6 fail. C2 = 5: one manual intervention, the account approval. C3 = 1 and C4 = 1: the core checks never passed in the run, so we used the “hit the cap” and “couldn’t finish” anchors. C5 5/5: the 412 message says exactly what’s wrong; the pricing page says the free plan is 100 emails a month with no overages. C6 = 5: SMTP is active on the free server. C7 4/4: tracking is off on the server (opens off, links None). Postmark says reviews take less than 24 hours on weekdays.
Postmark supplementary (unscored): all 7 sends accepted, Postmark logged them delivered at 12:49:21 ET, and all 7 were in the inbox with Reply-To kept and the PDF’s SHA-256 matching.