Test script: Automate a workflow on your own server for free

Job slug: automate-workflow-self-hosted
Version / status: v1 · frozen 2026-10-16 15:45 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
Rubric version: v1 (rubric)
Author / date: Lando (CEO), 2026-10-16

1. The job

When someone signs up for your product, you want the boring follow-up done automatically: add them to your customer list, ping your team chat for paying plans, retry if a service hiccups, and run a health check every minute. You want it without writing an app, without a per-task bill, on a machine you control. You’re done when the workflow handles every event correctly, recovers from one failure, keeps running on schedule and survives a restart.

2. Candidates

Tool	Version	How you get it	Terms check (2026-10-16)	Run mode
n8n (self-hosted, Community edition)	latest on npm	`npx n8n` (Node.js), SQLite storage, as its docs show	Sustainable Use License: free for your own internal business use or non-commercial use; no benchmark clause.	Agent-driven
Node-RED	latest on npm	`npm install -g --unsafe-perm node-red`, then `node-red`, as its docs show	Apache-2.0. No benchmark clause.	Agent-driven

How the operator builds the workflow: a person would drag nodes in each tool’s visual editor. Our agent has no screen, so it writes the same workflow as the tool’s own JSON format, using only nodes and settings shown in the official docs, and installs it with each tool’s documented method (n8n: the n8n import:workflow and n8n update:workflow --active CLI commands; Node-RED: its flow file or Admin API POST /flows). No code or function nodes.

3. Fixed inputs

Local test services (same for both, on the test machine): POST /crm (customer list), POST /chat (team chat), GET /health. Each request is logged with a timestamp. /crm returns HTTP 500 the first time it sees the email retry.me@example.com, and 200 after that.
20 signup events (published): JSON {name, email, plan, signedUpAt}; 7 on plan pro, 13 on free. Some emails have capital letters. One is retry.me@example.com (plan pro).
Expected output: for every event, /crm gets exactly {"email": <email in lowercase>, "name": <name>, "plan": <plan>, "source": "webhook"}. For each pro event, /chat gets exactly {"text": "New Pro signup: <name> (<email in lowercase>)"}. The webhook replies 200 {"ok": true} after the workflow has run.

4. Fixed task

Using <tool>'s built-in nodes only, following its official docs:
1. A webhook at POST /webhook/signup receives the event.
2. Send the transformed record to /crm. If the call fails, retry it (wait at most 5 s, at most 3 tries).
3. If plan is "pro", send the chat message to /chat.
4. Reply 200 {"ok": true}.
5. A second flow calls GET /health every minute.

5. Follow-up rules

Time cap: 60 active minutes per tool, from nothing to the first full test pass. Package downloads logged but excluded, up to 15 minutes.
Allowed: official docs, node references and error messages. Manual interventions (C2): any step the docs don’t cover, any workaround, and any fix to the workflow after the first test pass starts (each fix counts once).

6. Steps

Record the start time; install following the docs; start with default settings except the port.
Build and install the workflow (§2); activate it.
Send the 20 events one after another; wait 150 s for the schedule; restart the tool; send 1 more event (pro, new email).
Run the check script on the service logs and webhook replies; record idle memory after the test.

7. Pass/fail checks

ID	Check	Pass if
F1 [core]	Installed without a screen	Workflow installed and active using only documented commands or APIs
F2 [core]	Customer list right	/crm received all 20 records, each exactly as expected (duplicates from retries allowed only for the failed call)
F3 [core]	Team chat right	/chat received exactly the 7 expected messages and nothing else
F4	Recovers from a failure	The record whose first /crm call got 500 arrives on a later try
F5	Replies fast	All 20 replies are 200 `{"ok": true}`, 95th percentile ≤ 1 s
F6	Runs on schedule	/health called at least twice in the 150 s wait
F7	Survives a restart	After restarting, the workflow is still active without changes and the 21st event is handled exactly as expected

8. Quality checklist (C5)

ID	Item	Pass if
Q1	Export	Workflow exports to a JSON file with a documented command, API or menu
Q2	Quickstart works as written	The docs’ install-and-start command runs unchanged
Q3	No code needed	The whole scenario is built without code or function nodes
Q4	Run history	Each run’s success or failure is recorded and viewable in the tool
Q5	Light	Memory after the test ≤ 500 MB

9. Security checklist (C7)

ID	Item	Pass if
S1	Editor locked by default	The editor asks for a login on a fresh install
S2	Secrets encrypted	Docs say saved credentials are encrypted at rest by default
S3	No telemetry by default	No usage data is sent unless you turn it on (docs and default settings)
S4	Hardening guide	Official docs explain how to run it safely on the internet (HTTPS, auth)

10. Weights and tie-breaker

Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10. C1: rubric anchor on the share of F1–F7 passed.
C2: 5 = 0–1 interventions · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more. C3 (T = 60): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
C4: 5 = $0 self-hosted with no execution limits. C5 and C7: checklist-count anchor.
C6 ownership: 5 = OSI open-source licence, self-hosted, workflows export as JSON · 4 = source-available “fair-code” licence that’s free for internal use but limits reselling or hosting it for others · 1 = cloud only.
Tie-breaker: C1, then C7.

11. Cleanup

Stop both tools; delete their installs and data folders after publishing. Keep logs, workflow JSON and check output.

Amendments

v1.1 (2026-10-16, during the runs; none changes how a tool is scored):
1. n8n activation command. n8n 2.0 replaced the update:workflow --active command named in §2 with publish:workflow --id=<ID> (the old one is deprecated). We used the documented replacement. Both commands only take effect after a restart, as the docs say, so we imported and published with n8n stopped, then started it. Not an intervention.
2. Node.js. Both tools ran on Node.js 22.23.3 (installed from nodejs.org for the Readability test the same day; the machine’s Node 20 is too old for current n8n). Node-RED’s npm install -g went to the machine’s existing npm prefix (~/.local). Node-RED was started with -u <folder> (a documented flag) so each run had its own fresh data folder; otherwise default settings.
3. Check script fix (before any score was recorded). The first version of our script computed the 95th percentile of the 20 reply times as the slowest reply instead of the 19th (nearest-rank). Fixed to nearest-rank; n8n’s run 1 went from a false F5 fail to a pass (p95 0.12 s; the slowest reply, 1.1 s, was the event that waited for the retry).
4. Harness errors, re-done (no tool fault). In n8n run 1 our first restart signal went to the npx wrapper, not to n8n, so n8n never stopped and the 21st event went to the old process. We stopped n8n properly, restarted it, and sent the 21st event again; F7 is scored on that, and F2/F3 only count calls before the first attempt. Node-RED run 2 was aborted twice by harness problems (our shell session was interrupted; then our stop command matched the harness itself) and was re-run from a fresh data folder. Logs of the aborted attempts are kept.
5. Q5 memory. n8n starts a separate task-runner process by default; we count it. Run 1: 442 MB main + 108 MB runner = 550 MB. Run 2 recorded only the main process (448 MB); the runner is assumed similar. Node-RED is one process: 112 MB (run 1), 109 MB (run 2).
6. Second runs: the first-run scores were 2 points apart (Node-RED 96, n8n 94), so both ran again from fresh data folders. Second runs gave the same check results and scores. Averages: Node-RED 96, n8n 94. The tie-breaker (C1, then C7) doesn’t separate them: both C1 = 5 and both C7 = 4. The script names no third tie-breaker, so we fall back to the rubric’s main rule (highest weighted score): Node-RED. The page says it’s close.

Scoring notes (2026-10-16)

Node-RED 5.0.8 (nodered-r1, nodered-r2), 96 both runs: 7 of 7 both runs. 20 of 20 CRM records exact, 7 of 7 chat messages, the 500 retried after 1 s and succeeded, all replies 200 {"ok":true} (median 3.6 ms, p95 46 / 42 ms), /health called twice in 150 s, still running after restart and the 21st event exact. C2 = 5 (0 interventions). C3 = 5 (under 15 minutes). C4 = 5. C5 4 of 5 → 4: export via GET /flows and the Export menu ✔, npm install -g --unsafe-perm node-red then node-red ran ✔, no function nodes (change nodes with JSONata expressions, switch, delay) ✔, no run history ✘ (Node-RED keeps no record of past runs; only the debug sidebar), 112 MB ✔. C6 = 5 (Apache-2.0). C7 3 of 4 → 4: editor open with no login on a fresh install ✘ (GET /flows returned 200 without credentials; adminAuth is commented out in the default settings), credentials encrypted with a system-generated key ✔, telemetry off until you consent in the editor ✔ (settings.js), securing guide ✔.
n8n 2.35.7 Community edition (n8n-r1, n8n-r2), 94 both runs: 7 of 7 both runs. Same results (median reply 63 / 64 ms, p95 123 ms; /health 2 and 3 times in 150 s). C2 = 5, C3 = 5, C4 = 5 (no execution limits self-hosted). C5 4 of 5 → 4: n8n export:workflow ✔, npx n8n ran ✔, no Code nodes (built-in nodes with n8n expressions) ✔, execution history saved and viewable ✔ (28 executions recorded in run 1), 550 MB after the test ✘. C6 = 4 (Sustainable Use License, fair-code). C7 3 of 4 → 4: editor requires creating an owner account and logging in ✔ (note: whoever opens a fresh instance first creates that account), credentials encrypted with a key generated on first launch ✔, sends diagnostics, update and template data to n8n by default ✘ (docs: “By default, a self-hosted n8n instance sends data to n8n’s servers”), security guide ✔.