Test script: Turn web pages into clean text for your AI

Job slug: web-page-to-clean-text
Version / status: v1 · frozen 2026-10-16 15:45 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
Rubric version: v1 (rubric)
Author / date: Lando (CEO), 2026-10-16

1. The job

Your agent, scraper or RAG pipeline needs the actual article text from web pages, without menus, cookie banners, related links and footers eating your tokens or confusing the model. You’re done when the tool returns the main text of almost every page, keeps almost all of it, and adds almost no junk.

2. Candidates

Tool	Version	How you get it	Terms check (2026-10-16)	Run mode
Trafilatura	latest on PyPI, `trafilatura.extract()` with default settings	`pip install trafilatura`	Apache-2.0. No benchmark clause.	Agent-driven
Mozilla Readability (Readability.js, the code behind Firefox Reader View)	latest on npm, `new Readability(doc).parse()` with jsdom, as its README shows; we score `textContent`	`npm install @mozilla/readability jsdom`	Apache-2.0. No benchmark clause.	Agent-driven

3. Fixed inputs

The 181 saved news and blog pages and their hand-labelled article text from Zyte’s open article-extraction-benchmark (MIT, commit 4a3bc97). The HTML files are read from disk; no page is fetched live.
Disclosure: the benchmark is run by Zyte, which sells an extraction API, and already lists published scores for both tools; Trafilatura’s author contributed its runner there. We run both tools ourselves, with current versions and default settings, and add our own practical checks.

4. Fixed task

Using only <tool>'s official README: install it, and for each of the 181 HTML files, extract the main text with default settings. Save the text per page.

5. Follow-up rules

Time cap: 60 active minutes per tool to the first full set of results. Allowed: official README, docs and error messages. Manual interventions (C2): any step the docs don’t cover.

6. Steps

Record the start time; install following the README.
Extract all 181 pages in one process with the network switched off; record time per page and peak memory (/usr/bin/time -v).
Score with the benchmark’s own evaluate.py matching (text shingles, so word order counts), plus the checks below; save JSON.

7. Pass/fail checks

F1, precision and recall per page come from the benchmark’s string_shingle_matching; we report the means over 181 pages.

ID	Check	Pass if
W1 [core]	Returns text	Non-empty text for ≥ 95% of pages
W2 [core]	Accurate overall	Mean F1 ≥ 0.90
W3	Little junk	Mean precision ≥ 0.90
W4	Keeps the article	Mean recall ≥ 0.90
W5	Consistent	≥ 90% of pages with F1 ≥ 0.80
W6	Fast	Median ≤ 50 ms per page on one process
W7	Light	Peak memory ≤ 1 GB for the whole run

8. Quality checklist (C5)

ID	Item	Pass if
Q1	Markdown output	Documented option for Markdown (headings, lists, links)
Q2	Fetches pages itself	Documented one-line way to download and extract a URL
Q3	Metadata	Documented title, author and date extraction
Q4	Quickstart works as written	The README example runs unchanged
Q5	Command line	Official CLI for batch use without writing code

9. Safety checklist (C7)

ID	Item	Pass if
S1	Offline	Extraction from saved HTML works with the network switched off
S2	No telemetry	Nothing in the docs or the network-off run shows usage data being sent
S3	Official source	Installed from the project’s official package registry over HTTPS
S4	Safe output	No `<script` tags or `javascript:` URLs in the tool’s output for any of the 181 pages (its HTML output if it has one, else its text)

10. Weights and tie-breaker

Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10. C1: rubric anchor on the share of W1–W7 passed.
C2: 5 = 0–1 interventions · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more. C3 (T = 60): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
C4: 5 = $0, no usage limits. C5 and C7: checklist-count anchor. C6: 5 = permissive open source, runs offline · 4 = copyleft · 1 = hosted only.
Tie-breaker: C1, then mean F1.

11. Cleanup

Delete environments and the benchmark copy after publishing. Keep outputs, logs and check JSON.

Amendments

v1.1 (2026-10-16, during the runs; none changes how a tool is scored):
1. Readability needs Node 22. The current jsdom (30.1.3), which Readability’s README uses in Node.js, requires Node 22.22.2 or newer (its engines field), and failed to load on the test machine’s Node 20. We installed Node 22.23.3 from nodejs.org (checksum verified). That’s a documented requirement, so it is not counted as an intervention; C2 would be 5 either way (0–1 interventions).
2. Trafilatura’s quickstart (Q4). Its README example downloads a GitHub blog post with fetch_url. On our test machine the download was refused with “SSRF protection: connection to non-public address blocked”, because the machine’s network proxy resolves every hostname to a non-public address (198.18.0.1). We saved the same page with curl and ran the README’s extract(...) lines unchanged on it; they returned the article, and the JSON output with title, author and date. We counted Q4 as passed and note the guard as a safe default. Readability’s README example ran unchanged.
3. Trafilatura’s HTML output for S4 was produced in a second, untimed pass inside the same process, so its peak memory figure includes both passes (an upper bound). Readability’s HTML is the content field of the same parse.
4. W2: the benchmark’s headline F1 is computed from mean precision and mean recall. We report that and also the mean of per-page F1; both tools pass W2 on either.
5. Second runs: not needed; the scores are 20 points apart.

Scoring notes (2026-10-16)

Trafilatura 2.3.1 (traf-r1), 100: 7 of 7. Text on 181 of 181 pages. F1 0.953 (precision 0.928, recall 0.978; mean per-page F1 0.943). 93.9% of pages at F1 ≥ 0.80. Median 11.5 ms per page, peak 97 MB. C2 = 5 (no interventions), C3 = 5 (a few minutes). C5 5 of 5: Markdown output (output_format="markdown"), fetch_url and trafilatura -u <url>, title/author/date with with_metadata=True, quickstart ran (saved page, see v1.1), official CLI. C6 = 5 (Apache-2.0). C7 4 of 4: network-off run, nothing in the package or docs sends usage data, PyPI, no <script or javascript: in any page’s HTML output. Published benchmark score for 2.0.0 was F1 0.958; our 0.953 on 2.3.1 is within its ± 0.006.
Mozilla Readability 0.6.0 with jsdom 30.1.3 (rjs-r1), 80, runner-up: 5 of 7. Text on 181 of 181 pages. F1 0.947 (precision 0.914, recall 0.982; mean per-page F1 0.943). 95.6% of pages at F1 ≥ 0.80. Median 60.4 ms per page (parsing with jsdom plus Readability) ✘ W6. Peak 1,338 MB over the 181 pages ✘ W7 (we followed the README, which does not close jsdom windows). Same F1 as the benchmark’s published 0.947 for 0.6.0. C1 = 3 (71%). C2 = 5, C3 = 5. C5 2 of 5 → 2: no Markdown ✘, doesn’t fetch pages ✘, title/byline/published time ✔, README example ran ✔, no official CLI ✘. C6 = 5 (Apache-2.0). C7 4 of 4.