buildwiththis home

Test script: Extract tables from PDFs for free

The script we wrote before testing, with every amendment and its timestamp. The verdict it produced: Best free tool to extract tables from PDFs: Docling vs pdfplumber.

Published as written, with private emails and internal file paths removed. Times are Eastern. How we score: the rubric.

  • Job slug: extract-tables-from-pdf
  • Version / status: v1 · frozen 2026-10-16 15:45 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
  • Rubric version: v1 (rubric)
  • Author / date: Lando (CEO), 2026-10-16

1. The job

You have PDFs (invoices, reports, price lists, statements) and need the tables inside them as rows and columns you can put in a spreadsheet or feed to your app or an AI. You’re done when the tool finds every table and the cells come out right, including tables without borders, cells with wrapped text, tables that run over a page, and a scanned page.

2. Candidates

ToolVersionHow you get itTerms check (2026-10-16)Run mode
pdfplumberlatest on PyPI, page.extract_tables() with default settingspip install pdfplumberMIT. No benchmark clause.Agent-driven
Docling (IBM)latest on PyPI, DocumentConverter with default settings (CPU)pip install docling; models download on first useMIT (code); models from its documented Hugging Face repos. No benchmark clause.Agent-driven

3. Fixed inputs

7 PDFs with 8 tables, made for this test from fixed CSV files (the CSVs are the ground truth; files and SHA-256 sums published with the results). Pages are A4/Letter, made with headless Chrome:

  • T1 invoice table with full borders (6×4). T2 borderless financial table, right-aligned numbers with commas, minus signs and % (9×5). T3 striped table with horizontal lines only (12×6). T4 table whose cells wrap over several lines (6×3). T5 a 61-row table that runs over 2 pages (61×4). T6 and T7 two tables on one page with paragraphs between them (4×3, 5×3). T8 a scanned page: the page turned into a 150 dpi grey image, rotated 0.6° with light noise, with no text layer (6×4).
  • Every page also has paragraph text, so a tool has to tell text apart from tables.

4. Fixed task

Using only <tool>'s official docs, on CPU: install it, run it on each PDF, and save every table it finds as rows of cell text.

5. Follow-up rules

  • Time cap: 60 active minutes per tool to the first full set of results. Downloads (packages, models) logged but excluded, up to 15 minutes.
  • Allowed: official README, docs and error messages. Manual interventions (C2): any step the docs don’t cover, including installing an OCR engine they don’t mention.

6. Steps

  1. Record the start time; install following the docs.
  2. Run all 7 PDFs with the network switched off (after model downloads); record time per page and peak memory (/usr/bin/time -v).
  3. Run the check script; save its JSON.

7. How a table is scored

  • Cell text is normalized: whitespace collapsed (line breaks inside a cell become spaces), trimmed.
  • Each ground-truth table is matched to the extracted table from the same page(s) with the most matching cells. For T5, all tables a tool returns for pages 1–2 of that PDF are joined in order (repeated header rows dropped), for both tools.
  • Cell accuracy = ground-truth cells whose text appears at the same row and column ÷ all ground-truth cells. To forgive an extra empty row or column at the edge, the best alignment with up to 2 rows and 2 columns of offset is used.
  • A table is found if its cell accuracy is ≥ 50%. Extra tables = tables returned that match no ground-truth table.

8. Pass/fail checks

IDCheckPass if
X1 [core]Finds the tables≥ 7 of 8 found
X2 [core]Cells right on digital tablesAverage cell accuracy over T1–T7 ≥ 90%
X3No bordersT2 cell accuracy ≥ 90%
X4Wrapped cellsT4 cell accuracy ≥ 90%
X5Runs over a pageT5 cell accuracy ≥ 90%
X6Scanned pageT8 cell accuracy ≥ 80%
X7Speed and memory≤ 10 s per page on average (8 vCPU) and peak memory ≤ 4 GB

9. Quality checklist (C5)

IDItemPass if
Q1Spreadsheet-ready outputTables to CSV, Excel or a dataframe with one documented call
Q2Quickstart works as writtenThe README example runs unchanged
Q3Doesn’t invent tables≤ 2 extra tables across all 7 PDFs
Q4TuningDocumented settings to adjust table detection
Q5Other document typesCan also read Word, HTML or images (documented)

10. Privacy and safety checklist (C7)

IDItemPass if
S1Files stay on your computerWorks with the network switched off (after model downloads)
S2No telemetryNothing in the docs or the network-off run shows usage data being sent
S3Official sourcePackages and models from the project’s official, documented locations over HTTPS
S4Licence allows commercial useCode and models

11. Weights and tie-breaker

  • Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10. C1: rubric anchor on the share of X1–X7 passed.
  • C2: 5 = 0–1 interventions · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more. C3 (T = 60): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
  • C4: 5 = $0 with no usage limits (runs locally). C5 and C7: checklist-count anchor.
  • C6: 5 = permissive open-source code and models, runs offline, standard output · 4 = copyleft · 3 = model licence restricts commercial use · 1 = hosted only.
  • Tie-breaker: C1, then average cell accuracy over all 8 tables.

12. Cleanup

  • Delete virtual environments and downloaded models after publishing. Keep outputs, logs and check JSON.

Amendments

  • v1.1 (2026-10-16, during the runs; none changes how a tool is scored):
    1. Install: we installed both with uv pip instead of pip. For Docling’s CPU-only PyTorch, its docs give pip install docling --extra-index-url https://download.pytorch.org/whl/cpu; uv pip needs --index-strategy unsafe-best-match added to do the same. Not counted as an intervention.
    2. Models: Docling downloaded its layout, table and OCR models on a first conversion of one file with the network on (32 s, excluded as download time). The scored run then ran with the network off (unshare -rn), as did pdfplumber’s.
    3. Q2 (Docling quickstart): the README example converts an arXiv URL. On our test machine it failed with URL is not allowed, because the machine’s network proxy resolves every hostname to a non-public address (198.18.0.1) and Docling refuses to fetch from non-public addresses, a safety guard against server-side request forgery. The same example with a local file path (the README says “a local path or URL”) ran unchanged. We counted Q2 as passed and note the guard as a safe default.
    4. Second runs: not needed. The scores are 34 points apart, and pdfplumber fails core checks.

Scoring notes (2026-10-16)

  • Docling 2.140.0 (docling-r1), 100: 7 of 7. Found all 8 tables with 100% cell accuracy, including the borderless table, wrapped cells, the table that runs over 2 pages (returned as 2 tables and joined by our rule), and the scanned page (OCR with RapidOCR, every cell right). No extra tables. 2.3 s per page (18.8 s for 8 pages, plus about 4 s to load models), 1.5 GB peak. C2 = 5 (no interventions), C3 = 5 (about 1 minute from install to results, model download excluded). C5 5 of 5: export_to_dataframe in one call; quickstart (local file); no extra tables; documented pipeline options (e.g. table structure mode, OCR engine); reads DOCX, PPTX, HTML, images and more. C6 = 5 (MIT code; models Apache-2.0 / CDLA-Permissive-2.0). C7 4 of 4.
  • pdfplumber 0.11.10 (plumber-r1), 66, fails G1: 2 of 7. With default settings (extract_tables() looks for ruling lines), it got the fully bordered tables right (T1, T4, T6: 100%) but returned nothing for the borderless table (T2) or the 2-page table (T5), split the striped table into one table per row (T3: 8%), got 20% of T7, and nothing from the scanned page (no OCR). Found 3 of 8 tables; average 47% on digital tables. 7 extra tables. 0.03 s per page, 46 MB peak. C2 = 5, C3 = 5. C5 2 of 5 → 2: no CSV or dataframe export call ✘, README example ran ✔, 7 extra tables ✘, documented table_settings ✔, PDF only ✘. C6 = 5 (MIT). C7 4 of 4.