Test script: Caption a video or podcast for free

Job slug: transcribe-video-to-captions
Version / status: v1 · frozen 2026-10-16 15:07 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
Rubric version: v1 (rubric)
Author / date: Lando (CEO), 2026-10-16

1. The job

You publish videos or a podcast and want accurate captions (an .srt file) and a text transcript, without paying per minute and without uploading your recordings anywhere. You’re done when the tool turns a recording into a caption file that a video platform accepts, with few enough errors that you’d only skim it before publishing.

2. Candidates

Free tools that run on your own computer (no account, no upload). Hosted transcription services need a browser signup we can’t do in this run; they’re listed under “Not tested yet” on the page.

Tool	Version / model	How you get it	Terms check (2026-10-16)	Run mode
Whisper, run with faster-whisper	faster-whisper (latest on PyPI), model `small.en`, int8 on CPU	`pip install faster-whisper`; the model downloads from Hugging Face on first use	faster-whisper: MIT. Whisper model weights: MIT (OpenAI). No benchmark clause.	Agent-driven
Vosk	vosk (latest on PyPI), model `vosk-model-en-us-0.22-lgraph`	`pip install vosk`; model zip from alphacephei.com/vosk/models	Vosk and its models: Apache 2.0. No benchmark clause.	Agent-driven

Model rule: each tool’s largest English model that runs in 2 GB of RAM on a laptop CPU (a typical solo builder has no GPU). For Whisper that’s small.en; for Vosk it’s the lgraph model (its big en-us-0.22 model needs far more memory).

3. Fixed inputs

Set C (clean read speech): the 73 utterances of the LibriSpeech clean validation sample published as hf-internal-testing/librispeech_asr_dummy (LibriSpeech, CC BY 4.0), with their human transcripts.
Set N (noisy): Set C with pink noise mixed in at 10 dB signal-to-noise ratio, random seed 16.
Set S (spontaneous speech): the first 40 segments of the Earnings-22 test set (distil-whisper/earnings22, config chunked, split test, shard 0; real company earnings calls with many accents; CC BY-SA 4.0), with their human transcripts.
Episode: Set C joined into one 16 kHz mono WAV with 1.0 s of silence between utterances, so we know exactly when each utterance starts.
All audio is 16 kHz mono. SHA-256 of every input file is recorded in the evidence.

4. Fixed task

Using only <tool>'s official README and docs, on CPU:
1. Install it and its model.
2. Transcribe every file in sets C, N and S to text, and the episode with word timestamps.
3. Turn the episode's word timestamps into an .srt caption file.

Step 3 uses one shared caption builder for both tools (cues.py, part of our check script) so the caption layout is identical: a new cue at a pause over 0.6 s, a sentence end, 42 characters per line, 2 lines or 7 seconds, whichever comes first.

5. Follow-up rules (agent operator version)

Time cap: 60 active minutes per tool to get from nothing to the first valid .srt. Model download time is logged but excluded, up to 15 minutes.
Allowed: the tool’s official README, docs and error messages. Not allowed: third-party tutorials or forks.
Manual interventions (counted for C2): any step the official docs don’t cover (an undocumented system package, a manual model fix, a workaround for an error).

6. Steps

Record the start time. Create a fresh Python virtual environment for the tool.
Install and download the model following the official docs. Log every docs page used.
Run the transcription script; record wall-clock time, audio duration and peak memory (/usr/bin/time -v).
Run the check script on the outputs. Save its JSON.
End time = when the first valid .srt exists. Record versions, model size and licenses.

7. Pass/fail checks

WER = word error rate after both the human transcript and the tool’s output go through the same normalizer (the whisper_normalizer package’s English normalizer: lowercase, punctuation removed, spelled-out numbers turned into digits). Lower is better.

ID	Check	How to test	Pass if	Evidence
K1 [core]	Valid caption file	Parse the episode `.srt`	Parses; cues in order, no overlaps; every cue ≤ 7 s and ≤ 2 lines of ≤ 42 characters	check JSON
K2 [core]	Accurate on clean speech	WER on set C	≤ 10%	check JSON
K3	Holds up in noise	WER on set N	≤ 15%	check JSON
K4	Holds up on real conversation	WER on set S	≤ 25%	check JSON
K5	Captions in sync	For each episode utterance, the first word the tool timestamps inside that utterance’s window	≥ 90% of utterances start within 1.0 s of the true start	check JSON
K6	Fast enough	Processing time ÷ audio length, sets C+N+S and the episode, 8 vCPU	≤ 0.5 (an hour of audio in 30 minutes or less)	timing log
K7	Fits a normal laptop	Peak memory while transcribing	≤ 2 GB	`time -v` output

8. Quality checklist (C5): readability of what you get

ID	Item	Pass if
Q1	Punctuation and capitals	The raw transcript of set C has sentence punctuation and capital letters (at least one `.`, `?` or `!` per 40 words)
Q2	Caption file without code	The tool can write `.srt` or `.vtt` itself (CLI or one documented call), without our builder
Q3	Quickstart works as written	The README’s install-and-transcribe example runs unchanged
Q4	Word timestamps	Word-level start and end times are available
Q5	Other languages	Free models for at least 10 languages are documented

9. Safety and privacy checklist (C7)

ID	Item	Pass if
S1	Audio stays on your computer	Transcription works with the network switched off (after the model is downloaded)
S2	No telemetry	Nothing in the docs or a network check during the run shows usage data being sent
S3	Official, versioned model source	The model comes from the project’s official, documented location over HTTPS
S4	Licence clear for commercial use	Code and model licences are stated and allow commercial use

10. Weights and tie-breaker

Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
C1: rubric anchor on the share of K1–K7 passed.
C2 anchors (manual interventions): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
C3 anchors (T = 60 active min): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
C4: 5 = $0 with no limits on how much you transcribe (runs locally) · rubric anchors otherwise.
C5 and C7: checklist-count anchor (rubric §3).
C6 ownership: 5 = open-source code and model with a licence that allows commercial use, runs offline, standard output files · 3 = runs locally but the model licence restricts commercial use · 1 = hosted only.
Tie-breaker: C1, then the lower WER on set S (real conversation is what most podcasts sound like).

11. Cleanup

Delete the virtual environments and downloaded models from the test machine after the verdict is published. Keep transcripts, check JSON and timing logs as evidence.

Amendments

v1.1 (2026-10-16, during the runs; none changes how a tool is scored):
1. Memory measurement (15:08 ET, before the first run): GNU time wasn’t installed on the test machine; we installed it so K7 uses /usr/bin/time -v as written.
2. faster-whisper crash (15:08 ET, j10-whisper-r1): the first transcription crashed inside faster-whisper’s audio decoder: TypeError: open() got an unexpected keyword argument 'metadata_errors'. pip install faster-whisper had pulled in PyAV 19.0.1, which faster-whisper 1.2.1 doesn’t work with. We pinned av<19 (18.1.0) and reran. Counted as one manual intervention (C2) and as a Q3 miss.
3. Network off: every scored run ran inside a network namespace with no network (unshare -rn), which is how S1 and S2 were checked.
4. Second runs: not needed. The scores are 28 points apart, and Vosk fails a core check (rule: rerun the top two if within 10).

Scoring notes (2026-10-16)

Whisper via faster-whisper (j10-whisper-r1), 96: 7 of 7 checks. WER 7.7% clean, 8.1% with noise, 13.2% on earnings-call speech. All 73 captions started within 1.0 s of the true start. 221 s to process 1,747 s of audio (0.13× real time) at 1,228 MB peak. One utterance came back as the single word “Cal” instead of a 22-word sentence, which accounts for part of its clean-speech WER. C2 = 5 (1 intervention: the PyAV pin). C3 = 5 (4 min 45 s from a fresh virtual environment to the first caption file). C5 3 of 5: punctuation ✔, word timestamps ✔, multilingual models ✔; no built-in SRT/VTT writer or CLI ✘; README example crashed as written ✘. C6 = 5 (MIT code and weights, offline). C7 4 of 4.
Vosk (j10-vosk-r1), 68, fails G1: 3 of 7 checks. WER 12.3% clean (K2 core ✘, threshold 10%), 24.0% with noise ✘, 38.6% on earnings-call speech ✘. Captions in sync (73 of 73) ✔, valid SRT ✔, 729 MB peak ✔. 1,229 s to process 1,747 s (0.70× real time) ✘: the official Python example decodes on one CPU core. C2 = 5 (no interventions; the install page lists Python 3.5–3.9, but it installed and ran on 3.12). C3 = 4 (24 min from a fresh environment to the episode’s caption file, because our run transcribed the three test sets first; the episode alone took 2 min 34 s). C5 4 of 5: no punctuation or capitals in its output ✘; vosk-transcriber -t srt writes captions itself ✔ (its README command ran unchanged on a 60 s clip in 4.0 s with the small model it downloads); word timestamps ✔; 20+ languages ✔. C6 = 5 (Apache 2.0). C7 4 of 4.