Test script: Make a voiceover from a script for free

• Job slug: ai-voiceover-from-script
• Version / status: v1 · frozen 2026-10-16 15:07 ET, before the first run. Changes after freeze go under “Amendments”, dated, with the reason.
• Rubric version: v1 (rubric)
• Author / date: Lando (CEO), 2026-10-16

1. The job

You need a clear voiceover for a product demo, a tutorial or a short video, you don’t want to record yourself, and you don’t want a per-character bill or a licence that stops you using it commercially. You’re done when the tool reads the whole script into an audio file that’s clear enough to publish, numbers and dates included.

2. Candidates

Free text-to-speech that runs on your own computer (no account, no upload). Hosted voice services need a browser signup we can’t do in this run; they’re listed under “Not tested yet” on the page.

Scroll sideways for more columns →

Voice rule: each tool’s highest-quality US English voice whose licence allows commercial use, chosen from its own voice list before the run. For Kokoro, af_heart (top-graded in Kokoro’s VOICES.md, same Apache 2.0 licence as the model). For Piper, ljspeech at high quality (its default example voice, lessac, is trained on a dataset with its own licence terms, so we didn’t use it).

3. Fixed inputs

• Script: a 139-word product explainer (SHA-256 5c54bc210f863675a9279ec3fdfc7fe49b640d75c4d3ee651984052ffc05d1b1), one paragraph per line, including a price ($24), a hyphenated number (14-day), a number with a comma (3,000), an abbreviated date (Oct. 16, 2026), a time (9 AM) and a domain (buildwiththis.com). The full text is published with the results.
• Speed: each tool’s default speaking rate.
• Measuring ear: the same speech recognizer for both tools, faster-whisper with model small.en (int8, CPU), plus the whisper_normalizer English normalizer on both sides. It’s a measuring tool here, not a candidate.

4. Fixed task

Using only <tool>'s official README and docs, on CPU:
1. Install it and the chosen voice.
2. Read the whole script (all 6 paragraphs, passed as one text in one call) into a WAV file.

5. Follow-up rules (agent operator version)

• Time cap: 60 active minutes per tool to get from nothing to the first full WAV. Model download time is logged but excluded, up to 15 minutes.
• Allowed: the tool’s official README, docs and error messages. Not allowed: third-party tutorials or forks.
• Manual interventions (counted for C2): any step the official docs don’t cover, and any manual splitting or editing of the script.

6. Steps

1. Record the start time. Create a fresh Python virtual environment for the tool.
2. Install and download the voice following the official docs. Log every docs page used.
3. Generate the WAV; record wall-clock time, audio length and peak memory (/usr/bin/time -v). Generate it 3 times and use the median time.
4. Run the check script on the WAV. Save its JSON. Publish the WAV (as MP3) so readers can judge the voice themselves.

7. Pass/fail checks

Scroll sideways for more columns →

8. Quality checklist (C5)

Scroll sideways for more columns →

9. Safety and licence checklist (C7)

Scroll sideways for more columns →

10. Weights and tie-breaker

• Weights: C1 35 · C2 15 · C3 10 · C4 10 · C5 10 · C6 10 · C7 10.
• C1: rubric anchor on the share of V1–V7 passed.
• C2 anchors (manual interventions): 5 = 0–1 · 4 = 2–3 · 3 = 4–5 · 2 = 6–8 · 1 = more.
• C3 anchors (T = 60 active min): 5 = ≤15 · 4 = ≤30 · 3 = ≤45 · 2 = ≤60 · 1 = cap hit.
• C4: 5 = $0 with no limit on how much you generate (runs locally) · rubric anchors otherwise.
• C5 and C7: checklist-count anchor (rubric §3).
• C6 ownership: 5 = permissive open-source licence (MIT/Apache) for engine and voice, runs offline, standard WAV output · 4 = open source but copyleft (GPL), so embedding it in a closed-source app has conditions · 3 = runs locally but the voice licence restricts commercial use · 1 = hosted only.
• Tie-breaker: C1, then the lower round-trip WER (V2).
• Not scored: how natural the voice sounds. There’s no honest automatic score for that, so we publish both recordings for you to judge.

11. Cleanup

• Delete the virtual environments and downloaded voices from the test machine after the verdict is published. Keep the WAVs, transcripts, check JSON and timing logs as evidence.

Amendments

• v1.1 (2026-10-16 15:36 ET, after the first runs and before scoring; this changes how V2 is scored, for both tools equally):
  1. V2 reference fix. Reading the word-by-word alignment showed that most V2 “errors” for both voices were spelling, not speech: the script writes BuildWithThis as one word while the measuring ear writes “Build with this” (1 substitution + 2 insertions each time, twice), writes re-test where the ear writes “retest” (2 errors), and AM where the ear writes “a.m.” (2 errors). Under v1’s rule both voices scored the same 7.04% and both would fail V2 for reasons no voice could avoid. From v1.1, V2 joins these three split spellings (build with this, re test, a m) on both sides before computing WER. Nothing else changes: mishearings such as “calm” for “com” or “$1.24” for “$24” still count. v1 numbers are kept in the evidence and on the page.
  2. What the voices actually said. Where the ear and the script disagreed we checked each tool’s own phoneme output for the same sentences (evidence phonemes-*.txt). Both voices say “dollar twenty-four” for $24 and “oct” for Oct.. Kokoro says “build with this com” (no “dot”); Piper says “build with this dot com”. V3 still uses the ear, as frozen; its result for each tool is the same either way.
  3. Network off: every scored generation ran with no network (unshare -rn), which is how S1 and S2 were checked. GNU time was installed on the test machine before the runs so V7 uses /usr/bin/time -v.
  4. Second runs: the first-run scores tied (91 and 91), so both tools were run again (j11-kokoro-r2, j11-piper-r2) and the scores averaged. Still tied, so the tie-breaker decided: C1 tied, then the lower average V2 WER (v1.1).

Scoring notes (2026-10-16)

• Kokoro, voice af_heart (j11-kokoro-r1/r2), 91 and 91: 6 of 7 checks both runs. V1 ✔ (0.7% of words missing). V2 1.42% both runs (v1: 7.04%) ✔. V3 5 of 8 ✘ (missed: the price, the date, the domain). V4 ✔ 0.16× and 0.15× real time (7.48 s and 6.90 s median for 47.5 s of audio). V5 ✔ (no clipping, longest pause 0.62 s). V6 ✔ (Apache 2.0 weights and voice, MIT wrapper). V7 ✔ (929 MB). C2 = 5 (no interventions). C3 = 5 (under 5 active minutes from a fresh project to the first WAV). C5 5 of 5 (24 kHz; 28 US/UK English voices under the same licence; speed option; README example ran unchanged; phoneme input example). C6 = 5. C7 3 of 4 → 4: the model card says the training data was permissive or public-domain audio plus synthetic audio from closed TTS models, but doesn’t say what the af_heart voice was trained on (S3 partial).
• Piper, voice en_US-ljspeech-high (j11-piper-r1/r2), 91 and 91: 6 of 7 checks both runs. V1 ✔ (0.7%). V2 2.13% and 2.84% (v1: 7.04% and 7.75%) ✔. V3 6 of 8 ✘ (missed: the price, the domain as the ear spelled it “buildwithis” / “buildwithfizz”). V4 ✔ 0.12× and 0.10× real time (6.41 s and 5.08 s median for about 51 s of audio). V5 ✔ (0.0013% clipped, longest pause 0.40 s). V6 ✔ (GPL-3.0 engine allows commercial use; LJ Speech voice data is public domain). V7 ✔ (358 MB). C2 = 5. C3 = 5. C5 5 of 5 (22.05 kHz; at least 10 US/UK English voices on public-domain, CC0 or CC BY data; length_scale speed option; README example ran unchanged; raw phoneme input). C6 = 4 (GPL-3.0). C7 4 of 4.
• Tie-breaker: average V2 WER (v1.1) Kokoro 1.42%, Piper 2.49%. Kokoro is the pick. It’s close: Piper’s extra errors are the ear’s spelling of “build with this dot com” and “October” for a voice that said “oct”.