Skip to content
VocalCopyCat

Best-of-N TTS: When the Evaluator Favors the Selector

Understand ASR-family bias in best-of-N text-to-speech evaluation, with an example and a practical design for independent candidate assessment.

best of N TTSASR evaluation biasspeech candidate selectionTTS benchmarks
By Randy Wake5 min read
Best-of-N TTS evaluation diagram separating candidate selection by an ASR verifier from scoring with evaluators from different ASR families.
Best-of-N TTS evaluation diagram separating candidate selection by an ASR verifier from scoring with evaluators from different ASR families.

Generating several speech candidates and selecting one can improve an output. But the apparent improvement may depend on which speech recognizer selects the candidate and which recognizer scores it afterward. Best-of-N evaluation needs a clear boundary between the selection process and the evidence used to judge that process.

Research reviewed September 17, 2026.

What the 2026 study found

Taehyung Yu and Seongjae Kang's paper was submitted July 9, 2026 and revised August 5. Using F5-TTS on LibriSpeech-PC, it compares candidate selection and evaluation across Whisper, wav2vec 2.0, and HuBERT recognizers. The authors find that verifier rankings depend on the evaluation family and investigate cross-family rank ensembles. Their stated limitations include one TTS backbone and a single benchmark setting. Read the best-of-N evaluation paper.

This is evidence of a confound in a defined experiment. It does not mean every automatic recognition score is invalid, or that selecting among candidates cannot help.

Understand the selector-evaluator relationship

Imagine three generated readings of the same sentence. Recognizer A scores them and chooses the version it transcribes most accurately. If the final report uses A again, the selected version has already been optimized for that judge.

An independent recognizer may hear a different error. Human listeners may also prefer a candidate whose transcript score is slightly worse because its pronunciation or phrasing is clearer.

This original example illustrates a general evaluation problem: the selection rule influences the distribution of outputs being evaluated.

Write down every component's role. The generator creates candidates. The selector chooses one. The evaluator measures the chosen output. A human panel may assess intelligibility or preference. Naming those roles is a simple way to expose accidental overlap.

Hold the candidate pool fixed

To compare selectors, give them the same saved candidates. Otherwise, one selector might appear better because its generator happened to produce an easier pool.

Keep each candidate's audio, text, generation seed where available, and generation settings. Record which candidate each selector chooses.

Include a baseline that does not use the selection procedure, such as a predetermined first candidate. Choose that rule before looking at results.

Also record an upper-bound analysis separately if inspecting the best candidate with hindsight. A hindsight choice shows what the pool contains, but it is not a deployable selection method unless the required information is available at decision time.

This distinction keeps an impressive possible outcome from being confused with the performance of an actual system.

Use an explicit cross-evaluation table

Create rows for selection methods and columns for evaluators. Evaluate every selected output under every column.

An illustrative table might contain a selector from family A, one from family B, and a combined selector, with independent transcription checks and human review as separate columns. No numerical outcome is implied by that example.

Look for ranking reversals. If the preferred selector changes with the evaluator, investigate the affected clips instead of averaging away the disagreement immediately.

Check whether differences come from names, accents, punctuation normalization, or genuinely missing words. Keep a record of the normalization rules used to compare transcripts. An evaluation can change merely because two systems render the same spoken number differently in text.

Keep listening quality outside the transcript score

A transcript comparison focuses on words. It does not directly tell you whether an otherwise accurate sentence has an awkward pause or an unsuitable tone.

Ask listeners to assess content accuracy and listening preference separately. Provide the approved text for a content check, then use a suitable blind comparison for preference.

The voice-over quality assurance checklist offers practical categories for this review. The goal is to determine whether selection improves the recording people actually hear.

For recurring pronunciation problems, consult the AI pronunciation guide. Fixing an ambiguous script before generation may address a different cause than choosing among many imperfect readings afterward.

Include the cost of extra candidates

Best-of-N uses more than one generation attempt. Report candidate count and the work required to create, transcribe, rank, and retain those candidates.

A result at a larger candidate count should not be presented as a free improvement over a single generation. The relevant question is whether the benefit justifies its latency and computation for the intended workload.

Measure the rate of usable outputs as well as the average score. If additional candidates help mostly on a small subset of difficult scripts, a targeted selection policy might deserve investigation.

That is a proposed follow-up experiment, not a finding of this paper. Define the triggering rule before evaluating it so that the same test set is not repeatedly used to design and validate the policy.

Report a claim another team could reproduce

A useful report states the TTS version, candidate budget, selector, evaluator families, text normalization, dataset scope, and human-review procedure.

Present score differences with their uncertainty and include cases where evaluators disagree. Avoid treating a small improvement under one recognizer as a universal increase in speech quality.

Our interpretation is that the paper's most useful contribution is methodological: candidate selection and final assessment should be made visibly separate. Cross-evaluation gives readers a way to see whether the result survives a change in judge.

For a production team, the decision may remain narrow. A selector can be useful for one language and workload even if it is not universally best. The stronger conclusion requires broader evidence. Preserve the candidate pool, publish the decision rule, and let independent assessment determine whether the chosen audio truly improved.

Try Our Voice Clone Demo

Hear your words come to life

Choose a voice and try a short preview.

77 / 120 input characters
Continue with 2,000 welcome credits

Listen to sample voices

Hear examples before choosing a voice. Generated results can vary with the script and reference sample.

Looking for another voice?

Explore the library and listen to a sample before you create.

Morgan Freeman avatar

Morgan Freeman

Morgan Freeman voice sample0:00 --:--
Stephen Hawking avatar

Stephen Hawking

Stephen Hawking voice sample0:00 --:--
Christiano Ronaldo avatar

Christiano Ronaldo

Christiano Ronaldo voice sample0:00 --:--
Donald Trump avatar

Donald Trump

Donald Trump voice sample0:00 --:--
Kokoro avatar

Kokoro

Kokoro voice sample0:00 --:--
Disney XD Announcer avatar

Disney XD Announcer

Disney XD Announcer voice sample0:00 --:--
Cute Japanese Girl avatar

Cute Japanese Girl

Cute Japanese Girl voice sample0:00 --:--
Vin avatar

Vin

Vin voice sample0:00 --:--
Adam Stone avatar

Adam Stone

Adam Stone voice sample0:00 --:--

Transform Your Content with AI Voice Technology Today

Try a short voice preview, then create speech and save your audio in a workspace built for your next project.

Generate Your Voice Now

Latest Posts