Best-of-N TTS: When the Evaluator Favors the Selector
Understand ASR-family bias in best-of-N text-to-speech evaluation, with an example and a practical design for independent candidate assessment.

Generating several speech candidates and selecting one can improve an output. But the apparent improvement may depend on which speech recognizer selects the candidate and which recognizer scores it afterward. Best-of-N evaluation needs a clear boundary between the selection process and the evidence used to judge that process.
Research reviewed September 17, 2026.
What the 2026 study found
Taehyung Yu and Seongjae Kang's paper was submitted July 9, 2026 and revised August 5. Using F5-TTS on LibriSpeech-PC, it compares candidate selection and evaluation across Whisper, wav2vec 2.0, and HuBERT recognizers. The authors find that verifier rankings depend on the evaluation family and investigate cross-family rank ensembles. Their stated limitations include one TTS backbone and a single benchmark setting. Read the best-of-N evaluation paper.
This is evidence of a confound in a defined experiment. It does not mean every automatic recognition score is invalid, or that selecting among candidates cannot help.
Understand the selector-evaluator relationship
Imagine three generated readings of the same sentence. Recognizer A scores them and chooses the version it transcribes most accurately. If the final report uses A again, the selected version has already been optimized for that judge.
An independent recognizer may hear a different error. Human listeners may also prefer a candidate whose transcript score is slightly worse because its pronunciation or phrasing is clearer.
This original example illustrates a general evaluation problem: the selection rule influences the distribution of outputs being evaluated.
Write down every component's role. The generator creates candidates. The selector chooses one. The evaluator measures the chosen output. A human panel may assess intelligibility or preference. Naming those roles is a simple way to expose accidental overlap.
Hold the candidate pool fixed
To compare selectors, give them the same saved candidates. Otherwise, one selector might appear better because its generator happened to produce an easier pool.
Keep each candidate's audio, text, generation seed where available, and generation settings. Record which candidate each selector chooses.
Include a baseline that does not use the selection procedure, such as a predetermined first candidate. Choose that rule before looking at results.
Also record an upper-bound analysis separately if inspecting the best candidate with hindsight. A hindsight choice shows what the pool contains, but it is not a deployable selection method unless the required information is available at decision time.
This distinction keeps an impressive possible outcome from being confused with the performance of an actual system.
Use an explicit cross-evaluation table
Create rows for selection methods and columns for evaluators. Evaluate every selected output under every column.
An illustrative table might contain a selector from family A, one from family B, and a combined selector, with independent transcription checks and human review as separate columns. No numerical outcome is implied by that example.
Look for ranking reversals. If the preferred selector changes with the evaluator, investigate the affected clips instead of averaging away the disagreement immediately.
Check whether differences come from names, accents, punctuation normalization, or genuinely missing words. Keep a record of the normalization rules used to compare transcripts. An evaluation can change merely because two systems render the same spoken number differently in text.
Keep listening quality outside the transcript score
A transcript comparison focuses on words. It does not directly tell you whether an otherwise accurate sentence has an awkward pause or an unsuitable tone.
Ask listeners to assess content accuracy and listening preference separately. Provide the approved text for a content check, then use a suitable blind comparison for preference.
The voice-over quality assurance checklist offers practical categories for this review. The goal is to determine whether selection improves the recording people actually hear.
For recurring pronunciation problems, consult the AI pronunciation guide. Fixing an ambiguous script before generation may address a different cause than choosing among many imperfect readings afterward.
Include the cost of extra candidates
Best-of-N uses more than one generation attempt. Report candidate count and the work required to create, transcribe, rank, and retain those candidates.
A result at a larger candidate count should not be presented as a free improvement over a single generation. The relevant question is whether the benefit justifies its latency and computation for the intended workload.
Measure the rate of usable outputs as well as the average score. If additional candidates help mostly on a small subset of difficult scripts, a targeted selection policy might deserve investigation.
That is a proposed follow-up experiment, not a finding of this paper. Define the triggering rule before evaluating it so that the same test set is not repeatedly used to design and validate the policy.
Report a claim another team could reproduce
A useful report states the TTS version, candidate budget, selector, evaluator families, text normalization, dataset scope, and human-review procedure.
Present score differences with their uncertainty and include cases where evaluators disagree. Avoid treating a small improvement under one recognizer as a universal increase in speech quality.
Our interpretation is that the paper's most useful contribution is methodological: candidate selection and final assessment should be made visibly separate. Cross-evaluation gives readers a way to see whether the result survives a change in judge.
For a production team, the decision may remain narrow. A selector can be useful for one language and workload even if it is not universally best. The stronger conclusion requires broader evidence. Preserve the candidate pool, publish the decision rule, and let independent assessment determine whether the chosen audio truly improved.
Try Our Voice Clone Demo
Hear your words come to life
Choose a voice and try a short preview.
Listen to sample voices
Hear examples before choosing a voice. Generated results can vary with the script and reference sample.
Looking for another voice?
Explore the library and listen to a sample before you create.
Morgan Freeman
Stephen Hawking
Christiano Ronaldo
Donald Trump
Kokoro
Disney XD Announcer
Cute Japanese Girl
Vin
Adam Stone
Transform Your Content with AI Voice Technology Today
Try a short voice preview, then create speech and save your audio in a workspace built for your next project.
Generate Your Voice Now