VoiceLK Speaker 02 model: Results and Outcomes for Sinhala-English Code-Switched Speech

VoiceLK speaker 02 turns Sinhala-English code-switched text into a single synthetic voice. Its evaluation shows a useful combination: faster-than-real-time synthesis on the measured laptop CPU, strong similarity to the recorded speaker, and successful generation of several long technical passages. It also reveals a clear gap in pronunciation and a much weaker response to isolated words, dates and unusual punctuation.

This article follows one checkpoint through the measurements: what it preserves, where it struggles, and which changes the evidence supports next. The numbers come from the saved speaker 02 evaluation, with 30 validation recordings and 40 additional stress prompts. Automated scores help identify patterns; a human listening score is not yet available for this evaluation.

What we evaluated

Speaker 02 uses a single-speaker VITS model trained on recordings containing Sinhala speech and frequent English technical vocabulary. VITS combines variational inference, normalizing flows and adversarial training, with a stochastic duration predictor that allows variation in speaking rhythm. VITS paper

The model receives IPA produced by VoiceLK's normaliser and code-switched grapheme-to-phoneme (G2P) pipeline. Sinhala and English tokens are converted into a shared phonetic representation before synthesis. The evaluated model produces audio at 22,050 Hz and uses 36.30 million parameters at inference. The checkpoint contains approximately 83.04 million parameters when the training-only discriminator is included.

The dataset file list contains 3,087 clips. Reconstructing the training pipeline's 1% evaluation split yields 3,057 training clips and 30 validation clips. The reconstruction uses the original file order and NumPy shuffle seed of 0; it assumes that this file list is the one used for the training run.

These 30 clips were held out from gradient training, but they belong to the model's validation split. Because the checkpoint was selected during training, they should not be treated as a completely untouched test set. The 40 hand-written stress prompts provide additional checks, without matching reference recordings.

The evaluated duration-noise setting is important: the model configuration stores inference_noise_scale_dp = 1.0, while this evaluation uses noise_scale_w = 0.8. These results describe the recorded experiment, not every possible deployment configuration.

The results at a glance

Pronunciation: compare generated speech with the real voice

An automatic speech recogniser can make mistakes even on the original recordings. That matters especially for code-switched speech: a recogniser may write an English word using Sinhala characters while the reference transcript retains its Latin spelling. Word error rate (WER) then counts a spelling-system mismatch alongside actual recognition errors.

The evaluation therefore uses the same selected Sinhala XLS-R recogniser on both generated and real audio. It also maps recognised text back through the project's G2P pipeline to compute a phone error rate (PER), stored as ipa_cer in the results file. This reduces script mismatch, although it still depends on both the recogniser and G2P rules.

Generated PER is higher on 27 of the 30 sentences. The paired Wilcoxon test gives approximately p = 0.0000195, supporting a consistent difference within this sample. The 0.087 gap is 8.7 percentage points; it is not a direct estimate that listeners miss 8.7% more speech sounds.

The recogniser already produces PER of 0.352 on real recordings. Consequently, the absolute generated PER cannot be read as a human intelligibility percentage. The paired comparison is more useful: synthetic speech introduces additional difficulties under the same measurement pipeline.

The phone-confusion analysis points to vowel substitutions, lost vowel-length distinctions and consonant confusions. These patterns make pronunciation and alignment review a priority. They do not, by themselves, prove whether the underlying cause is the acoustic model, the transcript, the recogniser or a G2P rule.

Voice identity is retained more strongly than timing

Speaker embeddings provide a different view of the output. Using ECAPA-TDNN, generated validation clips have a mean cosine similarity of 0.863 to a leave-one-out centroid of the speaker's real recordings. Real recordings score 0.859 against their corresponding centroids. Generated audio compared directly with its own reference recording scores 0.800 on average.

These values support consistency with the target voice on this set. They are cosine similarities, not percentages of identity, and they do not establish human-level naturalness.

Generated sentences are 9.4% shorter on average, using the mean per-sentence duration ratio. The evaluation also records an average F0 correlation of 0.402, F0 RMSE of approximately 49.1 Hz, and DTW-aligned mel-cepstral distortion of 6.52 dB.

These measurements show differences from the reference performance. A low pitch correlation does not automatically imply an invalid intonation: VITS samples a delivery rather than copying the reference contour. Equally, a similar voice embedding does not guarantee that pauses, stress or technical terms sound right. Those questions need listening evaluation.

The MCD implementation is specific to this evaluation pipeline. Its most useful role is comparison with later VoiceLK checkpoints measured in the same way, rather than a direct ranking against values reported by unrelated systems.

CPU performance: fast synthesis with measurable waiting time

Real-time factor (RTF) is synthesis time divided by output-audio duration. A value below 1 means generation is faster than playback. After five warm-up runs, the benchmark records 50 timed runs in each mode.

The text-to-audio path includes normalisation and G2P, which averaged about 27 ms in this benchmark. These are local inference timings; they exclude network delay, application queues and browser playback setup. They also measure non-streaming synthesis: the complete waveform is produced before playback can begin.

Memory and startup time remain relevant. The benchmark records approximately 1,722 MiB peak resident memory and 40.7 seconds to initialise the model (front-end initialisation is outside that timer). Faster-than-real-time generation therefore does not mean an instant cold start. A deployment should measure startup separately and benchmark its actual serving configuration before setting latency expectations.

Stress testing: isolated words need more attention

The 40 stress prompts cover numbers, dates and times, abbreviations, very short inputs, very long inputs, punctuation and rare or out-of-domain terms.

For this experiment, a prompt is flagged if any of the following occurs: a short or empty output, an unusual seconds-per-phone ratio, an ASR repetition or skip pattern, or PER above 0.558. That threshold is the real-recording PER mean plus three standard deviations. It is a diagnostic heuristic calibrated on this sample, not a validated product failure threshold.

Overall, 14/40 stress prompts trigger a calibrated flag, compared with 1/30 validation clips. None trigger the pipeline's short-or-empty-output flag; the other diagnostics account for the flagged cases. These categories overlap and should not be summed as independent failures.

Very short prompts are the clearest weakness. Their mean PER is 0.919, and some are stretched relative to the training corpus's seconds-per-phone distribution. One returns an empty ASR transcript, which is evidence that the recogniser failed to recover speech, not a confirmed human judgment that the clip is unintelligible. Limited isolated-word coverage in the training material is a plausible contributing factor.

The four very long prompts, approximately 15-22 seconds of generated audio in the report, trigger no calibrated flags and have the lowest category mean PER. This supports further investigation of paragraph synthesis, while the tiny sample leaves many long-input behaviours untested.

The normalisation inspection also identifies concrete cases to fix:

• Phone numbers can be expanded as large cardinal numbers instead of digit sequences.
• Dates such as 2025/03/21 can become separate numbers without a date-aware reading.
• 10:30 can receive an English expansion inside Sinhala context.
• The unit mAh can be spelled out rather than expanded as a unit phrase.
• A full stop after Dr. can introduce an unwanted pause before a name.
• A spaced hyphen can be interpreted as a minus sign rather than a pause.

These are useful engineering targets because the text front-end determines what the acoustic model is asked to say. Correctly pronouncing the wrong expansion still produces the wrong reading.

Automated quality scores are not listener results

The evaluation includes UTMOS and DNSMOS as supplementary signals. UTMOS predicts listening scores from audio; DNSMOS P.835 estimates speech, background and overall quality for noise-suppression evaluation. Neither should be presented as a collected human MOS for this Sinhala-English voice. UTMOS paper, DNSMOS P.835 paper

The paired tests do not establish a difference for UTMOS (p = 0.177) or DNSMOS overall (p = 0.164). Nor do those results demonstrate equivalence. DNSMOS background is higher for generated audio in the unadjusted comparison (p = 0.016), but that is one of multiple proxy comparisons and should be interpreted cautiously.

The evaluation package includes blinded listening stimuli and a rating template. That template contains no collected ratings, so this article reports no human MOS. A listening study with Sinhala-English speakers is the next step for assessing naturalness, pronunciation, rhythm and code-switch boundaries together.

What these outcomes change next

The evidence supports a focused improvement cycle.

First, fix the text that reaches the model. Add regression cases for digit strings, dates, times, unit expansions and abbreviation punctuation. Keep the training and serving front-ends aligned. The current evaluation found exact agreement between regenerated and stored IPA for all 30 validation sentences, plus no out-of-vocabulary symbols on the stress prompts; that consistency is useful, although it does not prove the pronunciation rules are correct.

Second, broaden short-input coverage. Add carefully aligned isolated words, brief responses and short technical phrases, then rerun the stress tests. Verify the audio and transcripts before treating every recogniser error as a model error.

Third, compare checkpoints on a separate evaluation set. Track pronunciation, voice similarity, timing and compute cost together. The present results describe one checkpoint and one main synthesis seed; they do not establish the best checkpoint or inference settings. No completed sweep result is included in this snapshot.

Finally, collect listening judgments. Use blinded samples and report the number of listeners, number of utterances and uncertainty alongside the resulting scores. Human evaluation should test whether the measured pronunciation gap and timing differences are noticeable and important in actual code-switched speech.

Speaker 02 provides a working basis for VoiceLK's technical speech pipeline: the measured CPU can generate audio faster than playback, the voice embedding remains consistent on validation sentences, and several long prompts synthesise without diagnostic flags. Pronunciation, short-input behaviour and context-aware text expansion are the next measurable improvements. Progress should be judged with the same reproducible measurements and an independent listening study.