CosyVoice: The Model That Rethought What a "Speech Token" Should Be

By 2024, most LLM-based zero-shot TTS models — VALL-E among them — shared the same underlying assumption: represent speech as tokens learned in an unsupervised way, then let a language model predict them like it would predict text. Alibaba's FunAudioLLM team looked at that assumption and challenged it directly. CosyVoice built its speech tokens from a multilingual speech recognition model instead, and the resulting gain in content accuracy and voice-cloning fidelity is what put the model on the map — and laid the architectural foundation that CosyVoice 2 and 3 would later build on.

What Is CosyVoice?

CosyVoice is an open-source multilingual zero-shot text-to-speech model developed by FunAudioLLM, the speech AI research team at Alibaba. Released as a 300-million-parameter model, it introduced a scalable, LLM-based approach to voice cloning that could reproduce a speaker's voice from a short reference clip without any fine-tuning — a capability that was still relatively novel in open-source TTS at the time of its release.

CosyVoice supports five languages — Chinese, English, Japanese, Cantonese, and Korean — and ships as three distinct checkpoints depending on what you need:

• CosyVoice-300M — the base zero-shot and cross-lingual model, best suited for cloning an arbitrary voice from a reference clip.
• CosyVoice-300M-SFT — supervised fine-tuned on a fixed set of high-quality voices, trading cloning flexibility for more consistent, production-ready output on a known voice roster.
• CosyVoice-300M-Instruct — adds natural-language instruction control over speaking style, letting you steer speed, emotion, and dialect through descriptive text prompts rather than only a reference audio clip.

The Core Innovation: Supervised Semantic Tokens

This is the detail that actually explains why CosyVoice performs the way it does, and it's worth understanding rather than skipping past.

Earlier LLM-based TTS systems represented speech using tokens extracted in an unsupervised way — essentially, the model learned to compress audio into discrete units without being told what those units meant linguistically. CosyVoice's research team took a different approach: they derived speech tokens from the internal representations of a multilingual speech recognition model, inserting a vector-quantization step into its encoder. Because that source model was trained to understand speech content, the resulting tokens carry explicit semantic and textual alignment that unsupervised tokens don't have.

The practical payoff, confirmed in the team's published evaluations, was twofold: significantly better content consistency (the generated speech more reliably says what the text actually says) and better speaker similarity in zero-shot cloning, compared to unsupervised-token approaches like VALL-E's. In head-to-head evaluation against ChatTTS, a contemporary of similar scale, CosyVoice also showed markedly fewer insertion and deletion errors — notably avoiding the "speaker leakage" issue where fragments of a different speaker's voice would bleed into generated output.

Architecturally, generation happens in two stages: an autoregressive language model converts text into these semantic tokens, and a conditional flow-matching model converts the tokens into a mel spectrogram, which a vocoder then renders into the final waveform.

Getting Started with CosyVoice

1. Clone with submodules. CosyVoice depends on the Matcha-TTS project as a git submodule, so use git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git — a standard clone will leave required components missing.
2. Set up your environment and install dependencies. Create a dedicated Python environment (Conda is recommended) and run pip install -r requirements.txt; using Alibaba's PyPI mirror speeds this up noticeably if you're installing from within China.
3. Choose the right checkpoint for your task. Download CosyVoice-300M for zero-shot or cross-lingual cloning, CosyVoice-300M-SFT if a fixed high-quality voice roster is enough, or CosyVoice-300M-Instruct if you need natural-language style control.
4. Optionally install the text-frontend resource. The CosyVoice-ttsfrd package improves text normalization (numbers, dates, abbreviations) and is worth adding if your input text isn't already clean, spoken-form text.
5. Launch the WebUI to test before integrating. Running python3 webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M opens a Gradio interface where you can test voice cloning directly by uploading a reference clip and entering target text.

Tips for Better Results

• Match the checkpoint to the job, not the other way around. Trying to force the SFT model into arbitrary voice cloning, or the base model into fixed high-quality narration, both work against what each checkpoint was actually tuned for — pick the checkpoint first, based on your use case.
• Use the Instruct model when style matters more than a specific cloned voice. If your priority is controlling delivery — pace, emotion, accent — rather than reproducing one exact person's voice, CosyVoice-300M-Instruct's natural-language style prompts are more direct than trying to coax style out of a cloning-only model.
• Give clean, well-recorded reference clips. Because CosyVoice's supervised tokens are sensitive to genuine speech content, noisy or low-quality reference audio degrades cloning accuracy more noticeably than it would with a purely acoustic-token approach.
• Preprocess text with the ttsfrd package for numbers and dates. Raw text containing digits, abbreviations, or dates benefits from the optional text-normalization frontend rather than being passed to the model as-is.
• Treat CosyVoice 1.0 as the non-streaming baseline. If your project specifically needs low-latency streaming generation, that capability was introduced in CosyVoice 2 — this original release is better suited to offline generation where full-utterance latency isn't a constraint.

CosyVoice in Context

CosyVoice's contribution wasn't scale — at 300M parameters, it was modest even by 2024 standards — it was proving that how you represent speech tokens matters as much as how big the model is. That architectural bet is what the later CosyVoice 2 and CosyVoice 3 releases continued to build on.

Frequently Asked Questions

Who developed CosyVoice?

CosyVoice was developed by FunAudioLLM, the speech AI research team within Alibaba Group, and released as an open-source project on GitHub.

What makes CosyVoice's speech tokens different from other TTS models?

CosyVoice derives its tokens from a multilingual speech recognition model rather than learning them in an unsupervised way, giving the tokens explicit semantic alignment to text — which improves content accuracy and speaker similarity compared to unsupervised-token approaches.

Which CosyVoice checkpoint should I use?

Use the base CosyVoice-300M model for zero-shot or cross-lingual voice cloning, CosyVoice-300M-SFT for a fixed set of pre-tuned high-quality voices, or CosyVoice-300M-Instruct if you need natural-language control over speaking style.

Does CosyVoice support streaming generation?

No, not in this original release. Streaming synthesis was introduced in CosyVoice 2, which built on this model's token architecture with a chunk-aware design for lower-latency output.

Is CosyVoice free for commercial use?

Yes. CosyVoice's code and model weights are openly available on GitHub and Hugging Face for local deployment and further development.

Try a Comparable Voice Cloning Feature

CosyVoice is available through its open-source implementation and community tools. If you want to explore browser-based voice cloning, try Voice Clone as a comparable feature. It is not CosyVoice and does not run the CosyVoice model, but it lets you test a similar voice-cloning workflow without setting up the model locally.

Only clone voices you own or have explicit permission to use.

Explore Voice Clone →