CosyVoice2: Speech Synthesis Fast Enough for a Real Conversation
Turn text and recordings into audio you can hear
Dubbing, stems, covers, and sound effects — all in the browser.
- Text to speech, plus voice cloning
- Split stems, or generate a cover
- Sign up to preview — listen first, decide later
The original CosyVoice proved that supervised semantic tokens could produce more accurate, more consistent zero-shot voice cloning than earlier unsupervised-token models. What it couldn't do was talk back to you in real time — like most first-generation LLM-based TTS, it needed the complete text before it could start generating audio. CosyVoice2 was built to close exactly that gap, restructuring the same underlying approach into a streaming architecture with a first-packet latency low enough for live voice chat.
What's New in CosyVoice2
CosyVoice2, developed by Alibaba's Tongyi speech team, keeps the supervised-semantic-token foundation that made the original model work, but rebuilds the pipeline around three specific engineering changes:
Finite-scalar quantization. The original model's speech tokenizer left part of its codebook underused. CosyVoice2 replaces it with finite-scalar quantization, improving codebook utilization and giving the model a more efficient, better-populated representation of speech to work from.
A simplified text-speech language model. By streamlining the LM architecture, CosyVoice2 can use pretrained large language models directly as its backbone, rather than requiring a bespoke architecture — a change that both simplifies training and inherits the general language understanding of the base LLM.
Chunk-aware causal flow matching. This is the piece that actually enables streaming. Instead of waiting for the full input before generating a mel spectrogram, CosyVoice2 processes speech in causally-ordered chunks, unifying streaming and non-streaming synthesis into a single framework rather than shipping them as two separate models. The team's published results describe this streaming mode as producing output nearly indistinguishable in quality from full offline generation — a meaningfully different outcome than the quality trade-off streaming TTS has typically required.
The Numbers That Matter
- 150ms first-packet latency. CosyVoice2 can begin producing audio roughly 150 milliseconds after receiving text — fast enough that in a voice-chat pipeline (paired with an LLM generating the response text), the perceived delay stays low even though two models are working in sequence.
- 30–50% fewer pronunciation errors compared to the original CosyVoice, a direct result of the improved tokenizer and simplified LM architecture.
- MOS score up from 5.4 to 5.53, reflecting the human-rated naturalness improvement alongside the latency gains — CosyVoice2 isn't trading quality for speed, it's improving both simultaneously.
- Lowest character error rate on the Seed-TTS hard test set among compared models at the time of release, a benchmark specifically designed to stress-test difficult pronunciation cases.
Behind these numbers is a training detail worth knowing: CosyVoice2 uses reinforcement learning during fine-tuning, with speaker similarity and ASR-measured word error rate as the reward signals — directly optimizing the model toward sounding like the target speaker and getting the words right, rather than only optimizing a generic reconstruction loss.
Instructed Generation, Unified with Cloning
CosyVoice2 also merges two capabilities that were separate in the original release: zero-shot voice cloning and instruction-based style control. In the first-generation CosyVoice family, you had to choose between the base cloning model and the separate Instruct checkpoint. CosyVoice2 integrates both into one model, so you can clone a voice from a reference clip and independently steer emotion, accent, and role-style delivery through instructions — without switching checkpoints depending on which capability you need that session.
Getting Started with CosyVoice2
- Clone the repository with submodules. As with the original CosyVoice, use
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git— CosyVoice2 shares the same repository and Matcha-TTS dependency. - Download the CosyVoice2-0.5B checkpoint. It's available through Hugging Face and ModelScope, distinct from the original CosyVoice-300M checkpoints — make sure you're pulling the 2.0 weights specifically if streaming is what you need.
- Choose streaming or offline mode explicitly. Because CosyVoice2 unifies both in one framework, your inference call determines which path runs — streaming for interactive use cases, offline for maximum-quality batch generation where latency doesn't matter.
- Test instructed generation alongside cloning. Combine a reference audio clip with a style instruction (emotion, accent, or role) in the same call to see the unified capability in action, rather than testing cloning and instruction control separately.
- Benchmark first-packet latency on your own hardware. The published 150ms figure reflects the team's own test environment — validate actual latency on your deployment GPU before committing to real-time architecture decisions around it.
Tips for Better Results
- Use streaming mode specifically for conversational pipelines. If you're building a voice agent where an LLM's text output feeds directly into TTS, streaming mode is what keeps end-to-end response time low — offline mode reintroduces the full-utterance wait CosyVoice2 was built to eliminate.
- Don't assume streaming means lower quality. CosyVoice2's chunk-aware flow matching was specifically designed to keep streaming output close to offline quality — test both modes on your actual content rather than defaulting to offline out of caution.
- Lean on instructed control for tone rather than re-recording reference clips. If you need a different emotional delivery from the same cloned voice, an instruction change is faster to iterate on than sourcing a new reference clip with that emotional tone already present.
- Validate pronunciation on your specific domain vocabulary. The 30–50% error reduction is measured against general benchmarks; technical or domain-specific terminology should still be spot-checked before production use.
- Pair with a fast LLM if building voice chat. Since CosyVoice2's low latency is designed for chat-style pipelines, the overall response time still depends on how quickly your upstream LLM produces text — a slow text generator undercuts the benefit of a fast TTS stage.
CosyVoice2 vs. CosyVoice (Original)
| CosyVoice2 | CosyVoice (2024) | |
|---|---|---|
| Parameters | 0.5B | 300M |
| Streaming | Yes, unified with offline mode | No |
| First-packet latency | ~150ms | Not optimized for this metric |
| Zero-shot cloning + instructed style | Unified in one model | Separate checkpoints (base vs. Instruct) |
| Pronunciation accuracy | 30–50% fewer errors | Baseline |
| MOS score | 5.53 | 5.4 |
| Training method | RL fine-tuning (speaker similarity + WER reward) | Supervised training |
CosyVoice2's upgrade path is specific: if your project needs real-time or near-real-time generation — voice agents, live chat, interactive applications — this is the version built for that. If you're generating audio offline with no latency constraint, the gap matters less, though the accuracy and quality improvements still apply.
Frequently Asked Questions
What's the main difference between CosyVoice and CosyVoice2?
CosyVoice2 adds streaming synthesis with roughly 150ms first-packet latency, unifies zero-shot cloning with instructed style control into a single model, and improves pronunciation accuracy by 30–50% over the original — all while increasing quality (MOS 5.4 to 5.53) rather than trading it off for speed.
Is CosyVoice2 fast enough for real-time voice chat?
Yes, that's its primary design goal. A 150ms first-packet latency is fast enough to pair with an LLM in a voice-chat pipeline without introducing a noticeable delay from the TTS stage specifically.
Does CosyVoice2 sacrifice audio quality for streaming speed?
No. The chunk-aware causal flow-matching design was specifically built to keep streaming output close to offline-quality generation, and published MOS scores actually improved over the non-streaming original.
Can CosyVoice2 still do zero-shot voice cloning?
Yes, and it's improved — cloning and instruction-based style control (emotion, accent, role style) are now integrated into a single model rather than requiring separate checkpoints as in the original CosyVoice.
Is CosyVoice2 open-source?
Yes. Model weights and inference code are publicly available through the same FunAudioLLM/CosyVoice GitHub repository, alongside Hugging Face and ModelScope hosting.
Try a Comparable Voice Cloning Feature
CosyVoice2 is available through the FunAudioLLM/CosyVoice open-source implementation and its model hosts. If you want to explore browser-based voice cloning, try Voice Clone as a comparable feature. It is not CosyVoice2 and does not run the CosyVoice2 model, but it lets you test a similar voice-cloning workflow without setting up the model locally.
Only clone voices you own or have explicit permission to use.