Abstract This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model’s performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching (F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. Demo samples can be found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints to promote community development.
Contents
This page is for research demonstration purposes only.
Model Overview
Figure 1: An overview of F5-TTS training (left) and inference (right). The model is trained on the text-guided speech-infilling task and condition flow matching loss. The input text is converted to a character sequence, padded with filler tokens to the same length as input speech, and refined by ConvNeXt blocks before concatenation with speech input. The inference leverages Sway Sampling for flow steps, with the model and an ODE solver to generate speech from sampled noise.
All samples in this demo page are generated with F5-TTS (NFE=32 CFG=2 w/ SS) in one time (no cut), with pretrained Vocos as vocoder.
Zero-shot Generation
Prompt and text from the demo page of Seed-TTS.
| Language | Prompt | Same Language Generation | Cross-linugal Generation |
|---|---|---|---|
| EN | I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences. |
顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。” |
|
Perhaps they are driven by the delicious blend of flavors, or it could be the appealing visual presentation. At the end of the day, our choices in food reflect our personal preferences and sometimes, even our lifestyle or belief system. |
我抬起头，坚定地说：“身高不能决定一切，这世界在看我，我更看得到世界。无论是北上广，或是别的什么，我都将以我自己的方式去攀爬，去追逐。我可能小，但我绝不会被忽视。” |
||
Your safety and the pack's reputation are at stake. Your bravery is admirable, but sometimes bravery is knowing when to retreat. Please, consider returning with me. We can work out a plan, but only if you're willing to listen. |
你的安全以及族群的声誉都危在旦夕。你的勇敢令人钦佩，但有时候勇敢在于懂得何时撤退。拜托，考虑一下和我一起回去吧。我们可以制定一个计划，但前提是你愿意倾听。 |
||
| ZH | 突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道："我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？" |
Suddenly, there was a burst of laughter beside me. I looked at them, stood up straight with high spirit, shook the slightly fleshy arms, and smiled lightly, saying, "The flesh on my body is to hide my bursting charm. Otherwise, wouldn't it scare you?" |
|
顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。” |
Suddenly, the atmosphere became gloomy. At first glance, all the troubles seemed to surround me. I frowned, feeling that pressure, but I know I can't give up, can't admit defeat. So, I took a deep breath, and the voice in my heart told me, "Anyway, must calm down and start again." |
||
皇上的面色未变，宛如雕塑般静止，他的眼中闪过一丝动人的温度。他深深地看了那位忠心耿耿的臣子一眼，终于开口：“诺，我会再考虑考虑的。”他的声音低沉且坚定，留下空气中隐隐的无奈与柔情。 |
The emperor's complexion did not change, remaining as still as a sculpture, and a touch of touching warmth flashed in his eyes. He deeply glanced at the loyal minister, and finally spoke: "Well, I will consider it again." His voice was low and firm, leaving a faint hint of helplessness and tenderness in the air. |
Code-Switch, Text from FireRedTTS demo page.
| Prompt | Text | Code-Switched Generation |
|---|---|---|
| 你昨天的performance真是outstanding，完全展示了你的skills。 | ||
| 我觉得我们需要一个更clear的strategy来实现我们的goals。 | ||
| 这次旅行的schedule有点tight，我们需要plan得更efficient一些。 | ||
| 他今天的mood看起来不太好，可能需要一些space。 |
Speed Control
Prompt and Text from Seed-TTS, same as those used in MaskGCT demo page.
F5-TTS only needs a total duration, the character's position and duration will be automatically assigned by model.
| Prompt | Text | 0.7x Speed | 1.0x Speed | 1.3x Speed |
|---|---|---|---|---|
| I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences. | ||||
| Dealing with family secrets is never easy. Yet, sometimes, omission is a form of protection, intending to safeguard some from the harsh truths. One day, I hope you understand the reasons behind my actions. Until then, Anna, please, bear with me. | ||||
| 好呀，哈哈哈哈哈，喜欢笑的人运气都不会差哦，希望你每天笑口常开~ | ||||
| 顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。” | ||||
Prompt and Text from E2 TTS demo page.
| Prompt | Text | 0.7x Speed | 1.0x Speed | 1.3x Speed |
|---|---|---|---|---|
| He gave way to the others very readily and retreated unperceived by the Squire and Mistress Fitzooth to the rear of the tent. | ||||
| “How cheerfully he seems to grin, How neatly spread his claws, And welcome little fishes in With gently smiling jaws”! | ||||
| Yes; then something better, something still grander, will surely follow, or wherefore should they thus ornament me? | ||||
| And, though I have grown serene And strong since then, I think that God has willed A still renewable fear… | ||||
| He wore blue silk stockings, blue knee pants with gold buckles, a blue ruffled waist and a jacket of bright blue braided with gold. | ||||
| Not only this, but on the table I found a small ball of black dough or clay, with specks of something which looks like sawdust in it. | ||||
Emotion
Compare with most expressive results in E2 TTS demo page, Prompt from RAVDESS dataset.
| Emotion | Prompt | Text | E2 TTS | F5-TTS |
|---|---|---|---|---|
| Calm | So, I was, like, at the, um, grocery store, and, uh, I saw this, like, really yummy-looking, um, cake, y’know? And I, uh, totally wanted to, like, buy it, but, um, I was, like, on a diet, so, uh, I just, like, stared at it for a while, y’know? | |||
| Angry | ||||
| Disgust | ||||
| Happy | I was, like, talking to my friend, and she’s all, um, excited about her, uh, trip to Europe, and I’m just, like, so jealous, right? | |||
| Sad | ||||
| Fearful | ||||
Robustness
First prompt (Wukong) from FireRedTTS demo page, Second to Fourth from Bailing-TTS. Text from Seed-TTS hard testset same as in MaskGCT demo page.
| Prompt | Text | F5-TTS |
|---|---|---|
| 针蓝线蓝领子蓝，蓝针蓝线蓝领蓝。蓝针蓝线连蓝领，针蓝线蓝领子蓝。 | ||
| 墙上画凤凰，凤凰画在粉红墙。红凤凰、粉凤凰，红粉凤凰、花凤凰。红凤凰，黄凤凰，红粉凤凰，粉红凤凰，花粉花凤凰。 | ||
| 随后，民警还在店里发现一把锤子锤子锤子锤子锤子锤子。 | ||
| 北京在出行规模规模规模规模规模，城市影响力方面方面方面方面方面表现优异优异优异优异优异。 | ||
Hard sentences from ELLA-V. Same audio prompts as in E2 TTS demo page from LibriSpeech-PC test-clean.
| Prompt | Text | F5-TTS |
|---|---|---|
| Active artists always appreciate artistic achievements and applaud awesome artworks. | ||
| Brave bakers boldly baked big batches of brownies in beautiful bakeries. | ||
| Daring dancers dazzled during dynamic dance displays, drawing delighted crowds. | ||
| Excited engineers eagerly enjoyed exploring enormous engineering exhibits. | ||
| Friendly farmers faithfully fostered fields, favoring fruitful crops. | ||
| Gallant gophers gracefully gambled golden gooseberries on grandiose glaciers. | ||
| Happy hikers harmoniously hiked through hilly landscapes on heavenly holidays. | ||
| Inquisitive individuals ingeniously invented innovative inventions. | ||
| Jovial joggers joyfully joined jogging jaunts, justifying joyful jolliness. | ||
| Keen kids keenly knitted knotted knots in kindergartens. | ||
| F one F two F four F eight H sixteen H thirty two H sixty four. | ||
| Clever cats carefully crafted colorful collages creating cheerful compositions. | ||