Voxtral 3
Voxtral 3

原始链接: https://mistral.ai/news/voxtral-3/

Voxtral 3 是 Mistral 新推出的语音转文本与文本转语音模型系列,以及用于实时对话式 AI 的语音代理 SDK。Voxtral Small Transcribe 3 提供跨 50 多种语言的领先多语言转录,支持说话人分离、上下文偏置和轮次结束检测。Voxtral Mini TTS 3 提供 35 多种语言和 50 多种方言的富有表现力的语音,能根据上下文调整情感、语速和重音,并支持语音克隆和发音控制。语音代理 SDK 将听、推理、工具调用和说话整合到一条低延迟流式管道中,使代理能够检索信息、与业务系统交互,并通过电话或 Web 完成多步任务。Pro Voice 适配可从 30 分钟的音频中复现说话者的风格。Voxtral 3 支持通过 Mistral 的欧盟托管 API 或自托管进行主权部署,让组织掌控音频处理和数据保留。它可通过 Mistral API 使用。

黑客新闻 新 | 过去 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 Voxtral 3 ( mistral.ai ) 28 分,由 ilreb 发布于 28 分钟前 | 隐藏 | 过去 | 收藏 | 1 条评论 帮助 drum55 3 分钟前 [–] 可惜是闭源模型,延续了 TTS / STT 在本地部署上表现真的很差的趋势。 回复 考虑申请 YC 2027 冬季批次!申请开放至 11 月 2 日。 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索:
相关文章

原文

Speech transcription and voice generation have come a long way, but building end-to-end voice agents that can listen, reason, interact, and handle complex workflows remains difficult. 

A reliable voice agent needs to do four things very well (and simultaneously): 

  1. Listen accurately across languages, accents, speakers, and interruptions,

  2. Reason deeply about what users mean in specific contexts,

  3. Speak naturally with emotion and expression, and 

  4. Take action over multiple steps through connected systems. 

Getting all four to work together reliably in real time remains a challenge. 

To address this, we introduce Voxtral 3: a new family of state-of-the-art speech-to-text and text-to-speech models, alongside a voice agent SDK that brings listening, thinking, tool calling, and speaking into a single real-time streaming pipeline.

Voxtral 3 is built to create natural conversational experiences, from contact-center agents and service desks to virtual assistants and hands-free copilots. With support for dozens of languages, expressive speech, complex reasoning, and deep integration with your own tools and data, agents built with Voxtral can interpret nuanced intent, reason through sophisticated requests, and execute multi-step actions across connected systems.

As with other Mistral models, the Voxtral 3 family is designed for sovereign deployment. Available today through Mistral's EU-hosted API or for self-deployment on your own infrastructure, it gives organizations control over where audio is processed, how data is handled, and whether conversations are retained.

Highlights

  1. Accurate transcription across 50+ languages. The speech-to-text model, Voxtral Small Transcribe 3, is state-of-the-art in multilingual transcription, identifying who said what (diarization), using keyphrases to improve its accuracy (context-biasing). It detects when someone has finished speaking so agents can respond without delay (end-of-turn detection).

  2. Natural speech that adapts to the conversation. The text-to-speech model, Voxtral Mini TTS 3, introduces new voices that adjust their emotion, pace, and emphasis based on conversational context, with support for 35+ languages and 50+ dialects.

  3. Human-like intelligence at your (spoken) command. The new voice agent SDK combines listening, reasoning, tool calling, and speaking into a real-time experience. Agents can retrieve information, interact with business systems, and complete tasks during live conversations over phone or web.

  4. Pro Voice for distinctly accurate voice adaptation. For professional use cases where basic voice cloning does not suffice, Voxtral’s Pro Voice adaptation is able to reproduce a speaker’s distinctive character, emotional range, and speaking style across languages, using just 30 minutes of audio. 

  5. Your voice AI, on your infrastructure. Use Mistral's EU-hosted API or deploy Voxtral 3 on your own infrastructure. Keep sensitive conversations within your environment and control whether any audio is retained. 

Try Voxtral now  | Learn more

It hears 50+ languages, with state-of-the-art accuracy.

Voxtral Transcribe 3 represents a significant milestone across transcription, diarization (who spoke when) and context biasing capabilities. The supported list of languages is expanded to 50, including 24 official EU languages and many low-resource Asian and Indic languages.

The Small version achieves state-of-the-art results across diarization and multilingual transcription, including in English, French and Spanish. It outperforms Whisper, Scribe v2 and Gemini 3.5 Transcribe. The Mini variant is 4B parameters, enabling fast and accurate transcriptions at a price of $0.0035/min.

Both models support context biasing, where key terms and free-form text can be given to bias the transcription and improve recall on technical terms. This enables them to be used in production settings with domain-specific vocabulary without the need for fine-tuning.

Voxtral Realtime Transcribe 3 is a realtime transcription model built for agentic use-cases. It achieves high-quality transcriptions at sub-500ms latency in over 35 languages, including with diarization and context biasing.

FLEURS 35

Realtime transcription Word Error Rate — lower is better

4.0%4.2%5.7%5.8%6.0%6.4%7.4%8.4%8.7%Voxtral Mini Transcribe 3*Gemini 3.5 Realtime TranscribeVoxtral Mini Realtime 3 · 2400 msMicrosoft AI Transcribe 2 StreamingVoxtral Mini Realtime 3 · 960 msVoxtral Mini Realtime 3 · 480 msVoxtral Mini Realtime 3 · 240 msGPT Live TranscribeScribe v2 RealtimeWord Error Rate

Macro-average WER over 35 languages from the FLEURS benchmark. WER is used for all languages except Japanese, Korean, and Chinese, which use CER. Lower is better. *Voxtral Mini Transcribe 3 is an offline baseline. The remaining models are realtime.

Languages: AR, BN, BG, CA, CS, NL, EN, ET, FI, FR, DE, EL, GU, HI, ID, IT, JA, KN, KO, LV, MK, MS, ML, MR, NO, FA, PL, PT, RO, RU, SK, ES, FIL, TE, TR

End-of-turn prediction is natively supported, incorporating both acoustic and semantic information to accurately predict when a user is finished speaking. This allows for low-latency voice agent applications, while minimizing false cut-off rates when the user is prematurely interrupted.

It answers with the right emotion, in your or your customer’s brand's own voice and local dialect.

Voxtral Mini TTS 3.0, available in batch and realtime, achieves state-of-the-art expressiveness and instruction following for 35+ languages and 50+ dialects. At $13 per 1M characters, Voxtral Speech 3 costs less to run than comparable models from OpenAI TTS-1, Deepgram, ElevenLabs Flash and Gemini Flash.

The speech model implicitly interprets the semantic meaning of context better, producing natural speech across a diverse range of emotions and languages. New and improved features include voice cloning from 30+ minutes of sampled audio, custom prompt instruction following (IF), contextual TTS, and a vocabulary library, all of which serve the goal of speech reliability in deployment. Variables for Explicit IF include emotion, pacing, and emphasis.

Prompt

Let's end the debate here: it's GIF.

Non-standard phoneme hint: dʒ ɪ f

All natural-sounding speech relies on emotion, which determines its prosody, rhythm, pacing, emphasis, and transitions. Voxtral TTS natively interprets a prompt’s text context to determine the emotion and voice style, making the model inherently better at natural speech without requiring additional instructions or voice prompts.

Illustrating Voxtral hill-climbing in expressiveness, numbers and unique names.

Prompt

Thank you for calling Mistral! I understand you're having difficulty logging into AI Studio. Can I confirm your email address? I have here [email protected], is that correct?

Speech sounds local in 35 languages and over 50 dialects.

Voxtral TTS supports over 35 languages and regional dialects. Our multilingual evals adhere to dialect differences in pronunciation, formal vs. informal register, spelling, verbosity, and locale conventions (time, dates, temperatures, addresses, currencies).

Voxtral 3 multilingual library

Voice Presets are available in 18 languages: Arabic (Emirati, Modern Standard), Brazilian (Portuguese), Catalan, Chinese (Mandarin), Dutch, English (American, British), French (France), German, Greek, Hindi, Italian, Korean, Norwegian, Russian, Spanish (Spain), and Vietnamese.

The voice agent is in control of the whole conversation.

Voxtral 3 introduces a voice agent SDK, built on LiveKit, that runs speech in, reasoning, and speech out as one streaming pipeline. Voxtral Mini Transcribe Realtime 3 listens and detects when the caller has finished speaking, a general model—like Mistral Small—reasons and calls tools, and Voxtral Speech 3 replies in the tone of the conversation so far. No stage waits for the one before it, so each turn completes in under one second, and the agent stops talking when the caller cuts in.

During a call, the agent can verify a caller's identity, check a bill, or book an appointment, using function calling, web search, and retrieval over your own systems. It answers and places calls over SIP, runs in browsers and apps over WebRTC, and connects to the telephony, IVR, and contact-center software you already use.

Each stage is a separate model, so you can change the LLM, the voice, or the language without rewriting the agent, and run the whole stack on Mistral's API in Europe or on your own infrastructure.

Get started.

Voxtral 3 is available today through the Mistral API, hosted through our European, US, and global endpoints.

You can test out voice agents, audio transcription, speech generation, and voice cloning in the Studio playgrounds. 

Model

Price

Voxtral Mini Transcribe 3 (batch)

$0.0035/min

Voxtral Mini Transcribe Realtime 3

$0.008/min

Voxtral Small Transcribe 3

$0.0040/min

Voxtral Speech 3 (batch and realtime, same API)

$13/1M chars

To run the full stack on-premises or in a private cloud, talk to our team. 

联系我们 contact @ memedata.com