（评论）

（评论）
(comments)

原始链接: https://news.ycombinator.com/item?id=41582180

基于文本的模型和音频语言模型 (ALM) 之间的比较凸显了它们在处理和生成自然语言方面的差异。文本到文本 (STT/TTS) 模型严重依赖机器学习算法，将口语转换为书面形式，反之亦然，从而导致转换过程中由于误解、缺乏语调变化、节奏和情感细微差别而造成潜在损失。相反，ALM 旨在通过分析音频信号中更广泛的上下文来保留这些元素。与 TTS 模型相比，这使它们能够提供更真实、更有表现力和更流畅的响应，而 TTS 模型必须猜测语调变化，并且通常无法捕捉预期的语气。此外，与 STT/TTS 系统不同，ALM 可以不受干扰地参与自然对话、模仿人类交互、生成音乐作品或识别哼唱播放的歌曲，并提供额外的好处，例如改善响应延迟、说话者分类和增强对对话的反应停顿。然而，正如作者指出的，ALM 的有效性取决于高质量的训练数据，这使得它们的可靠性可能较低，除非配备足够的计算资源。最后，作者表示渴望进一步尝试语音模型，例如 OpenAI 的高级语音模式，该模式拥有改变语速、纠正不良发音以及识别用户发音不准确等功能。总体而言，虽然基于文本的 ALM 和 ALM 都表现出独特的优势，但 ALM 为捕捉自然交流的复杂性和参与动态的交互式对话提供了更多的机会。

I’m not clear on the virtues or potential of a model like this over a pure text model using STT/TTS to achieve similar results.

Is the idea that as these models grow in sophistication they can properly interpret (or produce) inflection, cadence, emotion that’s lost in TTS?

There's a lot of data loss and guessing with STT/TTS.

An STT model might misrecognize a word, but an audio LLM may understand the true word because of the broad context. A TTS model needs to guess the inflection and it can get it completely wrong, but an audio LLM could understand how to talk naturally and with what tone (e.g. use a higher tone if it's interjecting)

Speaking of interjection, an STT/TTS system will never interject because it relies on VAD and heuristics to guess when to start talking or when to stop, and generally the rule is to only talk after the user stopped talking. An audio LLM could learn how to conversate naturally, avoid taking up too much conversation time or even talk with a group of people.

An audio LLM could also produce music or sounds or tell you what the song is when you hum it. There's a lot of new possibility

I say "could learn" for most of this because it requires good training data, but from my understanding most of these are currently just trained with normal text datasets synthetically turned into voice with TTS, so they are effectively no better than a normal STT/TTS system; it's a good way to prove an architecture but it doesn't demonstrate the full capabilities

You need a lot more power. I found gpt4o struggles doing basic OCR of printed text by hallucinating alot, while tesseract engine (old skool) gets it perfect. You need the model to be powerful enough to do everything.

You can work around this by the way by sending the output through a checking stage.

So picture -> gpt4o -> out1, picture -> tesseract -> out2, out1,out2 -> llm.

Might work for sound too.

Interesting, I've actually been using gpt4o extensively for OCR and didn't encounter any significant issues - could I ask you to please give an example of an image of (otherwise legible) text that it hallucinates on?

Speech is inherently easier to represent as a sequence of tokens than a high-resolution image.

Best speech to text is already NN transformer based anyway, so in theory it's only better to use a combined model

Personally, I'm very much looking forward to using a speech model like OpenAI's advanced voice mode to learn language. It can already do things like speak quickly or slowly which traditional TTS systems can't. Also, in theory a speech model could tell me if my pronunciation is accurate. It could correct me by repeating my incorrect pronunciation and then providing the correct pronunciation. I don't actually know how capable OpenAI's advanced voice mode is in this regard because I haven't seen anyone actually test this but I'm extremely curious to try it myself. If other voice models can achieve this then it will be an incredible tool for language learning.

Lots has been written on this subject, check out OpenAI’s papers on -O for example.

Latency is a big one due to batching. You can’t really interrupt the agent, which makes actual conversation more clunky. And yes, multimodal has better understanding. (I haven’t seen analysis of perception of emotions, has anyone seen analysis of this capability for GPT-O?)

Essentially, there's data loss from audio -> text. Sometimes that loss is unimportant, but sometimes it meaningfully improves output quality.

However, there are some other potential fringe benefits here: improving the latency of replies, improving speaker diarization, and reacting to pauses better for conversations.

> I’m not clear on the virtues or potential of a model like this over a pure text model

you can't put pure text with keyboard on a robot. it will become a wheeled computer.

actually this is a cool thing as a companion / assistant.

Really

Yeah that's the point. Without punctuation, no one can tell what inflection my "really" above should have, but even if it'd been "Really?" or "Really!", there's still room for interpretation. With a bet on voice interfaces needing a Google moment (wherein, prior to Google, search was crap) to truely become successful (by interpreting and creating inflection, cadence, emotion, as you mentioned), creating such a model makes a lot of sense.

So it's not STT -> LLM -> TTS? If I scream Chewbacca noises as input, will the model recognize it as nonsense, or will it interpret it with some lousy STT as some random words?

It's not, but it probably won't recognize it as nonsense. According to the paper,

> we construct a dataset named InstructS2S-200K by rewriting existing text instruction data and performing speech synthesis

It has only been trained on questions spoken by TTS, it has never seen (heard) nonsense. Most likely it'll just hallucinate that you asked some question and it'll generate some answer instead of asking if you're good. There's just not many audio datasets with real voices, there's no audio version of StackOverflow to be scraped

I used to have fun with that. Set Google Translate to Chinese (Or some other language I don't speak, though tonal languages seemed to work better), make some vague noises into it, and get out coherent but crazy phrases in English.

The speed looks very nice. I just recently setup LMStudio + AnythingLLM to try out local voice chat and its still a little slower than I'd like but the PiperTTS voices are nicer than this.

Thank you.

Obviously it doesn’t sound human but that’s extremely impressive for an 8B model. Compared to the Moshi model also on the front page now, this model seems to be more coherent, but maybe less conversational?

Am I the only one who trusts a GitHub repo much less when it has one of those stupid star history graphs on the readme?

（评论） (comments)

（评论）
(comments)