SourceLLaMA-Omni: Seamless Speech Interaction with Large Language Models
When applied to speech-to-text instruction following on instructs2s-eval, Salmonn and Qwen2-Audio generate responses that include ordered lists, parentheses, and lengthy explanations not appropriate for speech synthesis. The paper attributes this to the fact that these models, designed as speech-to-text systems, have not been aligned with speech interaction scenarios. Their ChatGPT scores (3.44/3.40 for Salmonn, 3.47/3.38 for Qwen2-Audio) are lower than Llama-Omni's (3.99/3.47), and in the streaming scenario their scores and ASR-WER show significant gaps compared to offline, indicating additional errors from word-level streaming TTS.