NEWSLETTER
Real -time speech translation presents a complex challenge, which requires a perfect integration of voice recognition, automatic translation and text synthesis to voice. Traditional cascade approaches often introduce compound errors, do not retain the identity of the speaker and suffer slow processing, which makes them less suitable for real -time applications such as live interpretation. In addition, existing simultaneous translation models fight to balance precision and latency, trusting complex inference mechanisms that are difficult to climb. A significant barrier remains the lack of large -scale and well -aligned voice data sets, which limits the ability to train models that can generate contextually precise and natural translations with a minimum delay.
Kyutai has developed HibikiA decoder model of 2.7 billion parameters designed for the translation of voice voice in real time (S2ST) and voice to text (S2TT). Operating a 12.5Hz framework with a 2.2KBPS bit rateHibiki is currently compatible French to English translation and is designed to preserve the voice characteristics in the translated output. A distilled version, Hibiki-m (1.7b parameters), It is optimized for real -time performance on smartphones, which makes it more accessible to translation in the device.
Technical approach and benefits
Hibiki's only decoder architecture enable the simultaneous processing of speech using a multistream language model that predicts both Text and audio tokens. Use a Neural Audio Codec (Mimi) Compress the audio while maintaining fidelity, ensuring an efficient translation generation. A key aspect of its design is Contextual alignmentA method that takes advantage of the perplexity of a text translation model to determine the optimal moment to generate discourse, allowing Hibiki Adjust the translation delays dynamically while maintaining coherence. In addition, Hibiki admits Lot inferenceprocessing up 320 parallel sequences at GPU H100making it viable for large -scale applications. The model is trained in 7m Audio hours in English, 450k French hours and 40k hours of synthetic parallel datacontributing to his robustness through various speech patterns.
Performance and evaluation
Hibiki has demonstrated strong performance in the quality of the translation and loyalty of the speaker. Achieve a 30.5 Asr-200 scoreovercoming the existing baselines, including offline models. Human evaluations qualify their naturalness at 3.73/5approaching 4.12/5 Professional human interpreter score. The model also works well in Speedness of the speakerWith a 0.52 similarity score compared to 0.43 for perfect. Compared to Without seams and transmission spacesHibiki offers constantly Higher translation quality and Best voice transferwhile maintaining a Competitive latency. The distillate Hibiki-M The variant, although slightly lower in the similarity of the speakers, remains effective for use in the device in real time.
Conclusion
Hibiki provides a practical approach for real -time speech translation, integrating Contextual alignment, efficient compression and real -time inference Improve the quality of translation while preserving the characteristics of natural speech. Offering a Open source launch under a permissive CC-byHibiki has the potential to contribute significantly to advances in multilingual communication.
Verify he Paper, Models in the hugged face, Github page and Colab notebook. All credit for this investigation goes to the researchers of this project. Besides, don't forget to follow us <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter and join our Telegram channel and LINKEDIN GRsplash. Do not forget to join our 75K+ ml of submen.
<a target="_blank" href="https://x.com/i/communities/1670488129348960258″ target=”_blank” rel=”noreferrer noopener”>Unique our automatic learning community on twitter/<a target="_blank" href="https://x.com/i/communities/1670488129348960258″ target=”_blank” rel=”noreferrer noopener”>unknown
Asif Razzaq is the CEO of Marktechpost Media Inc .. as a visionary entrepreneur and engineer, Asif undertakes to take advantage of the potential of artificial intelligence for the social good. Its most recent effort is the launch of an artificial intelligence media platform, Marktechpost, which stands out for its deep coverage of automatic learning and deep learning news that is technically solid and easily understandable by a broad audience. The platform has more than 2 million monthly views, illustrating its popularity among the public.
