This paper was accepted into the Ninth Conference on Machine Translation (WMT24) at EMNLP 2024.
The prosody of a spoken utterance, including features such as stress, intonation and rhythm, can significantly affect the underlying semantics and, as a consequence, can also affect its textual translation. However, prosody is rarely studied in the context of speech-to-text translation (S2TT) systems. In particular, it has been proposed that end-to-end (E2E) systems are suitable for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but it is not yet known whether this is successful. in practice. limited. A main challenge is the difficulty of assessing prosodic awareness in translation. To address this challenge, we present an evaluation methodology and a focused benchmark (called ContraProSt) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrasting examples. Through experiments in translating English speech into German, Spanish, and Japanese, we found that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascade systems also capture prosodic information in translation, but only in minor extent, as it depends on the details of the transcription. superficial form.
