Table of Links Abstract and 1. Introduction Related Work
Model
Experiments
Ablation Study
Conclusion, Limitations, and Risks 2. Related Work S2ST. The series of Translatotron models have encoder-decoder architecture to translate source speech into target spectrogram, which could be synthesized as waveform with a separately trained vocoder. Translatotron uses a speaker encoder to enable voice conversion in translated speech (Jia et al., 2019). Translatotron 2 removes speaker encoder from the model design for the purpose of antispoofing, and creates speaker-aligned data with a cross-lingual TTS model for model training (Jia et al., 2022). Therefore the translation model learns to transfer vocal style with a data-driven approach. Translatotron models also leverage textual supervision by using phoneme in the auxiliary task of speech recognition task. With semantic units emerging as an efficient semantic representation of speech, unit based translation models are developed. The textless S2ST model learns to map source speech to target units without relying on textual data such as phonemes (Lee et al., 2022), and the target speech could be synthesized from semantic units with HiFi-GAN vocoder (Gong et al., 2023). Despite the good semantic quality, semantic units do not capture speaker vocal style, and thus the unit based S2ST lost the style information in speech translations. To enhance expressivity transfer, it is proposed by Duret et al. to integrate speaker and emotion encoder into the translation system. The system consists of speech-to-unit translator and unit-to-speech synthesizer with speaker and emotion embeddings. S2ST with language model. Recent progress in acoustic units such as EnCodec (Défossez et al., 2022) and Soundstream (Zeghidour et al., 2022) capture richer acoustic and style information than semantic speech units by using multiple codebooks to encode residual information in each codebook. This also unlocked the speech language modeling in vocal style transfer (Wang et al., 2023). With acoustic units as translation target, the translation model is trained to preserve the style of source speaker. Existing works are built upon a cascaded architecture (Dong et al., 2023; Zhang et al., 2023) consisting of speech-to-unit translation, primary acoustic unit and residual acoustic unit generation. The first component translates target semantic units from source speech. The other two components take care of speaker style transfer, and generate the first stream of acoustic units and remaining streams sequentially. This type of model has general limitations of cascaded architecture, which are inefficiency and error propagation. A multi-task learning framework is recently designed (Peng et al., 2024), which improves model efficiency by sharing parameters between the first and second components. However, it still goes with cascaded training and inference of the three components. We come up with SEAMLESSEXPRESSIVELM which is trained in an end-to-end manner with chain-of-thought. It shows better parameter efficiency and improved performance compared with the cascaded model. Authors:
(1) Hongyu Gong, Meta AI (hygong@meta.com);
(2) Bandhav Veluri, Meta AI (bandhav@meta.com). This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work Model Experiments Ablation Study Conclusion, Limitations, and Risks Related Work Related Work Model Model Experiments Experiments Ablation Study Ablation Study Conclusion, Limitations, and Risks Conclusion, Limitations, and Risks 2. Related Work S2ST . The series of Translatotron models have encoder-decoder architecture to translate source speech into target spectrogram, which could be synthesized as waveform with a separately trained vocoder. Translatotron uses a speaker encoder to enable voice conversion in translated speech (Jia et al., 2019). Translatotron 2 removes speaker encoder from the model design for the purpose of antispoofing, and creates speaker-aligned data with a cross-lingual TTS model for model training (Jia et al., 2022). Therefore the translation model learns to transfer vocal style with a data-driven approach. Translatotron models also leverage textual supervision by using phoneme in the auxiliary task of speech recognition task. S2ST With semantic units emerging as an efficient semantic representation of speech, unit based translation models are developed. The textless S2ST model learns to map source speech to target units without relying on textual data such as phonemes (Lee et al., 2022), and the target speech could be synthesized from semantic units with HiFi-GAN vocoder (Gong et al., 2023). Despite the good semantic quality, semantic units do not capture speaker vocal style, and thus the unit based S2ST lost the style information in speech translations. To enhance expressivity transfer, it is proposed by Duret et al. to integrate speaker and emotion encoder into the translation system. The system consists of speech-to-unit translator and unit-to-speech synthesizer with speaker and emotion embeddings. S2ST with language model. Recent progress in acoustic units such as EnCodec (Défossez et al., 2022) and Soundstream (Zeghidour et al., 2022) capture richer acoustic and style information than semantic speech units by using multiple codebooks to encode residual information in each codebook. This also unlocked the speech language modeling in vocal style transfer (Wang et al., 2023). S2ST with language model. With acoustic units as translation target, the translation model is trained to preserve the style of source speaker. Existing works are built upon a cascaded architecture (Dong et al., 2023; Zhang et al., 2023) consisting of speech-to-unit translation, primary acoustic unit and residual acoustic unit generation. The first component translates target semantic units from source speech. The other two components take care of speaker style transfer, and generate the first stream of acoustic units and remaining streams sequentially. This type of model has general limitations of cascaded architecture, which are inefficiency and error propagation. A multi-task learning framework is recently designed (Peng et al., 2024), which improves model efficiency by sharing parameters between the first and second components. However, it still goes with cascaded training and inference of the three components. We come up with SEAMLESSEXPRESSIVELM which is trained in an end-to-end manner with chain-of-thought. It shows better parameter efficiency and improved performance compared with the cascaded model. Authors: (1) Hongyu Gong, Meta AI (hygong@meta.com); (2) Bandhav Veluri, Meta AI (bandhav@meta.com). Authors : Authors (1) Hongyu Gong, Meta AI (hygong@meta.com); (2) Bandhav Veluri, Meta AI (bandhav@meta.com). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Experiments on SEAMLESSEXPRESSIVELM Reveal Its Superiority in Efficiency and Translation Quality

Future of Speech Translation with SEAMLESSEXPRESSIVELM

SpeechSynthesis.Tech

Too Long; Didn't Read

Boost your HackerNoon story @ $159.99! 🚀

Advancements in Speech Translation Address Inefficiency and Errors in Cascaded Models

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Ablation Study Reveals the Role of Semantic & Acoustic Prompts in SEAMLESSEXPRESSIVELM's Performance

SEAMLESSEXPRESSIVELM Unifies Semantic & Acoustic Modeling for Efficient Speech Translation

Future of Speech Translation with SEAMLESSEXPRESSIVELM

Getting the Most out of a Large Language Model

Deductive Verification of Chain-of-Thought Reasoning in LLMs

Ablation Study Reveals the Role of Semantic & Acoustic Prompts in SEAMLESSEXPRESSIVELM's Performance

SEAMLESSEXPRESSIVELM Unifies Semantic & Acoustic Modeling for Efficient Speech Translation

Future of Speech Translation with SEAMLESSEXPRESSIVELM

Getting the Most out of a Large Language Model

Deductive Verification of Chain-of-Thought Reasoning in LLMs

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps