Table of Links Abstract and 1. Introduction Related Work
Model
Experiments
Ablation Study
Conclusion, Limitations, and Risks 4. Experiments This section covers empirical details of model training and evaluation on the task of speechto-speech translation with speaker style transfer. We consider Spanish-to-English (Es-En) and Hungarian-to-English (Hu-En) translation as representative translations between similar and distant languages. Data. We used in-house semantically aligned data consisting of 250k Spanish-English and 300k Hungarian-English speech pairs. A set of 1k samples were randomly taken from training data as the validation set. Table 2 summarizes all data statistics. Automatic evaluation metrics. We reuse ASRBLEU metric to measure the semantic translation quality as existing studies of speech translation. Generated audios are first transcribed with ASR tools into texts, and we use the medium-sized Whisper English ASR model[1] in this work. BLEU evaluates the ngram overlap between the transcripts and ground truth translations. Another important metric is vocal style similarity (VSim), quantifying how similar the generated audios sound in comparison with the source speech in term of vocal style. Following (Barrault et al., 2023), we extract vocal style embeddings of generated and source speech with a pretrained WavLM encoder (Chen et al., 2022) and compute the cosine similarity of their embeddings. Higher embedding similarity indicates better vocal style transfer. Furthermore we report model size, the number of parameters, to reflect the model efficiency. Subjective metrics. Besides automatic metrics, we include Mean Opinion Score (MOS) as the subjective metric to measure the speech quality with scores ranging from 1 to 5 (higher score indicates better quality). We get 50 samples from the model outputs in each test set, and two annotators independently evaluate the quality, and we report the score averaged over annotators and test samples. Models. We included recent S2ST models which are only trained on semantically aligned speech data for a fair comparison with SEAMLESSEXPRESSIVELM. • Enc-Dec S2ST. We included a textless translation model with encoder-decoder architecture as a strong baseline for semantic translation (Duquenne et al., 2023). The source speech is encoded by convolutional and Transformer encoder layers, and its decoder predicts the target semantic units. A separately trained vocoder synthesizes target speech from semantic units. • Cascaded LMs[2]. PolyVoice (Dong et al., 2023) tackles S2ST with a cascade of three LMs for (1) semantic unit translation, (2) the first acoustic stream generation, (3) remaining acoustic streams generation. LM-1 is a decoder-only model translating source to target semantic units, LM-2 and LM-3 follow VALL-E for acoustic generation which is trained on target speech i.e., English audios. SEAMLESSEXPRESSIVELM consists of 12 AR decoder layers and 12 NAR layers. The token embedding dimension is 512. Each layer is Transformer decoder layer with 16 attention heads, the layer dimension of 1024 and the feedforward dimension of 4096. The model is trained with 0.1 dropout at the learning rate of 0.0002. The cascaded model consists of three LMs each of which has 12 Transformer decoder layers and other hyperparameters same as SEAMLESSEXPRESSIVELM. As for Enc-Dec S2ST, we set its convolution layers as 2, encoder layers of 6 and decoder layers of 6 with layer and feedforward dimensions of 1024 and 4096 so that its parameter size match that of semantic LM in the cascaded model. Training. For SEAMLESSEXPRESSIVELM, the training acoustic prompt ratio is uniformly sampled between 0.25 and 0.3, and the inference prompt ratio is 0.3 in Spanish-to-English translation. The training ratio is 0.20-0.25, and the inference ratio is 0.2 for Hungarian-to-English translation. Section 5.2 discusses how the training and inference ratios are selected for Spanish-to-English. For cascaded LMs, the acoustic prompt is fixed as 3 seconds as used in (Wang et al., 2023). Inference. We set beam search size of 10 to all models during inference. The SEAMLESSEXPRESSIVELMsamples acoustic units with the temperature of 0.9. For cascaded LMs, its semantic-toacoustic generation uses a temperature of 0.8 following (Wang et al., 2023). For LM-based S2ST models, target speech is synthesized by EnCodec decoder from acoustic units. For Enc-Dec, since it predicts semantic units, a pretrained unit based HiFi-GAN vocoder is applied to synthesize speech following (Lee et al., 2022). 4.1 Semantic Prompt Results. Table 3 reports model size, ASR-BLEU and VSim of different S2ST models for Es-En and Hu-En. Enc-Dec model serves as the upper bound of semantic translation quality, while it does not preserve speaker vocal style. SEAMLESSEXPRESSIVELM outperforms cascaded LMs with +10.7% and +7.2% similarity in two directions. It performs on par with Cascaded LMs in terms of semantic translation as reflected by comparable ASRBLEU scores. In Table 4, SEAMLESSEXPRESSIVELM shows good MOS in comparison with cascaded LMs. We note that SEAMLESSEXPRESSIVELM has better efficiency with 312M parameters in comparison with Cascaded LMs with 469M parameters. Authors:
(1) Hongyu Gong, Meta AI (hygong@meta.com);
(2) Bandhav Veluri, Meta AI (bandhav@meta.com). This paper is available on arxiv under CC BY 4.0 DEED license. [1] https://huggingface.co/openai/whisper-medium.en [2] We use the architecture of PolyVoice, and replace SoundStream with Encodec as acousitc units Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work Model Experiments Ablation Study Conclusion, Limitations, and Risks Related Work Related Work Model Model Experiments Experiments Ablation Study Ablation Study Conclusion, Limitations, and Risks Conclusion, Limitations, and Risks 4. Experiments This section covers empirical details of model training and evaluation on the task of speechto-speech translation with speaker style transfer. We consider Spanish-to-English (Es-En) and Hungarian-to-English (Hu-En) translation as representative translations between similar and distant languages. Spanish-to-English Hungarian-to-English Data . We used in-house semantically aligned data consisting of 250k Spanish-English and 300k Hungarian-English speech pairs. A set of 1k samples were randomly taken from training data as the validation set. Table 2 summarizes all data statistics. Data Automatic evaluation metrics. We reuse ASRBLEU metric to measure the semantic translation quality as existing studies of speech translation. Generated audios are first transcribed with ASR tools into texts, and we use the medium-sized Whisper English ASR model[1] in this work. BLEU evaluates the ngram overlap between the transcripts and ground truth translations. Automatic evaluation metrics. Another important metric is vocal style similarity (VSim), quantifying how similar the generated audios sound in comparison with the source speech in term of vocal style. Following (Barrault et al., 2023), we extract vocal style embeddings of generated and source speech with a pretrained WavLM encoder (Chen et al., 2022) and compute the cosine similarity of their embeddings. Higher embedding similarity indicates better vocal style transfer. Furthermore we report model size, the number of parameters, to reflect the model efficiency. vocal style similarity Subjective metrics. Besides automatic metrics, we include Mean Opinion Score (MOS) as the subjective metric to measure the speech quality with scores ranging from 1 to 5 (higher score indicates better quality). We get 50 samples from the model outputs in each test set, and two annotators independently evaluate the quality, and we report the score averaged over annotators and test samples. Subjective metrics. Models . We included recent S2ST models which are only trained on semantically aligned speech data for a fair comparison with SEAMLESSEXPRESSIVELM. Models • Enc-Dec S2ST. We included a textless translation model with encoder-decoder architecture as a strong baseline for semantic translation (Duquenne et al., 2023). The source speech is encoded by convolutional and Transformer encoder layers, and its decoder predicts the target semantic units. A separately trained vocoder synthesizes target speech from semantic units. • Cascaded LMs[2]. PolyVoice (Dong et al., 2023) tackles S2ST with a cascade of three LMs for (1) semantic unit translation, (2) the first acoustic stream generation, (3) remaining acoustic streams generation. LM-1 is a decoder-only model translating source to target semantic units, LM-2 and LM-3 follow VALL-E for acoustic generation which is trained on target speech i.e., English audios. SEAMLESSEXPRESSIVELM consists of 12 AR decoder layers and 12 NAR layers. The token embedding dimension is 512. Each layer is Transformer decoder layer with 16 attention heads, the layer dimension of 1024 and the feedforward dimension of 4096. The model is trained with 0.1 dropout at the learning rate of 0.0002. The cascaded model consists of three LMs each of which has 12 Transformer decoder layers and other hyperparameters same as SEAMLESSEXPRESSIVELM. As for Enc-Dec S2ST, we set its convolution layers as 2, encoder layers of 6 and decoder layers of 6 with layer and feedforward dimensions of 1024 and 4096 so that its parameter size match that of semantic LM in the cascaded model. Training . For SEAMLESSEXPRESSIVELM, the training acoustic prompt ratio is uniformly sampled between 0.25 and 0.3, and the inference prompt ratio is 0.3 in Spanish-to-English translation. The training ratio is 0.20-0.25, and the inference ratio is 0.2 for Hungarian-to-English translation. Section 5.2 discusses how the training and inference ratios are selected for Spanish-to-English. For cascaded LMs, the acoustic prompt is fixed as 3 seconds as used in (Wang et al., 2023). Training Inference . We set beam search size of 10 to all models during inference. The SEAMLESSEXPRESSIVELMsamples acoustic units with the temperature of 0.9. For cascaded LMs, its semantic-toacoustic generation uses a temperature of 0.8 following (Wang et al., 2023). For LM-based S2ST models, target speech is synthesized by EnCodec decoder from acoustic units. For Enc-Dec, since it predicts semantic units, a pretrained unit based HiFi-GAN vocoder is applied to synthesize speech following (Lee et al., 2022). Inference 4.1 Semantic Prompt Results . Table 3 reports model size, ASR-BLEU and VSim of different S2ST models for Es-En and Hu-En. Enc-Dec model serves as the upper bound of semantic translation quality, while it does not preserve speaker vocal style. SEAMLESSEXPRESSIVELM outperforms cascaded LMs with +10.7% and +7.2% similarity in two directions. It performs on par with Cascaded LMs in terms of semantic translation as reflected by comparable ASRBLEU scores. In Table 4, SEAMLESSEXPRESSIVELM shows good MOS in comparison with cascaded LMs. We note that SEAMLESSEXPRESSIVELM has better efficiency with 312M parameters in comparison with Cascaded LMs with 469M parameters. Results Authors: (1) Hongyu Gong, Meta AI (hygong@meta.com); (2) Bandhav Veluri, Meta AI (bandhav@meta.com). Authors : Authors (1) Hongyu Gong, Meta AI (hygong@meta.com); (2) Bandhav Veluri, Meta AI (bandhav@meta.com). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv [1] https://huggingface.co/openai/whisper-medium.en [2] We use the architecture of PolyVoice, and replace SoundStream with Encodec as acousitc units

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Ablation Study Reveals the Role of Semantic & Acoustic Prompts in SEAMLESSEXPRESSIVELM's Performance

Advancements in Speech Translation Address Inefficiency and Errors in Cascaded Models

SpeechSynthesis.Tech

Too Long; Didn't Read

Boost your HackerNoon story @ $159.99! 🚀

Experiments on SEAMLESSEXPRESSIVELM Reveal Its Superiority in Efficiency and Translation Quality

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Ablation Study Reveals the Role of Semantic & Acoustic Prompts in SEAMLESSEXPRESSIVELM's Performance

Getting the Most out of a Large Language Model

Deductive Verification of Chain-of-Thought Reasoning in LLMs

Solving the AI Hallucination Problem with Self-Verifying Natural Programs

Breaking Down Deductive Reasoning Errors in LLMs

Ablation Study Reveals the Role of Semantic & Acoustic Prompts in SEAMLESSEXPRESSIVELM's Performance

Getting the Most out of a Large Language Model

Deductive Verification of Chain-of-Thought Reasoning in LLMs

Solving the AI Hallucination Problem with Self-Verifying Natural Programs

Breaking Down Deductive Reasoning Errors in LLMs

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps