SpeechVerse vs. SOTA: Multi-Task Speech Models in Real-World Benchmarks

Table of Links

Abstract and 1 Introduction

2 Approach

2.1 Architecture

2.2 Multimodal Instruction Finetuning

2.3 Curriculum Learning with Parameter Efficient Finetuning

3 Experiments

4 Results

4.1 Evaluation of SpeechVerse models

4.2 Generalization Across Instructions

4.3 Strategies for Improving Performance

5 Related Work

6 Conclusion, Limitations, Ethics Statement, and References

A Appendix

A.1 Audio Encoder Pre-training

A.2 Hyper-parameters

A.3 Tasks

4.1 Evaluation of SpeechVerse models

We evaluate end-to-end trained joint speech and language models (E2E-SLM) leveraging the SpeechVerse framework on 11 unique tasks across multiple domains and datasets. We first evaluate SpeechVerse’s core speech understanding capability through ASR benchmarks. We then evaluate more complex SLU tasks and paralinguistic speech tasks in Tables 2 and 3 respectively.

4.1.1 Performance on ASR and SLU tasks

First, we evaluate the performance of SpeechVerse models on four public ASR benchmark datasets namely libri-test-clean, libri-test-other, Voxpopuli and CommonVoice. The WER numbers for each of these datasets are reported in Table 2. SpeechVerse ASR in row 2 uses same model as task-specific pretrained ASR model (Task-FT) in row 3. When comparing our task-specific pretrained ASR model, which also serves as the initialization for multi-task finetuning, to Whisper ASR, our model achieves slightly better performance on average. However, the WER increases in both multitask models with Multitask-WLM performing similarly to Whisper across three out of the four test sets. The lower performance of the multi-task SpeechVerse model compared to the task-specialized model is likely due to giving lower weight to ASR datasets when constructing batches during multi-task training. This was done to balance the performance across all tasks, since the data distribution is imbalanced between the different tasks.

When it comes to SLU tasks, a frequent question posed is if an end-to-end model can outperform a cascaded pipeline that transcribes speech via ASR and then feeds it to a language model. To investigate this, we conducted experiments on five semantic understanding tasks using the same foundation models as SpeechVerse. The text foundation model was further fine-tuned on data from the five SLU tasks separately, as we found the zero-shot performance of Flan-T5 on these benchmark test sets to be quite poor. We also report performance when feeding ground-truth transcripts into the fine-tuned LLM, to provide upper bound results. On 4 of the 5 tasks, excluding Keyword Extraction, the end-to-end trained models outperform the cascaded pipeline. In particular, the more commonly used tasks like intent classification, slot labeling, and speech translation are performing better than the cascaded system, demonstrating the efficacy of our models trained using SpeechVerse. We also observed that SpeechVerse models on KWS task are outperforming cascaded pipeline by an absolute 10% in accuracy, while performing significantly behind on KWE task. Since the keyword search task requires an attention span focused on a specific word of interest, joint modeling helps improve accuracy by overcoming error propagation present in a cascaded pipeline. We also conducted an ablation study to determine if the KWE task benefits further from joint decoding of ASR transcription and keywords. We noticed an improvement in performance, closing the gap to the cascaded pipeline. The results from this study are detailed further in the sub-section 4.3.2. When comparing the multitask models to the task-specific SpeechVerse models, there is a minor degradation in performance, but the difference is not substantial. Overall, the multitask model trained with either WavLM encoder or Best-RQ encoder outperformed cascaded systems in majority tasks.

4.1.2 Performance on paralinguistic tasks

The results in Table 3 demonstrate clear improvements in performance on various paralinguistic speech processing tasks when using multi-task learning compared to fine-tuning the WavLM model independently for each task. Specifically, the SpeechVerse model trained with multitask learning using Best-RQ audio encoder (Multitask-BRQ) achieves gains over the baseline WavLM model of 4.8% absolute on emotion recognition, 6.6% on audio sentiment classification, and 2.5% on accent classification. More modest gains are seen with the SpeechVerse model trained using multitask learning with WavLM encoder (Multitask-WLM). The unified representation’s adaptive combination of all encoder layers helps multitask BEST-RQ model improve diverse paralinguistic task performance. Overall, multi-task learning provides noticeable improvements in model generalization and effectiveness across a diverse set of speech tasks compared to task-specific fine-tuning of the baseline WavLM model. The results highlight the advantages of learning shared representations across related tasks using multi-task learning techniques.

4.1.3 Comparison against SOTA models

Table 4 benchmarks SpeechVerse models against state-of-the-art (SOTA) models on five diverse tasks: automatic speech recognition (ASR), speech translation (ST), intent classification (IC), slot filling (SF), and emotion recognition (ER). Across these tasks, SpeechVerse demonstrates competitive or superior performance compared to prior specialized models. When comparing our task-specific pretrained ASR model, which also serves as the initialization for multi-task finetuning, to Whisper ASR, our model achieves slightly better performance on average. However, the multitask model (Multitask-WLM) performed similarly to Whisper across three out of the four test sets. When evaluating on speech translation across three language pairs, the task-specialized SpeechVerse model

surpassed SeamlessM4T on two pairs, while the multi-task SpeechVerse model achieved competitive performance compared to prior work on average. Both models did not perform well on English to Romanian pair. The overall performance of the SpeechVerse models on speech translation is heavily limited by the capabilities of the underlying language model FlanT5. The speech translation capabilities cannot exceed the translation quality provided by FlanT5 as the base language model. To evaluate SpeechVerse on spoken language understanding tasks like intent classification (IC) and slot filling (SF), we retrained the task-specialized SpeechVerse model by incorporating all 69 intents (both seen and unseen) as well as all slots. This allowed us to compare SpeechVerse to prior work on the complete intent and slot sets. Our SpeechVerse model achieved competitive performance to the previous SOTA (PF-hbt-large) on slot filling, but was significantly behind on intent classification with 5% lower absolute accuracy. However, SpeechVerse outperformed the same SOTA model (Frozen-hbt-large) by 10% when the encoder weights were frozen during fine-tuning. To further analyze the gap to prior state-of-the-art, we conducted an experiment allowing the audio encoder weights to be tunable during fine-tuning. This achieved 89.5% accuracy, matching the prior SOTA. This suggests the intent classification performance can overfit to the specific acoustic conditions of the SLURP dataset when full fine-tuning is performed. The SpeechVerse model that was trained end-to-end specifically for the task of emotion recognition achieved an 8% absolute improvement in unweighted average recall over the previous state-of-the-art model (w2v2-L-robust). In contrast, the multitask SpeechVerse model performed 3% better than the prior state-of-the-art. However, one key difference is that the previous SOTA work trained on the MSP-Podcast 1.7 dataset, while we used version 1.11 for training. The test set version remained the same between the two approaches. Overall, the SpeechVerse model demonstrated competitive performance compared to prior specialized models in some cases when evaluated across the various tasks.