这个AI可以听到,翻译和回应100种语言

作者 : Loïc Barrault 朱元 Mariano Coria Meglioli David Dale Ning Dong Paul-Ambroise Duquenne Hady Elsahar 香港高 Kevin Heffernan John Hoffman Christopher Klaiber Pengwei Li Daniel Licht Jean Maillard Alice Rakotoarison Kaushik Ram Sadagopan Guillaume Wenzek Ethan Ye 作者 : 洛伊克·巴拉尔特 朱元 马里亚诺·科里亚·梅格利奥利 戴维·戴尔 唐宁 保罗·安布罗伊斯·杜克内 哈迪·艾尔萨哈 香港高 凯文·赫弗纳 约翰·霍夫曼 克里斯托弗·克莱伯 金正恩李 丹尼尔 光 吉恩·梅拉德 爱丽丝·拉科托里森 卡舒希克·拉姆·萨达戈潘 威廉·温兹克 伊丹·耶 抽象 创建一个帮助个人在任何两种语言之间翻译语言的工具Babel Fish需要先进的技术创新和语言专业知识,尽管传统的语音对话翻译系统由多个子系统组成,以随机的方式进行翻译。 , , ,可扩展和高性能的统一系统 , 为了解决这一差距,我们介绍了SEAMLESSM4T – 大量多语言和多式机器翻译 – 支持语音到语音翻译(101到36种语言),语音到文本翻译(从101到96种语言),文本到语音翻译(从96到36种语言),文本到文本翻译(96种语言)和公共语音识别(96种语言)的单一模型。使用一种新的多式语音到文本翻译和其他公开可用的数据,SEAMLESSM4T是第一个能够翻译语音到语音(96到36种语言)的多语言系统之一,可以翻译语音和文本(96种语言)的任务。此外,它优于现有的最先进的语音识别系统(96种语言)。 1 2 3 4 5 主要 巴比尔鱼从 这是一个虚构的工具,可以翻译两种语言之间。在当代的全球环境中,其特点是互联性和移动社交性日益增加,在数字世界和物理世界中实现这些技术的现代化和按需语音对话翻译(S2ST)的社会迫切需要从未如此之大。尽管日常通信中语音的中心性,今天的机器翻译(MT)系统仍然以文本为导向。 了解更多关于为什么在MT中优先考虑演讲的细节,尽管单一的模型,如没有语言落后(NLLB) 通过将文本翻译(T2TT)覆盖范围推向200多种语言,统一的S2ST模型远未达到类似的范围或性能,这种差异可以归因于许多原因,但音频数据稀缺和建模限制仍然是关键障碍。 《Hitchhiker’s Guide to the Galaxy》 I1 的 6 现有的 S2ST 系统有三个主要缺点:第一,这些系统倾向于专注于高资源语言,留下许多低资源语言。第二,这些系统主要服务于从源语言翻译到英语(X-eng),而不是反向翻译(eng-X)。 , , ,他们不匹配他们的反弹对手的表现 参见附加信息部分 更多关于当前技术景观的细节。 1 4 5 7 二、 为了解决这些局限性,我们引入了SEAMLESSM4T(大规模多语言和多式机器翻译),一个支持ASR、T2TT、语音到文本翻译(S2TT)、文本到语音翻译(T2ST)和S2ST的统一系统。 然后,我们将这个集合物的过滤子集与人标签和假标签数据相结合,开发了第一个从100多种语言执行S2ST到36种语言的多任务系统,S2TT和ASR到96种语言,零射 T2ST到36种语言,以及T2TT到96种语言(参见表)。 关于语言覆盖和补充信息部分的比较概述 由于SEAMLESSM4T的统一架构(图。 ),该模型可以以零射程的方式执行非英语方向的T2TT,S2TT或S2ST(X-X)。它还可以执行T2ST而没有被明确训练为此任务。 ”),它可以处理混合两种或多种语言的发音。 8 1 二 1 未经监督的演讲预练 是的 UNITY2(S2ST精细调节)的三个主要块与其非自我退缩(NAR)T2U在左上方显示。多任务-UNITY2及其附加文本编码器在左下方显示。 为了评估我们模型的输出质量,我们使用了多个现有指标,涵盖任务和模式,以及四个主要的评估数据集。 对于 T2TT,蓝色 对于 S2TT,ASR-BLEU 对于 S2ST 和 WER 对于 ASR. 参见补充表 我们还测试了我们的模型对背景噪音或扬声器变异的弹性,以及其他负责任部署的战线,如性别偏见,使用MULTILINGUAL HOLISTICBIAS数据集,或增加毒性,使用新的基于语音的指标(ASR-ETOX和MuTox)。 . 9 10 5 2 11 除了建造SEAMLESSM4T之外,我们还讨论了我们工作的社会影响以及它如何有助于提高世界准备水平。 从长远来看(参见“ 为了推动未来的研究,我们将数据工具、代码和两种尺寸的SEAMLESSM4T模型提供给公众非商业用途。 12 社会影响与结论 在接下来的部分中,我们追踪了开发我们的SEAMLESSM4T模型的关键结果。 首先,我们概述了我们挖掘对齐语音和文本数据的努力,从语音识别开始,到使用模型无形编码器挖掘对齐语音和文本细分。 接下来,我们报告了我们直接翻译系统的主要结果,部分由上述自动对齐语音数据训练。 这些结果凸显了SEAMLESSM4T模型在ASR、T2TT、S2TT和S2ST中实现多语言最先进的性能的任务多功能性。 数据 培训语音翻译系统需要标签数据;即语音对文本和语音对语音对准数据.然而,对于低资源语言,这些资源非常有限。 以及大量的原始言论和文本,如在 ,自动挖掘对齐资源,补充现有人标签和伪标签数据。 8 方法 语言识别 从网络中处理原始语音包括将发言分成更短的片段,然后是语言识别,建立在使用ECAPA-TDNN架构的VoxLingua107训练的开源模型上 , ,我们开发了一种新的基于语音的语言识别(LID)模型,涵盖本文所介绍的所有100种语言(参见 ,” ” 更多细节)。 13 14 方法 音频处理和语音 LID 为了衡量LID模型的精度和召回,我们在扩展数据表中报告测试数据上的F1分数 数据为100个SEAMLESSM4T语言(总体)和79个SEAMLESSM4T和VoxLingua107(中间部分)之间的通用语言。请注意,VL107HF的所有语言中的宏观F1是低的,因为21种语言不受该模型所涵盖。我们发现,对额外的语言的培训略有降低了共同语言的总体表现,这是较多的近语言的直接结果。例如,Zulu(zul)经常与Nyanja(nya),Ibo(ibo)与Yoruba(yor)混淆,以及现代标准阿拉伯语(arb)与摩洛哥阿拉伯语(ary)和埃及阿拉伯语(arz模型)的培训略有降低。我们提高分类准确性(F1差异大于5%)在17种语言中,平均增长 1 SONAR 文本嵌入空间 为了从语言识别的细分中提取自动对齐的翻译数据,我们依赖于语言和模式不符的编码器。 目前,我们为 200 种语言提供单个文本编码器和解码器,并为 37 种语言提供语音编码器。 在多语言相似性搜索中,xsim 和 xsim++ 是两个已知的代理指标,用于评估用于采矿目的的多语言嵌入空间。 ,SONAR显著超过其他流行的方法,如LASER3 或拉布斯 用较低的 xsim 和 xsim++。 8 6 15 2 16 17 此外,我们在 T2TT 任务中评估了 SONAR 文本编码器和解码器. 超过 200 种语言的平均性能与中型 NLLB 密集模型相比具有竞争力,尽管 SONAR 将编码器和解码器的注意力替换为固定大小的瓶颈嵌入(参见扩展数据表的 T2TT 列) 这个结果表明,使用注意力不是合理的翻译准确性的要求。 2 演讲编码器培训 语音编码器只在语音转录上接受了教师-学生方法的培训(参见 ,” ’) 评估每个语音编码器在一个端到端循环中的迭代,即采矿和培训S2TT或S2ST翻译系统,将是计算密集的。相反,我们将语音编码器与SONAR文本编码器连接,并评估了这个零打 S2TT系统作为编码器质量的代理。 ,SONAR语音编码器与FLORES上的WHISPER-LARGE-V2这样的模型比较有利 和花 在一些高资源语言(如德语,俄语或葡萄牙语)中,可观察到精度差距,但SONAR在一些低资源语言(如斯瓦希里语或孟加拉语)中优于WHISPER-LARGE-V2(参见补充表) )。 方法 索尼尔 2 6 18 8 西雅图 SONAR 文本和语音编码器用于采集三种类型的对齐数据:(1)英语语音与非英语文本(Sen2Txx);(2)非英语语音与英语文本(Sxx2Ten);以及(3)非英语语音与英语语音(Sxx2Sen)。SEAMLESSALIGN 在 Sen2Txx中提供 202,796 小时的音频,在 Sxx2Ten 中提供 239,767 小时的音频,在 Sxx2Ten 中提供 29,161 小时的音频。 . 8 对于文本域,我们使用了NLLB项目合并的相同数据 这些数字从马耳他语或斯瓦希里语等低资源语言的3300万条或5500万条短语到22000万条英语短语。 6 除了马耳他语,我们只获得了少量的原始音频,我们能够为所有语言挖掘超过100小时的英语语音对比。 与英语文本的对比达到了大多数语言的1000小时,高资源语言超过10,000小时。 添加如此大量的数据来训练多语言翻译系统是一个巨大的计算挑战。 ,” 由于我们的挖掘数据可以帮助支持许多不同的使用案例,我们开源了指导其复制所需的元数据(高达 1.15 的 SONAR 门槛;参见 ,” ’)允许社区重建SEAMLESSALIGN并将其用于自己的目的,因此可以根据任务调整最佳门槛,平衡数据集大小和对齐质量,我们的采矿代码也在STOPES库中开源。 方法 模型 方法 发言人 多任务翻译系统模型 结合在模型中描述的模型技术 ,” ’,与SEAMLESSALIGN的其他数据(见 ,” ’),我们训练了两种尺寸的SEAMLESSM4T模型:具有 2.3B参数的大型和具有 1.2B参数的中型。SEAMLESSM4T-MEDIUM旨在成为一个易于访问的测试床,无论是精细调节,改进或参与分析。 ,” ’)和更强大的单元解码器(参见“ 所有 SEAMLESSM4T 模型都支持文本模式中的 96 种源语言,语音模式中的 100 多种源语言。在目标方面,模型可以以文本形式输出 96 种语言,语音形式输出 35 种语言。 和 这表明,对于一些翻译方向和缺乏监督数据,我们的模型将被评估为零射击。 方法 模型 方法 数据 方法 未经监督的演讲预练 S2ST 精密调节 12 13 我们对所有四个受监督的任务(T2TT,ASR,S2TT和S2ST)以及文本对话翻译的零拍任务(T2ST,也称为跨语言文本对话合成)进行了评估。 ) 为了生成文本假设,我们用雷射搜索解码。我们用 chrF2++ 给 T2TT 和 BLEU 给 S2TT。 对于ASR,我们用WER(单词错误率)在标准化转录和引用后得分。 . 19 2 20 在 S2ST 和 T2ST 推断中,我们执行了双通路射线搜索解码;从第一通路解码中获取的最佳假设与文本解码器嵌入,并发送到一个文本到单元模块(T2U;参见“部分”)。 ’)寻找最佳单元序列假设. 我们为两次搜索使用了5的光束宽度. 我们使用Whisper模型评估了ASR-BLEU的S2ST和T2ST准确性.我们将Whisper的解码温度设置为零,并使用贪婪的解码来确保ASR模型的确定性行为。 在计算蓝色分数之前。 S2ST 精密调节 20 与语音翻译的散步方法的比较 对于 SEAMLESSM4T 和 WHISPER 支持的语言集,我们在表中比较 (S2TT列) 我们的直接 S2TT 模型的性能,即 WHISPER ASR 模型和 NLLB T2TT 模型的组合。 SEAMLESSM4T-V2 以 4.6 BLEU 点(从 22.0 到 26.6)在 X 方向和 1 BLEU 点(从 21.1 到 22.2 方向)在 X-eng 方向中超过了 3B 参数。 这些模型超过了4B参数,大多超过了SEAMLESSM4T-V2在X-eng (+3.9);他们在eng-X方向只有微小地超过SEAMLESSM4T-V2的0.2 BLEU点。 3 3 与之前直接S2TT SOTA模型相比,这些模型落后于形系统(例如,AUDIOPALM-2-8B-AST。 ), SEAMLESSM4T-V2 提高了 FLEURS X-eng S2TT BLEU 得分 6.9 点(从 19.7 到 26.6;即 35% 的提高)。 21 桌子 (S2ST列)还将S2ST与SEAMLESSM4T模型进行比较.对于S2ST,我们探讨了两种可变的选项:(1)与ASR、T2TT 和 TTS 进行三阶段,(2)与S2TT 和 TTS 进行两阶段。 当英语语言和MMS的合成时 我们的SEAMLESSM4T-LARGE在FLEURS X-eng方向上的两阶段滑动模型比作8个ASR-BLEU点(对比为SEAMLESSM4T的支持和MMS的TTS系统的支持)。我们的SEAMLESSM4T-LARGE在FLEURS X-eng方向上的两阶段滑动模型比作8个ASR-BLEU点(对比为S2 FSTLE Xeng的支持为17.8-25.8).它还比S2 FSTLE Xeng更强的三阶段滑动模型(WHISPER-LARGE-V2 + NLLB-3.3B + YOURTTS)更强劲,以21个ASR-BLEU点(23.7-25.8).改进的SEAMLESSM4T-V2进一步增强了S2 FSTLE 3 22 23 我们分享在补充信息部分 对 S2TT 和 S2ST 任务的评估结果与额外的指标,包括我们的模块性不确定性 BLASER 2.0。 四一 多任务结果 我们在桌子上报告 对于ASR和零射击T2ST任务的FLEURS基准(X-eng和eng-X),以及T2TT的相关FLORES基准(X-eng和eng-X)的结果;在ASR中,SEAMLESSM4T-LARGE优于WISPER-LARGE-V2 在重叠的77种支持的语言中,WER减少了46%(从41,7到22,6),而SEAMLESSM4T-V2比WHISPER-LARGE-V2提高了56%(从41,7到18,5)。 对MMS SEAMLESSM4T-V2 比 CTC 评估的 MMS 变体高出 38% 的 WER (从 31.0 到 19.1),但它被利用单语言 n-gram 语言模型的变体所超越(比 18.6 更好的 WER 更好 5%)。 4 20 9 23 在 T2TT 支持任务中,表中的结果 顯示我們的 SEAMLESSM4T 模型與 NLLB-3.3B (ref. )在X-eng和eng-X方向。 4 6 接下来,我们以零拍的方式评估了SEAMLESSM4T模型,以T2ST的任务。鉴于FLEURS为每个样本收集了三个不同的母语音器的三个记录,我们随机选择了一个用于T2ST的任务(输入是文本)。 (T2ST 列)是 SEAMLESSM4T 模型与 NLLB 和 YOURTTS (英语 TTS) 或 MMS (非英语 TTS) 之间进行的比较,用于合成翻译文本。我们在 88 个 X-eng 方向上平均了 ASR-BLEU 得分(FLEURS 和 SEAMLESSM4T 支持的语言的重叠)。我们还平均了 ASR-BLEU 在 26 个 eng-X 方向(SEAMLESSM4T 与 TTS 模型的重叠)。 与 cascading 模型相比,SEAMLESSM4T-LARGE V2 的零射击能力与 NLLB-3.3B + YOURTTS 在 X - 中相同,并且超过了 NLLB-3.3B + MMS 的 +3.9 4 为了进一步了解FLEURS S2TT X-eng方向的改进来自何方,我们根据资源级别对语言进行了排列(参见补充表中的语言的确切列表) )并在表中报告每个资源级别的平均BLEU分数 结果表明,SEAMLESSM4T 显著提高了低资源语言的翻译质量,提高了 +10.2 BLEU(从 18.0 到 28.2,即比 AUDIOPALM-2-8B-AST 提高了 57%)。 12 5 自动和人为评估 语音翻译中的语义准确性通常用自动计量BLEU进行评估。 对于 S2TT 或其扩展 ASR-BLEU 对于 S2ST. 此外,我们使用 BLASER 2.0 (ref. )的Blaser的扩展 ,现在允许对语音和文本进行模式性评估和质量评估。 10 24 25 为了补充自动指标的实用性,我们还依赖于对我们的模型进行广泛的人类评估,以下是使用XSTS(跨语言语义文本相似性)协议为S2TT和S2ST任务提供人类评估。 MOS(平均意见分数)协议用于语音输出(见 ,” 然而,由于资源限制,我们涵盖了有限的模型(SEAMLESSM4T-LARGE、SEAMLESSM4T-LARGE V2 和由 WHISPER-LARGE-V2 用于 ASR、NLLB 3.3B 用于翻译和 YOURTTS 或 MMS 用于 S2ST 任务)和翻译指令(英语和英语的 23 种语言,MOS 的 10 种语言 X-eng)。 26 方法 人类评价 XSTS 得分表明,SEAMLESSM4T-LARGE V2 在平均语言水平的 XSTS 得分和赢利率(评估语言的比例优于 XSTS)方面,对于所有任务和语言方向都具有较高的信心。对于 S2ST 任务来说,SEAMLESSM4T-V2 的相对表现是最强的,SEAMLESSM4T-V2 方法的赢利率与 cascaded 基线和 SEAMLESSM4T-LARGE 两种方向相比为 100%,语言平均 XSTS 得分大约 0.5 点高于 cascaded 基线和 0.36-0.5 点高于 SEAMLESSM4T-LARGE for eng–X 和 X-eng,参见相应的附加表。 和 分别为完整的语言水平和总结的 XSTS 结果。 22 24 我们还使用中等意见分数协议来衡量S2ST的语音输出质量,该协议评估了(1)声音质量、(2)语音清晰度和(3)自然性。我们发现,一般来说,在所有MOS方面,SEAMLESSM4T-LARGE V2倾向于优先于SEAMLESSM4T-LARGE,该协议倾向于优先于随机模型基线,除了X-eng,为此SEAMLESSM4T-LARGE世代更受欢迎 (+ SEAMLESSM4T-LARGE和SEAMLESSM4T-LARGE V2世代之间平均差异1点),这可能是由于模型架构的差异而导致的意外结果。 和 for full language-level and summarized MOS results, respectively. 23 24 使用 S2ST 任务的 XSTS 评估,BLASER 2.0 (在给定语言方向的所有评估项目的平均值) 实现了优越的 Spearman 相关性,以校准的语言水平的 XSTS 得分在 X-eng 方向 (0.845 对于 BLASER 2.0 与 0.74 对于 ASR-BLEU) 以及特别是在 eng-X 方向 (0.81 对于 BLASER 2.0 与 0.246 对于 ASR-BLEU) 类似的结果持有 S2TT 任务(参见补充表) for full results). 21 最后,我们测试了我们的模型在噪音和扬声器变化方面的强度,通过基于FLEURS创建开放的强度基准(见FLEURS)。 ,” 为此,我们发现,与WHISPER-LARGE-V2相比,SEAMLESSM4T-V2平均对背景噪音和扬声器变异的抵抗力大约为42%和66%(参见附加信息部分的完整结果)。 ). Methods 可靠性 V2 的 负责的人 Toxicity 毒性可以定义为可能煽动对个人或团体(如宗教,种族或性别)的仇恨,暴力或虐待的邪恶或语言的例子。 在语音翻译的背景下,我们主要担心增加的毒性 - 引入在译文中不存在的有毒元素。 and, recently, for tens of languages with MuTox . 27 28 29 因此,对于语音和文本的多语言毒性检测,我们使用ETOX(或ASR-ETOX为S2ST)和MuTox(为语音和文本)作为指标来检测和评估添加毒性。 时间的推移(参见) ,” ” )。 11 方法 毒性检测 We computed added toxicity in two datasets (FLEURS and HOLISTICBIAS ) 跨 24 个英语翻译方向(arb, ben, cat, ces, dan, deu, est, fin, fra, hin, ind, ita, nld, pes, pol, por, rus, slk, spa, swh, tgl, tur, urd, vie),使用系统覆盖的交叉点上的语言和 MuTox 进行了基准(请注意,MuTox 具有更广泛的语言覆盖范围,类似于 SONAR,但仅在 30 种语言中进行了基准) ) 请参阅 ETOX 评估的更多翻译指南的结果,参见附加信息部分 桌子 该研究表明,虽然添加毒性水平和类型因语言和数据集而异,但我们系统中的添加毒性在我们两种毒性检测指标(<0.4%)中具有相对较低的普遍性。 对于所有模式、方向和数据集,最低的毒性是与 SEAMLESSM4T-V2 + MinTox 一致获得的,在与基线相比,在不使用 MinTox 的情况下与相同模型相比,在 MuTox 方面达到 5% 的毒性减少(在 ETOX 方面达到 80%),在 MuTox 方面达到 20% 的毒性减少(在 ETOX 方面达到 90% )(参见附加信息部分的完整结果)。 )。 27 29 见1 6 6 见1 Gender bias Gender bias in the context of MT can be defined as errors in grammatical gender determination. This bias may manifest explicitly as an overgeneralization to one gender when translating non-gendered to gendered forms (for example, outputs favouring masculine representations) or as a lack of robustness when varying the quality of the translation for sentences that differ only in gender inflection. Previous work on this matter is mostly in the text modality , , and tends to be English-centric, with few demographic axes and multilingual references. Similar efforts for the speech modality remain sparse , . 30 31 32 33 34 We used MULTILINGUAL HOLISTICBIAS 和它的语音扩展来比较S2TT和S2ST的性能。eng-X方向允许在男性或女性参照的情况下比较性能,而X-eng方向允许在我们改变性别转换时进行翻译的可靠性比较。 典型的英语 - 西班牙语语言对的例子是“我是一名家务工”和相应的翻译“Soy amo de casa”和“Soy ama de casa”在西班牙语中。 翻译从英语到西班牙语时,我们可以测量系统是否过度通俗化到一个性别,而在其他方向,我们可以评估转换到性别转换的可靠性(见) , ‘ ’). 35 方法 Speech extension of MULTILINGUAL HOLISTICBIAS We conducted a set of comprehensive evaluations on translation biases for S2TT and S2ST (see Extended Data Table 平均结果和补充信息部分 for detailed results). SEAMLESSM4T-V2 consistently improves robustness in gender variations across metrics and tasks. When compared with the previous model, SEAMLESSM4T-V2 improves over SEAMLESSM4T-LARGE by 0.4% in S2TT and by 0.1% BLASER 2.0 in S2ST, and it beats the external baseline system of WHISPER-LARGE-V2 (+YOURTTS) by 0.1% in S2TT and by 0.9% BLASER 2.0 for S2ST. However, SEAMLESSM4T-V2 is not able to consistently improve in terms of gender overgeneralization compared with the previous model. SEAMLESSM4T-V2 is comparable in terms of BLASER 2.0 to SEAMLESSM4T-LARGE, but it lags far behind in terms of ASRchrf (by 2.2%), and overgeneralization is increased by 0.2% when it comes to S2TT. Although we can increase bias robustness by improving the overall quality of the model, it seems that we need specific techniques to counteract the overgeneralization of the model towards a specific gender. 3 六二 社会影响与结论 The world we live in has never been more interconnected—the global proliferation of the internet, mobile devices, communicative platforms and social media exposes individuals to more multilingual content than ever before 当前的社会秩序对一个人的世界准备性提出了要求。 最初开发在语言学习的背景下,世界准备强调了能够在我们的母语以外的语言中沟通的重要性,无论是工具(即就业或学习)还是文化原因(即成为全球公民)。 36 12 The downstream applications that SEAMLESSM4T supports could allow on-demand access to world-readiness by streamlining multilingual exchange across various contexts. SEAMLESSM4T-supported applications could act as a co-piloting mechanism that supports users in multilingual conversations and boosts their confidence in speech-heavy interactions. As speech-based interfaces (for example, audio assistants, voice memos and live transcriptions) and auditory content (for example, podcasts, audiobooks and short-form videos) become ever more present, SEAMLESSM4T-enabled downstream applications could unlock a greater variety of multilingual experiences. From an inclusion standpoint, the focus of SEAMLESSM4T on multimodality could make a meaningful difference in augmenting the world-readiness of those with accessibility needs and those whose languages can be transcribed with multiple writing systems. For many who lack reading or writing skills, or cannot rely on sight (that is, people who are blind or with visual impairment), voice-assisted technologies are essential to how they communicate and stay connected . The ability to translate speech gives these groups more comprehensive access to information not only beyond their native languages but also in a manner that is better suited for their communicative needs. 37 As with most technologies, the distribution of benefits varies based on user demographics and social situation . Although we make the case that SEAMLESSM4T could augment world-readiness by lowering the barriers in cross-lingual communication, some users may experience more difficulties using our work than others. For instance, similar to many other speech technologies, the ASR performance of SEAMLESSM4T may vary based on gender, race, accent or language , 此外,我们的系统在语法或适当的词汇翻译中的表现也可能在高和低资源语言之间不一致。 38 39 40 S2ST的另一个挑战是,语音倾向于比书面语言更容易接收和反馈。换句话说,扬声器在确认输出质量或在现场对话中进行编辑的能力有限。没有能力通过背后的翻译或母语翻译来规划和修订,S2ST在错误翻译或毒性方面可能带来更高的交互风险。我们敦促使用SEAMLESSM4T精心调节或构建文物的研究人员和开发人员批判性地思考可以帮助用户绕过这些潜在的障碍的设计功能。重要的是,我们认为SEAMLESSM4T支持的应用程序最好被视为协助翻译的增强装置,而不是替代语言学习或可靠的人类口译员的需求的工具。 最后,言语不是口语文本,它包括一套前语(例如,节奏,压力,语调或语调)和需要进一步研究的情感成分。 . To create S2ST systems that feel organic and natural, more research should be directed at output generation that preserves expressivity 此外,巴贝尔鱼的最终实现需要对低延迟语音翻译的研究进行更深入的投资。 , . We hope that SEAMLESSM4T opens up new possibilities for both these research areas. 41 42 43 44 Methods 数据 Developing an effective multilingual and multimodal translation system such as SEAMLESSM4T requires sizeable resources across languages and modalities. Some human-labelled resources for translation are freely available, but often limited to a small set of languages or in very specific domains. Well-known examples are parallel text collections such as Europarl 联合国科普斯 . A few human-created collections also involve the speech modality, such as CoVoST , and mTEDx . Yet no open dataset currently matches the size of those used in initiatives such as WHISPER or USM , which proved to unlock unprecedented performance. 45 46 47 48 49 20 50 平行数据挖掘出现作为使用封闭数据的替代方案,无论是语言覆盖范围还是体积大小。今天的主导方法是将来自不同语言和模式的句子编码到一个固定大小的嵌入空间中,并根据相似度指标找到平行实例。 , . This approach was first introduced using the multilingual LASER space . Teacher–student training was then used to scale this approach to 200 languages , 其次,演讲方式 , . 51 52 53 6 16 54 55 音频处理和语音 LID We started with 4 million hours of raw audio originating from a publicly available repository of crawled web data on which we applied several cleaning and filtering operations. To maximize the recall of mining, it is important that all segments have a similar granularity. For the text domain, a sentence is generally well-defined. This is less obvious for raw speech because pauses are not necessarily used at sentence boundaries. First, we used an open Voice Activity Detection model to split audio files into shorter segments. Second, a newly developed speech LID model was applied to each segment. Our model follows the ECAPA-TDNN architecture 并扩展了VoxLingua107上训练的开源模型 by 15 new languages. Finally, we applied an over-segmentation approach that simultaneously proposed multiple, potentially overlapping speech segmentations. We relied on the mining approach to align the most likely ones. Supplementary Fig. shows this pipeline. 56 13 14 1 SONAR The SONAR text and speech encoders were developed in ref. using a two-step approach (Supplementary Fig. 首先,仅为文本模式学习了大量的多语言表示。然后,教师-学生培训被用来扩展嵌入空间到语音模式。文本嵌入空间被用编码器-解码器方法训练,使用多个目标的组合:翻译,解码自动编码和句子嵌入空间中的平均平方错误(MSE)损失目标。 , that is, parallel data to translation from 200 to 200 languages. Speech encoders were trained only on ASR data and by grouping languages into linguistic genealogical groups following ref. , for example, Italic, Common Turkic or Indo-Iranian languages. To obtain optimal performance, we determined the optimal convergence separately for each language (that is, when to stop training). This yielded a separate speech encoder for each language. The amount of available ASR data for each language is provided in Supplementary Table . The speech encoders were initialized with w3v-best 2.0 speech front end. Preceding work performed max pooling or mean pooling of the output states of the speech front end to obtain a fixed-size embedding of the speech signal , 一项废除研究表明,可以通过使用三层变压器解码器获得更好的结果。 . Teacher–student training consisted of minimizing the MSE loss with respect to the embedding of the ASR text transcriptions. These embeddings were obtained by the SONAR text encoder, which was kept constant. No translations (into English) were used. 8 2 6 16 8 54 57 8 发言人 We first calculated the embeddings of all over-segmented speech segments. For the text domain, we used exactly the same texts as the NLLB project and embedded them with the SONAR encoder. Exhaustive pairwise comparison can be efficiently performed with the FAISS toolkit . Similarity is measured with a margin criterium as first introduced in ref. : 6 58 52 哪里 和 are the source and target sentences, and ( )标志着 nearest neighbours of 在其他语言中,我们设定 十六。 x y NNK 的 x k x k As an example, this amounts to comparing a hundred thousand hours of speech with more than 20,000 million English sentences, which yielded about eight thousand hours of aligned Arabic speech. Modelling SEAMLESSM4T 模型基于我们的多任务 UNITY 架构,我们提出的统一翻译模型基于 Vanilla UNITY ,一个两个通道的解码框架,首先生成文本,然后通过预测分散的声学单元生成语音(参见“部分” )。 ’). Compared with the vanilla UNITY model , (1) the core S2TT model, initialized from scratch in UNITY, is replaced with an X2T model that supports text as input and is pretrained to jointly optimize the tasks of ASR, S2TT and T2TT (see section ‘ ’),和(2)浅薄的T2U模型(被称为T2U单元编码器和第二通道单元解码器。 ) is replaced with a deeper transformer-based encoder–decoder model with six transformer layers that are pretrained on ASR data (see section ‘ ’). An improved version of UNITY, dubbed UNITY2, replaces the autoregressive T2U with a new non-autoregressive (NAR) T2U decoder. This NAR T2U model delivers stronger accuracy because of its hierarchical upsampling from subwords to characters and then to units. 59 Multilingual discrete acoustic units 59 X2T fine-tuning 59 S2ST 精密调节 X2T的预训练产生了更强的语音编码器和更高质量的第一通道文本解码器,而T2U模型的扩展和预训练使我们能够更好地处理无干扰的多语言单元生成。 Multilingual discrete acoustic units 最近的作品通过使用自我监督的分散声学单元作为构建直接语音翻译模型的目标,实现了最先进的翻译性能 , 这包括将S2ST问题分解为语音对单元的翻译步骤和单元对单元的语音转换步骤。 and mapped these representations to discrete tokens. The set of discrete tokens (also referred to as unit vocabulary) is learnt by applying a -means algorithm to a set of multilingual audio samples. The -means centroids resemble a codebook that is used to map a sequence of XLS-R speech representations into a sequence of centroid indices or acoustic units. We used a unit vocabulary size = 10,000 具有来自 XLS-R-1B 35 层的功能,代表 35 种支持的目标语言。 5 60 61 k k K 对于单元到语音转换步骤,我们遵循ref。 该模型负责从SEAMLESSM4T模型将预测的单元序列中合成音频。 62 Unsupervised speech pretraining 使用未标记的语音数据进行自我监督预培训是利用未标记的数据的一种实用方法. 通过预训练,我们可以启动翻译模型的质量,并充分利用我们的受监督的配对数据。 in combining contrastive learning with masked prediction learning. W2V-BERT 2.0 uses more codebooks and an additional masked prediction task using random projection quantizers (RPQ). Our W2V-BERT 2.0 model is first trained on 1 million hours of open speech audio data that covers over 143 languages. It follows the w2v-BERT XL architecture ,具有 24 个 Conformer 层 并且大约有 600 万个模型参数。对于 v2 版本,我们将未标记的数据量从 1 万小时扩展到 4.5 万小时的音频。 它只接受了150万小时的培训,涵盖了1400多种语言,规模最大的模型是USM . It is a proprietary multilingual speech pretrained model with 12 million hours of data and more than 300 languages in coverage. 63 64 63 65 23 50 Text-to-text translation models The text processing components of our SEAMLESSM4T models were pretrained on the task of text-to-text translation, a much more resourced task than speech translation. Consider, for instance, the English–Italian direction, one of the highly resourced pairs in T2TT with more than 128 million parallel sentences—only 2 million pairs of English text paired with Italian audio are available for S2TT. A key step in training multilingual text-to-text translation models is learning a shared vocabulary with a text tokenizer. Following ref. ,我们使用SentencePiece with the BPE algorithm for this purpose. The tokenizer used in NLLB-200 由于采样文物而缺乏中国关键字符,这种采样不利于具有大量独特符号的标志图形写作系统,为了解决这个问题,我们强迫将这些字符纳入其中。 6 66 67 6 To train our multilingual text-to-text model, we followed the same data preparation and training pipelines in ref. 使用停止 由于语言覆盖范围较小,我们可以将模型的大小显著降低到 1.3B 参数,并且只使用 95 种 SEAMLESSM4T 语言的 NLLB-200 培训数据。 6 68 通过伪标签增加数据 As with any sequence-to-sequence task, speech translation performance is dependent on the availability of high-quality training data. However, the amount of human-labelled data is scarce compared with its T2TT or ASR counterparts. To address this shortage of labelled data, we resort to pseudo-labelling , 使用多语言的 T2TT 模型(例如 NLLB 模型)生成伪标记 S2TT 数据的 ASR 数据。 69 70 为了增加 S2ST 数据,通常使用 TTS 模型将语音数据集中的文本转换为合成语音。 , . This synthetic speech is, in turn, converted into discrete units for training. This two-step unit extraction process is a slow process and is harder to scale given the dependencies on TTS models. We circumvented the need for synthesizing speech and trained multilingual text-to-unit (T2U) models on all 36 target speech languages. These models can directly convert the text into target discrete units and can be trained on ASR data sets that are readily available. 4 5 X2T fine-tuning The first key part of our multitask UNITY framework is the X2T model, a multi-encoder sequence-to-sequence model with a conformer-based encoder 用于语音输入和其他基于变压器的编码器 两种编码器都与相同的文本解码器相结合,并共同精心调整,以优化ASR、S2TT和T2TT的任务。 65 71 我们的X2T模型包括将语音编码器,W2V-BERT 2.0从M2连接,后固定与长度适配器,以下示长的音频序列,与文本编码器 - 解码器从M3(补充图。 )对于长度适配器,我们使用了修改后的M适配器。 , in which we replaced the three independent pooling modules for Q, K and V with a shared pooling module to improve efficiency. 4 72 ( speech) in a source language ⟨ ⟩, paired with its transcription ( text) and text translation ( text) in a target language ⟨ ⟩. To enable meaning transfer across modalities, X2T model was fine-tuned to jointly optimize the following objective functions: x x y ℓt We additionally optimized an auxiliary objective function in the form of token-level knowledge distillation () to further transfer knowledge from the strong MT model to the student speech translation task (S2TT). The final loss is a weighted sum of all three losses: where , and are scalar hyper-parameters tuned on the development data. α β γ S2ST 精密调节 In the last stage of fine-tuning multitask UNITY, we initialized the model with the pretrained X2T model (see section ‘ ’)和一个预先训练的T2U模型,类似于在M4中用于伪标签S2ST数据的模型。用于伪标签的T2U模型被称为教师T2U模型,具有12个转换器层的编码器-解码器。为初始化,我们使用了一个小学生T2U模型,只有六层来优化推断和蒸更强的T2U的标签。 and extended it to discrete unit generation. UNITY2 starts with hierarchically upsampling the T2U encoder output from subword length to character length and then to unit length. The unit duration predictor, the key to the hierarchical upsampling, is supervised during training by a multilingual aligner based on RAD-TTS . The architecture is shown in detail in Supplementary Information section . X2T 精密调节 73 74 四八 我们用X-eng和eng-X S2ST翻译数据的组合对S2ST任务进行了精心调整,总计为121,000小时,我们冻结了与X2T模型相符的模型重量,仅对T2U组件进行了精心调整,以确保模型在上一阶段精心调整的任务上的性能保持不变。 自动和人为评估 蓝色 2.0 BLASER 2.0 (ref. )是BLASER的新版本 , which works with both speech and text modalities, and hence being modality-agnostic. Like the first version, our approach leverages the similarity between input and output sentence embeddings. The new version uses SONAR embeddings Supplementary Information section ,支持57种语音语言和202种文本语言(本文提交时由SONAR覆盖语言),并可扩展到未来的编码器,用于共享相同嵌入空间的新语言或模式。 24 75 三、三一 更具体地说,在BLASER 2.0中,我们将源输入、任何S2ST、S2TT 或 T2TT 模型的翻译输出和参考语音部分或文本转换为SONAR嵌入引擎。 人类评价 除了 (ASR) BLEU 和 BLASER 2.0 等自动指标之外,我们还使用了 XSTS 等人类指标。 , which measures semantic similarity between a source and target translation, and a standard Mean Opinion Score (as standardized in Recommendation ITU-T P.800, henceforth MOS), which measures (1) naturalness, (2) sound quality and (3) clarity of audio generations to evaluate our models. To obtain more robust language-level scores, we also incorporate a calibration set and calibration methodology, the same used to evaluate the NLLB models 除了 XSTS,我们还获得了 MOS 评估,以了解目标语音中的其他音频质量方面。 . 26 6 v1 的 Robustness 我们建立了一个可重复的噪音强度评估基于FLEURS(噪音FLEURS)的基准,涵盖102种语言,两个语音任务(S2TT和ASR),以及各种噪音类型(自然噪音和音乐)。 在“噪音”和“音乐”类别中,我们将它们与原来的FLEURS语音音相混合,在不同的信号与噪音比率(SNR): 10, 5, 0, −5, −10, −15 和 −20。我们以 BLEU-SNR 曲线(对于 S2TT)或 WER-SNR 曲线(对于 ASR)来比较模型,这说明了语音输入的噪音水平增加(即 SNR 下降)时模型性能下降的程度。 76 We followed ref. to evaluate model robustness against speaker variations by calculating the average by-group mean score and by-group coefficient of variation using an utterance-level quality metric. Instead of using BLEU as the quality metric, we used chrF, which has better stability at the utterance level. The calculation of both robustness metrics does not require explicit speaker subgroup labels. We grouped evaluation samples and corresponding utterance-level chrF scores by content (transcript) and then calculated the average by-group mean score chrFMS and average by-group coefficient of variation CoefVarMS, defined as follows: 47 哪里 两个指标是互补的:chrFMS提供了一个标准化的质量指标,与传统的体级指标不同,它考虑了扬声器的变异,而CoefVarMS提供了一个标准化的质量变异指标在扬声器的变异中。 G 负责的人 毒性检测 受ASR-BLEU的启发,这项工作建议使用ASR-ETOX作为一种新的指标来检测语音中增加的毒性,并评估SEAMLESSM4T的S2ST能力增加的毒性。 ), then the toxicity detection module, ETOX ,使用毒性200单词列表 对于 S2TT,翻译的输出可以直接用 ETOX 进行评估。在两种情况下(S2ST 和 S2TT),我们在发音或句子级别测量了增加的毒性。我们首先计算了评估数据集中的每个输入和相应的输出的毒性检测。然后,我们比较了它们,并将一个案例计算为仅当输出值超过输入时添加的毒性。 , MuTox 同一数量的语言由零拍属性。然而,考虑到验证的质量,我们只报告 MuTox 那些已被基准的语言 在这两种情况下(S2ST和S2TT),我们又测量了发音或句子级别的附加毒性,在这种情况下,如果输出中的MuTox 得分为 >0.9 和输入中的 <0.5 时,一个句子包含附加毒性。 2 27 6 29 29 For toxicity mitigation, we implement two techniques for the mitigation of added toxicity. Before training, we filter out training pairs with imbalanced toxicity. Furthermore, we use Mintox at inference time. In particular, the main workflow generates a translation hypothesis with an unconstrained search. Then, the toxicity classifier is run on this hypothesis. If no toxicity is detected, we provide the translation hypothesis as it is. However, if toxicity is detected in the output, we run the classifier on the input. If the toxicity is unbalanced (that is, no toxicity is detected in the input), we re-run the translation with mitigation, which is the BEAMFILTERING step. This BEAMFILTERING consists of taking as input the multi-token expressions that should not appear in the output and excluding them from the beam-search hypotheses. Note that we do not apply mitigation in cases in which there is toxicity in the input (in other words, we do not deal with cases in which there is toxicity in the input but more toxicity in the output). 11 We used two datasets to analyse added toxicity. First, we deployed FLEURS to better align with our human evaluation effort and other evaluative components of this work. Furthermore, we used the English-only HOLISTICBIAS framework , which has been shown to trigger true added toxicity in previous studies 在本文中,我们将HOLISTICBIAS扩展到使用MMS的默认英语TTS模型。 . 77 27 23 Speech extension of MULTILINGUAL HOLISTICBIAS 为了比较各个模式(S2ST和S2TT)的表现,我们将多语种HOLISTICBIAS扩展到语音 (一) 我們使用此生成的 TTS 資料作為 S2TT 和 S2ST 的入口,並作為 S2ST 的引用。我們以兩個方向進行了翻譯:eng–X 和 X–eng. 特別是,在 X–eng 中,我們翻譯了男性和女性的語言版本。 值得注意的是,在 SEAMLESSM4T S2ST 模型中,有些目標語言並非可用,所以我們只對 17 種語言進行了翻譯,以便在 eng–X 方向進行 S2ST 任務。 = 25). For reference, the complete language list used in our experiments can be found in Supplementary Table . 23 https://github.com/facebookresearch/fairseq/tree/main/examples/mms#tts-1 n 26 对于S2ST,我们使用ASRchrf(转录由WHISPER-LARGE和WHISPER-MEDIUM完成) for eng–X and X–eng, respectively, and chrF has been calculated the same way as S2TT except that in S2ST, the text from both prediction and reference were normalized) and BLASER 2.0. It is worth noting that when evaluating on BLASER 2.0, we only included 14 languages (arb, cat, deu, eng, fra, nld, por, ron, rus, spa, swe, tha, ukr and urd) for the eng–X direction (overlapping languages from the generated TTS data and the languages available in our S2ST model). 20 数据可用性 为了使我们的工作可供社区使用,我们在GitHub上提供以下数据集、模型和代码的开源( ):(1) SEAMLESSM4T 模型,包括 SEAMLESSM4T-LARGE (2.3B 参数) 和 SEAMLESSM4T-MEDIUM (1.2B 参数) 的模型重量,以及其推断代码和精细调节配方,由我们新的模型工具包 FAIRSEQ2 ( ); (2)创建对齐语音数据的工具,包括元数据,以重塑SEAMLESSALIGN基于STOPES的无过滤470,000小时管道( )创建类似SEAMLESSALIGN和SONAR的语音编码器在37种语言和文本编码器在200种语言的对齐( )和(3)一个无文本的S2ST自动评估模型,BLASER 2.0,包括模型重量和推断脚本。 https://github.com/facebookresearch/seamless_communication https://github.com/facebookresearch/fairseq2 https://github.com/facebookresearch/stopes https://github.com/facebookresearch/SONAR 参考 Lavie, A. et al. Janus-III: speech-to-speech translation in multiple languages. In Vol. 1, 99–102 (IEEE, 1997). Proc. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing Wahlster, W. (ed.) (Springer, 2000). Verbmobil: Foundations of Speech-to-Speech Translation Nakamura, S. et al. The atr multilingual speech-to-speech translation system. , 365–376 (2006). IEEE Trans. Audio Speech Lang. Process. 14 Article Google Scholar Jia, Y. et al. Direct speech-to-speech translation with a sequence-to-sequence model. In , 1123–1127 (ISCA, 2019). Proc. Interspeech 2019 Lee, A. et al. Direct speech-to-speech translation with discrete units. In (eds Muresan, S. et al.) Vol. 1, 3327–3339 (Association for Computational Linguistics, 2022). Proc. 60th Annual Meeting of the Association for Computational Linguistics NLLB Team Scaling neural machine translation to 200 languages. , 841–846 (2024). Nature 630 Article CAS Google Scholar Agarwal, M. et al. Findings of the IWSLT 2023 evaluation campaign. In (eds Salesky, E. et al.) 1–61 (Association for Computational Linguistics, 2023). Proc. 20th International Conference on Spoken Language Translation (IWSLT 2023) Duquenne, P.-A., Schwenk, H. & Sagot, B. SONAR: sentence-level multimodal and language-agnostic representations. Preprint at (2023). https://arxiv.org/abs/2308.11466 Popović, M. chrF: character n-gram F-score for automatic MT evaluation. In (eds Bojar, O. et al.) 392–395 (Association for Computational Linguistics, 2015). Proc. Tenth Workshop on Statistical Machine Translation Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In (eds Isabelle, P. et al.) 311–318 (Association for Computational Linguistics, 2002). Proc. 40th Annual Meeting of the Association for Computational Linguistics Costa-jussà, M. R., Dale, D., Elbayad, M. & Yu, B. Added toxicity mitigation at inference time for multimodal and massively multilingual translation. In (eds Scarton, C. et al.) Vol. 1, 360–372 (European Association for Machine Translation, 2024). Proc. 25th Annual Conference of the European Association for Machine Translation ACTFL. World-readiness standards for learning languages. (2023). https://www.actfl.org/educator-resources/world-readiness-standards-for-learning-languages Desplanques, B., Thienpondt, J. & Demuynck, K. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In (eds Meng, H. et al.) 3830–3834 (ISCA, 2020). Proc. Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event Valk, J. & Alumäe, T. VOXLINGUA107: a dataset for spoken language recognition. In 652–658 (IEEE, 2021). Proc. 2021 IEEE Spoken Language Technology Workshop, Chen, M., Heffernan, K., Çelebi, O., Mourachko, A. & Schwenk, H. xSIM++: An improved proxy to bitext mining performance for low-resource languages. In (eds Rogers, A. et al.) Vol. 2, 101–109 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Heffernan, K., Çelebi, O. & Schwenk, H. Bitext mining using distilled sentence representations for low-resource languages. In , 2101–2112 (eds Goldberg, Y. et al.) (Association for Computational Linguistics, 2022). Findings of the Association for Computational Linguistics: EMNLP 2022 Feng, F., Yang, Y., Cer, D., Arivazhagan, N. & Wang, W. Language-agnostic BERT sentence embedding. In (eds Muresan, S. et al.) Vol. 1, 878–891 (Association for Computational Linguistics, 2022). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Conneau, A. et al. FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech. In , 798–805 (IEEE, 2023). 2022 IEEE Spoken Language Technology Workshop (SLT) Zhang, Z.-H. et al. Speak foreign languages with your own voice: cross-lingual neural codec language modeling. Preprint at (2023). https://arxiv.org/abs/2303.03926 Radford, A. et al. Robust speech recognition via large-scale weak supervision. In , article no. 1182, 28492–28518 (ACM, 2023). Proceedings of the 40th International Conference on Machine Learning Rubenstein, P. K. et al. AudioPaLM: a large language model that can speak and listen. Preprint at (2023). https://arxiv.org/abs/2306.12925 Casanova, E. et al. YourTTS: towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In , Vol. 162, 2709–2720 (PMLR, 2022). Proc. 39th International Conference on Machine Learning Pratap, V. et al. Scaling speech technology to 1,000+ languages. , 1–52 (2024). J. Mach. Learn. Res. 25 Dale, D. & Costa-jussà, M. R. Blaser 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation. In (eds Al-Onaizan, Y. et al.) 16075–16085 (Association for Computational Linguistics, 2024). Findings of the Association for Computational Linguistics: EMNLP 2024 Chen, P.-J. et al. Speech-to-speech translation for a real-world unwritten language. In (eds Rogers, A. et al.) 4969–4983 (Association for Computational Linguistics, 2023). Findings of the Association for Computational Linguistics: ACL 2023 Licht, D. et al. Consistent human evaluation of machine translation across language pairs. In (eds Duh, K. & Guzmán, F.) Vol. 1, 309–321 (Association for Machine Translation in the Americas, 2022). Proc. 15th Biennial Conference of the Association for Machine Translation in the Americas Costa-jussà, M. et al. Toxicity in multilingual machine translation at scale. In , 9570–9586 (Association for Computational Linguistics, 2023). Findings of the Association for Computational Linguistics: EMNLP 2023 Ghosh, S., Lepcha, S., Sakshi, S., Shah, R. R. & Umesh, S. DeToxy: a large-scale multimodal dataset for toxicity classification in spoken utterances. In 2022, 5185–5189 (2022); . Proc. Interspeech https://api.semanticscholar.org/CorpusID:247940269 Costa-jussà, M. R. et al. MuTox: universal multilingual audio-based toxicity dataset and zero-shot detector. In (eds Ku, L.-W. et al.) 5725–5734 (Association for Computational Linguistics, 2024). Findings of the Association for Computational Linguistics: ACL 2024 Stanovsky, G., Smith, N. A. & Zettlemoyer, L. Evaluating gender bias in machine translation. In (eds Korhonen, A. et al.) 1679–1684 (Association for Computational Linguistics, 2019). Proc. 57th Annual Meeting of the Association for Computational Linguistics Levy, S., Lazar, K. & Stanovsky, G. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In (eds Moens, M. F. et al.) 2470–2480 (Association for Computational Linguistics, 2021). Findings of the Association for Computational Linguistics: EMNLP 2021 Costa-jussà, M. R. et al. Interpreting gender bias in neural machine translation: Multilingual architecture matters. , 11855–11863 (2022). Proc. AAAI Conf. Artif. Intell. 36 Google Scholar Costa-jussà, M. R., Basta, C. & Gállego, G. I. Evaluating gender bias in speech translation. In (eds Calzolari, N. et al.) 2141–2147 (European Language Resources Association, 2022). Proceedings of the Thirteenth Language Resources and Evaluation Conference Bentivogli, L. et al. Gender in danger? Evaluating speech translation technology on the MuST-SHE corpus. In (eds Jurafsky, D. et al.) 6923–6933 (Association for Computational Linguistics, 2020). Proc. 58th Annual Meeting of the Association for Computational Linguistics Costa-jussà, M. et al. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In , 14141–14156 (Association for Computational Linguistics, 2023). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Zuckerman, E. The polyglot internet. (2008). https://ethanzuckerman.com/the-polyglot-internet/ Belekar, A., Sunka, S., Bhawar, N. & Bagade, S. Voice based e-mail for the visually impaired. , 8–12 (2020). Int. J. Comput. Appl. 175 Google Scholar Wang, S., Cooper, N. & Eby, M. From human-centered to social-centered artificial intelligence: Assessing ChatGPT’s impact through disruptive events. (2024). Big Data Soc. https://doi.org/10.1177/20539517241290220 Koenecke, A. et al. Racial disparities in automated speech recognition. , 7684–7689 (2020). Proc. Natl Acad. Sci. USA 117 Article ADS CAS PubMed PubMed Central Google Scholar Ngueajio, M. K. & Washington, G. Hey ASR system! Why aren’t you more inclusive? Automatic speech recognition systems’ bias and proposed bias mitigation techniques. A literature review. In (eds Chen, J. Y. C. et al.) Vol. 13518, 421–440 (Springer, 2022). Proc. International Conference on Human-Computer Interaction Elbow, P. The shifting relationships between speech and writing. , 283–303 (1985). Coll. Compos. Commun. 36 Article Google Scholar Trilla, A. & Alias, F. Sentence-based sentiment analysis for expressive text-to-speech. , 223–233 (2013). IEEE Trans. Audio Speech Lang. Process. 21 Article Google Scholar Iranzo-Sánchez, J., Civera, J. & Juan, A. From simultaneous to streaming machine translation by leveraging streaming history. In , 6972–6985 (Association for Computational Linguistics, 2022). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Rybakov, O. et al. Streaming Parrotron for on-device speech-to-speech conversion. Preprint at (2022). https://arxiv.org/abs/2210.13761 Koehn, P. Europarl: A parallel corpus for statistical machine translation. In , 79–86 (2005). Proceedings of Machine Translation Summit X: Papers Ziemski, M., Junczys-Dowmunt, M. & Pouliquen, B. The United Nations Parallel Corpus v1.0. In (eds Calzolari, N. et al.) 3530–3534 (European Language Resources Association, 2016). Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16) Wang, C., Pino, J., Wu, A. & Gu, J. CoVoST: a diverse multilingual speech-to-text translation corpus. In (eds Calzolari, N. et al.) 4197–4203 (European Language Resources Association, 2020). Proceedings of the Twelfth Language Resources and Evaluation Conference Wang, C., Wu, A., Gu, J. & Pino, J. CoVoST 2 and Massively Multilingual Speech Translation. In , 2247–2251 (ISCA, 2021). Proc. Interspeech 2021 Salesky, E. et al. The multilingual TEDx corpus for speech recognition and translation. In , 3655–3659 (ISCA, 2021). Proc. Interspeech 2021 Zhang, Y. et al. Google USM: scaling automatic speech recognition beyond 100 languages. Preprint at (2023). https://arxiv.org/abs/2303.01037 Schwenk, H. Filtering and mining parallel data in a joint multilingual space. In (eds Gurevych, I. & Miyao, Y.) Vol. 2, 228–234 (Association for Computational Linguistics, 2018). Proc. 56th Annual Meeting of the Association for Computational Linguistics Artetxe, M. & Schwenk, H. Margin-based parallel corpus mining with multilingual sentence embeddings. In (eds Korhonen, A. et al.) 3197–3203 (Association for Computational Linguistics, 2019). Proc. 57th Annual Meeting of the Association for Computational Linguistics Artetxe, M. & Schwenk, H. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. , 597–610 (2019). Trans. Assoc. Comput. Linguist. 7 Article Google Scholar Duquenne, P.-A., Gong, H. & Schwenk, H. Multimodal and multilingual embeddings for large-scale speech mining. In (eds Ranzato, M. et al.) Vol. 34, 15748–15761 (Curran Associates, 2021). Advances in Neural Information Processing Systems Duquenne, P.-A. et al. SpeechMatrix: a large-scale mined corpus of multilingual speech-to-speech translations. In (eds Rogers, A. et al.) Vol. 1, 16251–16269 (2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Silero. Silero VAD: pre-trained enterprise-grade voice activity detector (VAD), number detector and language classifier. (2021). GitHub https://github.com/snakers4/silero-vad Khurana, S., Laurent, A. & Glass, J. SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation. , 1493–1504 (2022). IEEE J. Sel. Top. Signal Process. 16 Douze, M. et al. The Faiss library. Preprint at (2024). https://arxiv.org/abs/2401.08281 Inaguma, H. et al. UnitY: two-pass direct speech-to-speech translation with discrete units. In (eds Rogers, A. et al.) Vol. 1, 15655–15680 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Tjandra, A., Sakti, S. & Nakamura, S. Speech-to-speech translation between untranscribed unknown languages. In , 593–600 (IEEE, 2019). 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Babu, A. et al. XLS-R: self-supervised Cross-lingual Speech Representation Learning at Scale. In , 2278–2282 (ISCA, 2022). Proc. Interspeech 2022 Gong, H. et al. Multilingual speech-to-speech translation into multiple target languages. Preprint at (2023). https://arxiv.org/abs/2307.08655 Chung, Y.-A. et al. w2v-BERT: combining contrastive learning and masked language modeling for self-supervised speech pre-training. In , 244–250 (IEEE, 2021). 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Chiu, C.-C., Qin, J., Zhang, Y., Yu, J. & Wu, Y. Self-supervised learning with random-projection quantizer for speech recognition. In , 3915–3924 (PMLR, 2022). Proc. 39th International Conference on Machine Learning Gulati, A. et al. Conformer: convolution-augmented transformer for speech recognition. In , 5036–5040 (ISCA, 2020). Proc. Interspeech 2020 Kudo, T. & Richardson, J. SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. In (eds Blanco, E. & Lu, W.) 66–71 (Association for Computational Linguistics, 2018). Proc. 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations Sennrich, R., Haddow, B. & Birch, A. Neural machine translation of rare words with subword units. In (eds Erk, K. & Smith, N. A.) Vol. 1, 1715–1725 (Association for Computational Linguistics, 2016). Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Andrews, P. et al. stopes - modular machine translation pipelines. In (eds Che, W. & Shutova, E.) 258–265 (Association for Computational Linguistics, 2022). Proc. 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations Jia, Y. et al. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In , 7180–7184 (IEEE, 2019). 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Pino, J., Xu, Q., Ma, X., Dousti, M. J. & Tang, Y. Self-training for end-to-end speech translation. In , 1476–1480 (2020). Proc. Interspeech 2020 Vaswani, A. et al. Attention is all you need. In (eds Guyon, I. et al.) Vol. 30 (Curran Associates, 2017). Advances in Neural Information Processing Systems Zhao, J., Yang, H., Haffari, G. & Shareghi, E. M-Adapter: modality adaptation for end-to-end speech-to-text translation. In , 111–115 (ISCA, 2022). Proc. Interspeech 2022 Ren, Y. et al. FastSpeech 2: fast and high-quality end-to-end text to speech. In (2021). Proc. 2021 International Conference on Learning Representations Shih, K. J. et al. RAD-TTS: parallel flow-based tts with robust alignment learning and diverse synthesis. In (2021). ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models Chen, M. et al. BLASER: A text-free speech-to-speech translation evaluation metric. In (eds Rogers, A. et al.) Vol. 1, 9064–9079 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Snyder, D., Chen, G. & Povey, D. MUSAN: a music, speech, and noise corpus. Preprint at (2015). https://arxiv.org/abs/1510.08484 Smith, E. M., Hall, M., Kambadur, M., Presani, E. & Williams, A. “I’m sorry to hear that”: finding new biases in language models with a holistic descriptor dataset. In (eds Goldberg, Y. et al.) 9180–9211 (Association for Computational Linguistics, 2022). Proc. 2022 Conference on Empirical Methods in Natural Language Processing 认可 我们感谢S. Edunov和A. Fan帮助塑造该项目的早期阶段;S. Bhosale,V. Goswami,F. Hernandez和Y. Zhang为他们在构建更强的音频模型进行基准支持;M. Seejoor和M. Duppenthaler为他们对BLASER 1.0的贡献;K. Klyushkin为他帮助构建更好的体验;A. Moghevnikov为他对FAIRSEQ2和SONAR推断的贡献;A. Ni和X. Zhang为他们在构建更强的音频模型的帮助;N. Seejoor和M. Duppenthaler为他们在建立演示的帮助;V. Goswami,S. Huhan,B. Acun-Uyan和C.-J. Wu为他帮助提供新的效率优化建议;B. Al 本文在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下可用。 本文在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下可用。