这个AI可以听到,翻译和回应100种语言

作者 : 洛伊克·巴拉尔特 朱元 Mariano Coria Meglioli 戴维·戴尔 唐宁 保罗·安布罗伊斯·杜克内 哈迪·艾尔萨哈 香港高 Kevin Heffernan 约翰·霍夫曼 Christopher Klaiber 金正恩李 Daniel Licht 吉恩·梅拉德 Alice Rakotoarison Kaushik Ram Sadagopan Guillaume Wenzek Ethan Ye 作者 : 洛伊克·巴拉尔特 朱元 马里亚诺·科里亚·梅格利奥利 戴维·戴尔 唐宁 保罗·安布罗伊斯·杜克内 哈迪·艾尔萨哈 香港高 凯文·赫弗纳 约翰·霍夫曼 克里斯托弗·克莱伯 金正恩李 丹尼尔 光 吉恩·梅拉德 爱丽丝·拉科托里森 卡舒希克·拉姆·萨达戈潘 威廉·温兹克 伊丹·耶 抽象 创建一个帮助个人在任何两种语言之间翻译语言的工具Babel Fish需要先进的技术创新和语言专业知识,尽管传统的语音对话翻译系统由多个子系统组成,以随机的方式进行翻译。 , , ,可扩展和高性能的统一系统 , 为了解决这一差距,我们介绍了SEAMLESSM4T – 大量多语言和多式机器翻译 – 支持语音到语音翻译(101到36种语言),语音到文本翻译(从101到96种语言),文本到语音翻译(从96到36种语言),文本到文本翻译(96种语言)和公共语音识别(96种语言)的单一模型。使用一种新的多式语音到文本翻译和其他公开可用的数据,SEAMLESSM4T是第一个能够翻译语音到语音(96到36种语言)的多语言系统之一,可以翻译语音和文本(96种语言)的任务。此外,它优于现有的最先进的语音识别系统(96种语言)。 1 2 3 4 5 主要 巴比尔鱼从 这是一个虚构的工具,可以翻译两种语言之间。在当代的全球环境中,其特点是互联性和移动社交性日益增加,在数字世界和物理世界中实现这些技术的现代化和按需语音对话翻译(S2ST)的社会迫切需要从未如此之大。尽管日常通信中语音的中心性,今天的机器翻译(MT)系统仍然以文本为导向。 了解更多关于为什么在MT中优先考虑演讲的细节,尽管单一的模型,如没有语言落后(NLLB) 通过将文本翻译(T2TT)覆盖范围推向200多种语言,统一的S2ST模型远未达到类似的范围或性能,这种差异可以归因于许多原因,但音频数据稀缺和建模限制仍然是关键障碍。 《Hitchhiker’s Guide to the Galaxy》 I1 的 6 现有的 S2ST 系统有三个主要缺点:第一,这些系统倾向于专注于高资源语言,留下许多低资源语言。第二,这些系统主要服务于从源语言翻译到英语(X-eng),而不是反向翻译(eng-X)。 , , ,他们不匹配他们的反弹对手的表现 参见附加信息部分 更多关于当前技术景观的细节。 1 4 5 7 二、 为了解决这些局限性,我们引入了SEAMLESSM4T(大规模多语言和多式机器翻译),一个支持ASR、T2TT、语音到文本翻译(S2TT)、文本到语音翻译(T2ST)和S2ST的统一系统。 然后,我们将这个集合物的过滤子集与人标签和假标签数据相结合,开发了第一个从100多种语言执行S2ST到36种语言的多任务系统,S2TT和ASR到96种语言,零射 T2ST到36种语言,以及T2TT到96种语言(参见表)。 关于语言覆盖和补充信息部分的比较概述 由于SEAMLESSM4T的统一架构(图。 ),该模型可以以零射程的方式执行非英语方向的T2TT,S2TT或S2ST(X-X)。它还可以执行T2ST而没有被明确训练为此任务。 ”),它可以处理混合两种或多种语言的发音。 8 1 二 1 未经监督的演讲预练 是的 UNITY2(S2ST精细调节)的三个主要块与其非自我退缩(NAR)T2U在左上方显示。多任务-UNITY2及其附加文本编码器在左下方显示。 为了评估我们模型的输出质量,我们使用了多个现有指标,涵盖任务和模式,以及四个主要的评估数据集。 对于 T2TT,蓝色 对于 S2TT,ASR-BLEU 对于 S2ST 和 WER 对于 ASR. 参见补充表 我们还测试了我们的模型对背景噪音或扬声器变异的弹性,以及其他负责任部署的战线,如性别偏见,使用MULTILINGUAL HOLISTICBIAS数据集,或增加毒性,使用新的基于语音的指标(ASR-ETOX和MuTox)。 . 9 10 5 2 11 除了建造SEAMLESSM4T之外,我们还讨论了我们工作的社会影响以及它如何有助于提高世界准备水平。 从长远来看(参见“ 为了推动未来的研究,我们将数据工具、代码和两种尺寸的SEAMLESSM4T模型提供给公众非商业用途。 12 社会影响与结论 在接下来的部分中,我们追踪了开发我们的SEAMLESSM4T模型的关键结果。 首先,我们概述了我们挖掘对齐语音和文本数据的努力,从语音识别开始,到使用模型无形编码器挖掘对齐语音和文本细分。 接下来,我们报告了我们直接翻译系统的主要结果,部分由上述自动对齐语音数据训练。 这些结果凸显了SEAMLESSM4T模型在ASR、T2TT、S2TT和S2ST中实现多语言最先进的性能的任务多功能性。 数据 培训语音翻译系统需要标签数据;即语音对文本和语音对语音对准数据.然而,对于低资源语言,这些资源非常有限。 以及大量的原始言论和文本,如在 ,自动挖掘对齐资源,补充现有人标签和伪标签数据。 8 方法 语言识别 从网络中处理原始语音包括将发言分成更短的片段,然后是语言识别,建立在使用ECAPA-TDNN架构的VoxLingua107训练的开源模型上 , ,我们开发了一种新的基于语音的语言识别(LID)模型,涵盖本文所介绍的所有100种语言(参见 ,” ” 更多细节)。 13 14 方法 音频处理和语音 LID 为了衡量LID模型的精度和召回,我们在扩展数据表中报告测试数据上的F1分数 数据为100个SEAMLESSM4T语言(总体)和79个SEAMLESSM4T和VoxLingua107(中间部分)之间的通用语言。请注意,VL107HF的所有语言中的宏观F1是低的,因为21种语言不受该模型所涵盖。我们发现,对额外的语言的培训略有降低了共同语言的总体表现,这是较多的近语言的直接结果。例如,Zulu(zul)经常与Nyanja(nya),Ibo(ibo)与Yoruba(yor)混淆,以及现代标准阿拉伯语(arb)与摩洛哥阿拉伯语(ary)和埃及阿拉伯语(arz模型)的培训略有降低。我们提高分类准确性(F1差异大于5%)在17种语言中,平均增长 1 SONAR 文本嵌入空间 为了从语言识别的细分中提取自动对齐的翻译数据,我们依赖于语言和模式不符的编码器。 目前,我们为 200 种语言提供单个文本编码器和解码器,并为 37 种语言提供语音编码器。 在多语言相似性搜索中,xsim 和 xsim++ 是两个已知的代理指标,用于评估用于采矿目的的多语言嵌入空间。 ,SONAR显著超过其他流行的方法,如LASER3 或拉布斯 用较低的 xsim 和 xsim++。 8 6 15 2 16 17 此外,我们在 T2TT 任务中评估了 SONAR 文本编码器和解码器. 超过 200 种语言的平均性能与中型 NLLB 密集模型相比具有竞争力,尽管 SONAR 将编码器和解码器的注意力替换为固定大小的瓶颈嵌入(参见扩展数据表的 T2TT 列) 这个结果表明,使用注意力不是合理的翻译准确性的要求。 2 演讲编码器培训 语音编码器只在语音转录上接受了教师-学生方法的培训(参见 ,” ’) 评估每个语音编码器在一个端到端循环中的迭代,即采矿和培训S2TT或S2ST翻译系统,将是计算密集的。相反,我们将语音编码器与SONAR文本编码器连接,并评估了这个零打 S2TT系统作为编码器质量的代理。 ,SONAR语音编码器与FLORES上的WHISPER-LARGE-V2这样的模型比较有利 和花 在一些高资源语言(如德语,俄语或葡萄牙语)中,可观察到精度差距,但SONAR在一些低资源语言(如斯瓦希里语或孟加拉语)中优于WHISPER-LARGE-V2(参见补充表) )。 方法 索尼尔 2 6 18 8 西雅图 SONAR 文本和语音编码器用于采集三种类型的对齐数据:(1)英语语音与非英语文本(Sen2Txx);(2)非英语语音与英语文本(Sxx2Ten);以及(3)非英语语音与英语语音(Sxx2Sen)。SEAMLESSALIGN 在 Sen2Txx中提供 202,796 小时的音频,在 Sxx2Ten 中提供 239,767 小时的音频,在 Sxx2Ten 中提供 29,161 小时的音频。 . 8 对于文本域,我们使用了NLLB项目合并的相同数据 这些数字从马耳他语或斯瓦希里语等低资源语言的3300万条或5500万条短语到22000万条英语短语。 6 除了马耳他语,我们只获得了少量的原始音频,我们能够为所有语言挖掘超过100小时的英语语音对比。 与英语文本的对比达到了大多数语言的1000小时,高资源语言超过10,000小时。 添加如此大量的数据来训练多语言翻译系统是一个巨大的计算挑战。 ,” 由于我们的挖掘数据可以帮助支持许多不同的使用案例,我们开源了指导其复制所需的元数据(高达 1.15 的 SONAR 门槛;参见 ,” ’)允许社区重建SEAMLESSALIGN并将其用于自己的目的,因此可以根据任务调整最佳门槛,平衡数据集大小和对齐质量,我们的采矿代码也在STOPES库中开源。 方法 模型 方法 发言人 多任务翻译系统模型 结合在模型中描述的模型技术 ,” ’,与SEAMLESSALIGN的其他数据(见 ,” ’),我们训练了两种尺寸的SEAMLESSM4T模型:具有 2.3B参数的大型和具有 1.2B参数的中型。SEAMLESSM4T-MEDIUM旨在成为一个易于访问的测试床,无论是精细调节,改进或参与分析。 ,” ’)和更强大的单元解码器(参见“ 所有 SEAMLESSM4T 模型都支持文本模式中的 96 种源语言,语音模式中的 100 多种源语言。在目标方面,模型可以以文本形式输出 96 种语言,语音形式输出 35 种语言。 和 这表明,对于一些翻译方向和缺乏监督数据,我们的模型将被评估为零射击。 方法 模型 方法 数据 方法 未经监督的演讲预练 S2ST 精密调节 12 13 我们对所有四个受监督的任务(T2TT,ASR,S2TT和S2ST)以及文本对话翻译的零拍任务(T2ST,也称为跨语言文本对话合成)进行了评估。 ) 为了生成文本假设,我们用雷射搜索解码。我们用 chrF2++ 给 T2TT 和 BLEU 给 S2TT。 . For ASR, we scored with WER (word error rate) on normalized transcriptions and references following ref. . 19 2 20 在 S2ST 和 T2ST 推断中,我们执行了双通路射线搜索解码;从第一通路解码中获取的最佳假设与文本解码器嵌入,并发送到一个文本到单元模块(T2U;参见“部分”)。 ’)寻找最佳单元序列假设. 我们为两次搜索使用了5的光束宽度. 我们使用Whisper模型评估了ASR-BLEU的S2ST和T2ST准确性.我们将Whisper的解码温度设置为零,并使用贪婪的解码来确保ASR模型的确定性行为。 在计算蓝色分数之前。 S2ST 精密调节 20 与语音翻译的散步方法的比较 对于 SEAMLESSM4T 和 WHISPER 支持的语言集,我们在表中比较 (S2TT列) 我们的直接 S2TT 模型的性能,即 WHISPER ASR 模型和 NLLB T2TT 模型的组合。 SEAMLESSM4T-V2 以 4.6 BLEU 点(从 22.0 到 26.6)在 X 方向和 1 BLEU 点(从 21.1 到 22.2 方向)在 X-eng 方向中超过了 3B 参数。 这些模型超过了4B参数,大多超过了SEAMLESSM4T-V2在X-eng (+3.9);他们在eng-X方向只有微小地超过SEAMLESSM4T-V2的0.2 BLEU点。 3 3 与之前直接S2TT SOTA模型相比,这些模型落后于形系统(例如,AUDIOPALM-2-8B-AST。 ), SEAMLESSM4T-V2 提高了 FLEURS X-eng S2TT BLEU 得分 6.9 点(从 19.7 到 26.6;即 35% 的提高)。 21 桌子 (S2ST列)还将S2ST与SEAMLESSM4T模型进行比较.对于S2ST,我们探讨了两种可变的选项:(1)与ASR、T2TT 和 TTS 进行三阶段,(2)与S2TT 和 TTS 进行两阶段。 当英语语言和MMS的合成时 我们的SEAMLESSM4T-LARGE在FLEURS X-eng方向上的两阶段滑动模型比作8个ASR-BLEU点(对比为SEAMLESSM4T的支持和MMS的TTS系统的支持)。我们的SEAMLESSM4T-LARGE在FLEURS X-eng方向上的两阶段滑动模型比作8个ASR-BLEU点(对比为S2 FSTLE Xeng的支持为17.8-25.8).它还比S2 FSTLE Xeng更强的三阶段滑动模型(WHISPER-LARGE-V2 + NLLB-3.3B + YOURTTS)更强劲,以21个ASR-BLEU点(23.7-25.8).改进的SEAMLESSM4T-V2进一步增强了S2 FSTLE 3 22 23 我们分享在补充信息部分 对 S2TT 和 S2ST 任务的评估结果与额外的指标,包括我们的模块性不确定性 BLASER 2.0。 四一 多任务结果 我们在桌子上报告 对于ASR和零射击T2ST任务的FLEURS基准(X-eng和eng-X),以及T2TT的相关FLORES基准(X-eng和eng-X)的结果;在ASR中,SEAMLESSM4T-LARGE优于WISPER-LARGE-V2 在重叠的77种支持的语言中,WER减少了46%(从41,7到22,6),而SEAMLESSM4T-V2比WHISPER-LARGE-V2提高了56%(从41,7到18,5)。 against MMS SEAMLESSM4T-V2 比 CTC 评估的 MMS 变体高出 38% 的 WER (从 31.0 到 19.1),但它被利用单语言 n-gram 语言模型的变体所超越(比 18.6 更好的 WER 更好 5%)。 4 20 9 23 在 T2TT 支持任务中,表中的结果 顯示我們的 SEAMLESSM4T 模型與 NLLB-3.3B (ref. )在X-eng和eng-X方向。 4 6 接下来,我们以零拍的方式评估了SEAMLESSM4T模型,以T2ST的任务。鉴于FLEURS为每个样本收集了三个不同的母语音器的三个记录,我们随机选择了一个用于T2ST的任务(输入是文本)。 (T2ST 列)是 SEAMLESSM4T 模型与 NLLB 和 YOURTTS (英语 TTS) 或 MMS (非英语 TTS) 之间进行的比较,用于合成翻译文本。我们在 88 个 X-eng 方向上平均了 ASR-BLEU 得分(FLEURS 和 SEAMLESSM4T 支持的语言的重叠)。我们还平均了 ASR-BLEU 在 26 个 eng-X 方向(SEAMLESSM4T 与 TTS 模型的重叠)。 与 cascading 模型相比,SEAMLESSM4T-LARGE V2 的零射击能力与 NLLB-3.3B + YOURTTS 在 X - 中相同,并且超过了 NLLB-3.3B + MMS 的 +3.9 4 为了进一步了解FLEURS S2TT X-eng方向的改进来自何方,我们根据资源级别对语言进行了排列(参见补充表中的语言的确切列表) )并在表中报告每个资源级别的平均BLEU分数 结果表明,SEAMLESSM4T 显著提高了低资源语言的翻译质量,提高了 +10.2 BLEU(从 18.0 到 28.2,即比 AUDIOPALM-2-8B-AST 提高了 57%)。 12 5 自动和人为评估 语音翻译中的语义准确性通常用自动计量BLEU进行评估。 对于 S2TT 或其扩展 ASR-BLEU 对于 S2ST. 此外,我们使用 BLASER 2.0 (ref. )的Blaser的扩展 ,现在允许对语音和文本进行模式性评估和质量评估。 10 24 25 为了补充自动指标的实用性,我们还依赖于对我们的模型进行广泛的人类评估,以下是使用XSTS(跨语言语义文本相似性)协议为S2TT和S2ST任务提供人类评估。 MOS(平均意见分数)协议用于语音输出(见 ,” 然而,由于资源限制,我们涵盖了有限的模型(SEAMLESSM4T-LARGE、SEAMLESSM4T-LARGE V2 和由 WHISPER-LARGE-V2 用于 ASR、NLLB 3.3B 用于翻译和 YOURTTS 或 MMS 用于 S2ST 任务)和翻译指令(英语和英语的 23 种语言,MOS 的 10 种语言 X-eng)。 26 方法 人类评价 XSTS 得分表明,SEAMLESSM4T-LARGE V2 在平均语言水平的 XSTS 得分和赢利率(评估语言的比例优于 XSTS)方面,对于所有任务和语言方向都具有较高的信心。对于 S2ST 任务来说,SEAMLESSM4T-V2 的相对表现是最强的,SEAMLESSM4T-V2 方法的赢利率与 cascaded 基线和 SEAMLESSM4T-LARGE 两种方向相比为 100%,语言平均 XSTS 得分大约 0.5 点高于 cascaded 基线和 0.36-0.5 点高于 SEAMLESSM4T-LARGE for eng–X 和 X-eng,参见相应的附加表。 和 分别为完整的语言水平和总结的 XSTS 结果。 22 24 我们还使用中等意见分数协议来衡量S2ST的语音输出质量,该协议评估了(1)声音质量、(2)语音清晰度和(3)自然性。我们发现,一般来说,在所有MOS方面,SEAMLESSM4T-LARGE V2倾向于优先于SEAMLESSM4T-LARGE,该协议倾向于优先于随机模型基线,除了X-eng,为此SEAMLESSM4T-LARGE世代更受欢迎 (+ SEAMLESSM4T-LARGE和SEAMLESSM4T-LARGE V2世代之间平均差异1点),这可能是由于模型架构的差异而导致的意外结果。 和 for full language-level and summarized MOS results, respectively. 23 24 使用 S2ST 任务的 XSTS 评估,BLASER 2.0 (在给定语言方向的所有评估项目的平均值) 实现了优越的 Spearman 相关性,以校准的语言水平的 XSTS 得分在 X-eng 方向 (0.845 对于 BLASER 2.0 与 0.74 对于 ASR-BLEU) 以及特别是在 eng-X 方向 (0.81 对于 BLASER 2.0 与 0.246 对于 ASR-BLEU) 类似的结果持有 S2TT 任务(参见补充表) 得到完整的结果)。 21 最后,我们测试了我们的模型在噪音和扬声器变化方面的强度,通过基于FLEURS创建开放的强度基准(见FLEURS)。 ,” 为此,我们发现,与WHISPER-LARGE-V2相比,SEAMLESSM4T-V2平均对背景噪音和扬声器变异的抵抗力大约为42%和66%(参见附加信息部分的完整结果)。 )。 方法 可靠性 V2 的 负责的人 毒性 毒性可以定义为可能煽动对个人或团体(如宗教,种族或性别)的仇恨,暴力或虐待的邪恶或语言的例子。 在语音翻译的背景下,我们主要担心增加的毒性 - 引入在译文中不存在的有毒元素。 并且,最近,使用MuTox的数十种语言 . 27 28 29 因此,对于语音和文本的多语言毒性检测,我们使用ETOX(或ASR-ETOX为S2ST)和MuTox(为语音和文本)作为指标来检测和评估添加毒性。 时间的推移(参见) ,” ” )。 11 方法 毒性检测 我们在两个数据集中计算了增加的毒性(FLEURS和HOLISTICBIAS) ) 跨 24 个英语翻译方向(arb, ben, cat, ces, dan, deu, est, fin, fra, hin, ind, ita, nld, pes, pol, por, rus, slk, spa, swh, tgl, tur, urd, vie),使用系统覆盖的交叉点上的语言和 MuTox 进行了基准(请注意,MuTox 具有更广泛的语言覆盖范围,类似于 SONAR,但仅在 30 种语言中进行了基准) ) 请参阅 ETOX 评估的更多翻译指南的结果,参见附加信息部分 桌子 该研究表明,虽然添加毒性水平和类型因语言和数据集而异,但我们系统中的添加毒性在我们两种毒性检测指标(<0.4%)中具有相对较低的普遍性。 对于所有模式、方向和数据集,最低的毒性是与 SEAMLESSM4T-V2 + MinTox 一致获得的,在与基线相比,在不使用 MinTox 的情况下与相同模型相比,在 MuTox 方面达到 5% 的毒性减少(在 ETOX 方面达到 80%),在 MuTox 方面达到 20% 的毒性减少(在 ETOX 方面达到 90% )(参见附加信息部分的完整结果)。 )。 27 29 见1 6 6 见1 Gender bias 在MT的背景下,性别偏见可以被定义为语法性别定义中的错误,这种偏见可以明确地表现为将非性别转化为性别化形式(例如,有利于男性代表的输出)时的过度通俗化,或在仅在性别转化上不同的句子的翻译质量变化时缺乏强度。 此事的以前工作主要在文本模式中。 , , and tends to be English-centric, with few demographic axes and multilingual references. Similar efforts for the speech modality remain sparse , . 30 31 32 33 34 We used MULTILINGUAL HOLISTICBIAS 和它的语音扩展来比较S2TT和S2ST的性能。eng-X方向允许在男性或女性参照的情况下比较性能,而X-eng方向允许在我们改变性别转换时进行翻译的可靠性比较。 典型的英语 - 西班牙语语言对的例子是“我是一名家务工”和相应的翻译“Soy amo de casa”和“Soy ama de casa”在西班牙语中。 翻译从英语到西班牙语时,我们可以测量系统是否过度通俗化到一个性别,而在其他方向,我们可以评估转换到性别转换的可靠性(见) ,” ” )。 35 Methods Speech extension of MULTILINGUAL HOLISTICBIAS 我们对 S2TT 和 S2ST 的翻译偏差进行了综合评估(参见扩展数据表) for average results and Supplementary Information section for detailed results). SEAMLESSM4T-V2 consistently improves robustness in gender variations across metrics and tasks. When compared with the previous model, SEAMLESSM4T-V2 improves over SEAMLESSM4T-LARGE by 0.4% in S2TT and by 0.1% BLASER 2.0 in S2ST, and it beats the external baseline system of WHISPER-LARGE-V2 (+YOURTTS) by 0.1% in S2TT and by 0.9% BLASER 2.0 for S2ST. However, SEAMLESSM4T-V2 is not able to consistently improve in terms of gender overgeneralization compared with the previous model. SEAMLESSM4T-V2 is comparable in terms of BLASER 2.0 to SEAMLESSM4T-LARGE, but it lags far behind in terms of ASRchrf (by 2.2%), and overgeneralization is increased by 0.2% when it comes to S2TT. Although we can increase bias robustness by improving the overall quality of the model, it seems that we need specific techniques to counteract the overgeneralization of the model towards a specific gender. 3 六二 Social impact and conclusion The world we live in has never been more interconnected—the global proliferation of the internet, mobile devices, communicative platforms and social media exposes individuals to more multilingual content than ever before . The current social order places a demand on the world-readiness of a person 最初开发在语言学习的背景下,世界准备强调了能够在我们的母语以外的语言中沟通的重要性,无论是工具(即就业或学习)还是文化原因(即成为全球公民)。 36 12 The downstream applications that SEAMLESSM4T supports could allow on-demand access to world-readiness by streamlining multilingual exchange across various contexts. SEAMLESSM4T-supported applications could act as a co-piloting mechanism that supports users in multilingual conversations and boosts their confidence in speech-heavy interactions. As speech-based interfaces (for example, audio assistants, voice memos and live transcriptions) and auditory content (for example, podcasts, audiobooks and short-form videos) become ever more present, SEAMLESSM4T-enabled downstream applications could unlock a greater variety of multilingual experiences. From an inclusion standpoint, the focus of SEAMLESSM4T on multimodality could make a meaningful difference in augmenting the world-readiness of those with accessibility needs and those whose languages can be transcribed with multiple writing systems. For many who lack reading or writing skills, or cannot rely on sight (that is, people who are blind or with visual impairment), voice-assisted technologies are essential to how they communicate and stay connected . The ability to translate speech gives these groups more comprehensive access to information not only beyond their native languages but also in a manner that is better suited for their communicative needs. 37 As with most technologies, the distribution of benefits varies based on user demographics and social situation . Although we make the case that SEAMLESSM4T could augment world-readiness by lowering the barriers in cross-lingual communication, some users may experience more difficulties using our work than others. For instance, similar to many other speech technologies, the ASR performance of SEAMLESSM4T may vary based on gender, race, accent or language , . Moreover, the performance of our system in translating slang or proper nouns may also be inconsistent across high and low-resource languages. 38 39 40 S2ST的另一个挑战是,语音倾向于比书面语言更容易接收和反馈。换句话说,扬声器在确认输出质量或在现场对话中进行编辑的能力有限。没有能力通过背后的翻译或母语翻译来规划和修订,S2ST在错误翻译或毒性方面可能带来更高的交互风险。我们敦促使用SEAMLESSM4T精心调节或构建文物的研究人员和开发人员批判性地思考可以帮助用户绕过这些潜在的障碍的设计功能。重要的是,我们认为SEAMLESSM4T支持的应用程序最好被视为协助翻译的增强装置,而不是替代语言学习或可靠的人类口译员的需求的工具。 Finally, speech is not spoken text—it encompasses a suite of prosodic (for example, rhythm, stress, intonation or tone) and emotional components that deserve further research . To create S2ST systems that feel organic and natural, more research should be directed at output generation that preserves expressivity 此外,巴贝尔鱼的最终实现需要对低延迟语音翻译的研究进行更深入的投资。 , . We hope that SEAMLESSM4T opens up new possibilities for both these research areas. 41 42 43 44 方法 Data Developing an effective multilingual and multimodal translation system such as SEAMLESSM4T requires sizeable resources across languages and modalities. Some human-labelled resources for translation are freely available, but often limited to a small set of languages or in very specific domains. Well-known examples are parallel text collections such as Europarl 联合国科普斯 . A few human-created collections also involve the speech modality, such as CoVoST , mTEDx 的 . Yet no open dataset currently matches the size of those used in initiatives such as WHISPER 或 USM ,这证明释放了前所未有的性能。 45 46 47 48 49 20 50 平行数据挖掘出现作为使用封闭数据的替代方案,无论是语言覆盖范围还是体积大小。今天的主导方法是将来自不同语言和模式的句子编码到一个固定大小的嵌入空间中,并根据相似度指标找到平行实例。 , 这种方法最初采用了多语言激光空间。 然后,教师学生培训被用来将这种方法扩展到200种语言 , and, subsequently, the speech modality , . 51 52 53 6 16 54 55 Audio processing and speech LID We started with 4 million hours of raw audio originating from a publicly available repository of crawled web data on which we applied several cleaning and filtering operations. To maximize the recall of mining, it is important that all segments have a similar granularity. For the text domain, a sentence is generally well-defined. This is less obvious for raw speech because pauses are not necessarily used at sentence boundaries. First, we used an open Voice Activity Detection model 将音频文件分成更短的段子;其次,对每个段子应用了新开发的语音LID模型;我们的模型遵循ECAPA-TDNN架构 and extends the open-source model trained on VoxLingua107 by 15 new languages. Finally, we applied an over-segmentation approach that simultaneously proposed multiple, potentially overlapping speech segmentations. We relied on the mining approach to align the most likely ones. Supplementary Fig. shows this pipeline. 56 13 14 1 SONAR SONAR文本和语音编码器是在ref中开发的。 using a two-step approach (Supplementary Fig. 首先,仅为文本模式学习了大量的多语言表示。然后,教师-学生培训被用来扩展嵌入空间到语音模式。文本嵌入空间被用编码器-解码器方法训练,使用多个目标的组合:翻译,解码自动编码和句子嵌入空间中的平均平方错误(MSE)损失目标。 , that is, parallel data to translation from 200 to 200 languages. Speech encoders were trained only on ASR data and by grouping languages into linguistic genealogical groups following ref. , for example, Italic, Common Turkic or Indo-Iranian languages. To obtain optimal performance, we determined the optimal convergence separately for each language (that is, when to stop training). This yielded a separate speech encoder for each language. The amount of available ASR data for each language is provided in Supplementary Table . The speech encoders were initialized with w3v-best 2.0 speech front end. Preceding work performed max pooling or mean pooling of the output states of the speech front end to obtain a fixed-size embedding of the speech signal , 一项废除研究表明,可以通过使用三层变压器解码器获得更好的结果。 . Teacher–student training consisted of minimizing the MSE loss with respect to the embedding of the ASR text transcriptions. These embeddings were obtained by the SONAR text encoder, which was kept constant. No translations (into English) were used. 8 2 6 16 8 54 57 8 发言人 首先,我们计算了所有过分化的语音细分的嵌入,对于文本领域,我们使用了与NLLB项目完全相同的文本。 and embedded them with the SONAR encoder. Exhaustive pairwise comparison can be efficiently performed with the FAISS toolkit . Similarity is measured with a margin criterium as first introduced in ref. : 6 58 52 where and 是源和目标句子,以及 ( )标志着 最接近的邻居 in the other language. We set to 16. x y NNK 的 x k x k As an example, this amounts to comparing a hundred thousand hours of speech with more than 20,000 million English sentences, which yielded about eight thousand hours of aligned Arabic speech. Modelling SEAMLESSM4T 模型基于我们的多任务 UNITY 架构,我们提出的统一翻译模型基于 Vanilla UNITY ,一个两个通道的解码框架,首先生成文本,然后通过预测分散的声学单元生成语音(参见“部分” )。 与梵蒂冈 UNITY 模型相比 , (1) the core S2TT model, initialized from scratch in UNITY, is replaced with an X2T model that supports text as input and is pretrained to jointly optimize the tasks of ASR, S2TT and T2TT (see section ‘ ’),和(2)浅薄的T2U模型(被称为T2U单元编码器和第二通道单元解码器。 )被更深的基于变压器的编码器 - 解码器模型取代,其中有六个变压器层,这些变压器在ASR数据上预先训练(参见“ ’). An improved version of UNITY, dubbed UNITY2, replaces the autoregressive T2U with a new non-autoregressive (NAR) T2U decoder. This NAR T2U model delivers stronger accuracy because of its hierarchical upsampling from subwords to characters and then to units. 59 Multilingual discrete acoustic units 59 X2T fine-tuning 59 S2ST fine-tuning The pretraining of X2T yielded a stronger speech encoder and a higher quality first-pass text decoder, whereas the scaling and pretraining of the T2U model allowed us to better handle multilingual unit generation without interference. Furthermore, the switch to non-autoregressive T2U decoding improved S2ST inference speed by three times. 多语种单独声学单元 最近的作品通过使用自我监督的分散声学单元作为构建直接语音翻译模型的目标,实现了最先进的翻译性能 , . This consists of decomposing the S2ST problem into a speech-to-unit translation step and a unit-to-speech conversion step. We extracted continuous speech representations using XLS-R 并将这些代表性绘制为分散的代币。 分散的代币(也称为单元词汇)的集合通过应用程序来学习 -means algorithm to a set of multilingual audio samples. The -means centroids 类似于一个代码簿,用于将一个 XLS-R 语音表示序列绘制成一个序列的 centroid 指数或语音单元。 = 10,000 具有来自 XLS-R-1B 35 层的功能,代表 35 种支持的目标语言。 5 60 61 k k K 对于单元到语音转换步骤,我们遵循ref。 and built a multilingual vocoder for speech synthesis from the learnt multilingual units. This model is responsible for synthesizing audios from a sequence of units that SEAMLESSM4T models will predict. 62 未经监督的演讲预练 Self-supervised pretraining with unlabelled speech audio data is a practical approach for leveraging unlabelled data. With pretraining, we can bootstrap the quality of translation models and make the most out of our supervised paired data. We pretrained a speech encoder following our improved W2V-BERT 2.0. It follows w2v-BERT W2V-BERT 2.0 使用更多的代码簿和使用随机投射量化器的额外的伪装预测任务 (RPQ) 我们的 W2V-BERT 2.0 模型首次通过超过 143 种语言的 1 百万小时的开放语音音数据进行培训。 ,具有 24 个 Conformer 层 并且大约有 600 万个模型参数。对于 v2 版本,我们将未标记的数据量从 1 万小时扩展到 4.5 万小时的音频。 . It is trained on only 0.5 million hours, spanning over 1,400 languages. The largest model in scale is USM . It is a proprietary multilingual speech pretrained model with 12 million hours of data and more than 300 languages in coverage. 63 64 63 65 23 50 Text-to-text translation models The text processing components of our SEAMLESSM4T models were pretrained on the task of text-to-text translation, a much more resourced task than speech translation. Consider, for instance, the English–Italian direction, one of the highly resourced pairs in T2TT with more than 128 million parallel sentences—only 2 million pairs of English text paired with Italian audio are available for S2TT. A key step in training multilingual text-to-text translation models is learning a shared vocabulary with a text tokenizer. Following ref. , we used SentencePiece with the BPE algorithm 在 NLLB-200 中使用的代币器 suffers from missing key Chinese characters because of artefacts of sampling. This sampling does not favour logo-graphic writing systems with a large number of unique symbols. To fix this issue, we forced the inclusion of these characters. Our new tokenizer improves the coverage of the MTSU top 5K Chinese characters from 54% to 84%. 6 66 67 6 To train our multilingual text-to-text model, we followed the same data preparation and training pipelines in ref. 使用停止 . Having a smaller language coverage allowed us to significantly decrease the size of the model to 1.3B parameters and only use NLLB-200 training data in the 95 SEAMLESSM4T languages. 6 68 通过伪标签增加数据 As with any sequence-to-sequence task, speech translation performance is dependent on the availability of high-quality training data. However, the amount of human-labelled data is scarce compared with its T2TT or ASR counterparts. To address this shortage of labelled data, we resort to pseudo-labelling , 使用多语言的 T2TT 模型(例如 NLLB 模型)生成伪标记 S2TT 数据的 ASR 数据。 69 70 为了增加 S2ST 数据,通常使用 TTS 模型将语音数据集中的文本转换为合成语音。 , 这种合成语音,反过来,转换为分散的单元进行训练。这两个步骤的单元提取过程是一个缓慢的过程,由于TTS模型的依赖性更难扩展。我们绕过了语音合成的需要,并在36个目标语音语言中训练了多语言文本对单元(T2U)模型。 4 5 X2T fine-tuning 我们的多任务UNITY框架的第一个关键部分是X2T模型,该模型是基于合规程序的编码器的多编码器序列到序列模型。 for speech input and another transformer-based encoder 两种编码器都与相同的文本解码器相结合,并共同精心调整,以优化ASR、S2TT和T2TT的任务。 65 71 我们的X2T模型包括将语音编码器,W2V-BERT 2.0从M2连接,后固定与长度适配器,以下示长的音频序列,与文本编码器 - 解码器从M3(补充图。 )对于长度适配器,我们使用了修改后的M适配器。 ,我们将Q、K和V的三个独立聚合模块替换为共享聚合模块,以提高效率。 4 72 ( 用來源的語言 ⟩, paired with its transcription ( 文本翻译( text) in a target language ⟨ 为了允许跨模式的意义转移,X2T模型进行了精细调整,共同优化了以下客观功能: x x y ℓt 此外,我们还优化了一项辅助客观功能,以代码级知识蒸(),以进一步将知识从强大的MT模型转移到学生语音翻译任务(S2TT)。 最终损失是所有三个损失的重量的总和: , 和 are scalar hyper-parameters tuned on the development data. α β γ S2ST fine-tuning In the last stage of fine-tuning multitask UNITY, we initialized the model with the pretrained X2T model (see section ‘ ’) and a pretrained T2U model, similar to the one used for pseudo-labelling S2ST data in M4. The T2U model used for pseudo-labelling is referred to as the teacher T2U model with 12 transformer layers encoder–decoder. For initialization, we used a smaller student T2U model with only six layers to optimize inference and distill the labels of the stronger T2U. In the second version of SEAMLESSM4T, UNITY2 replaces the second-pass autoregressive unit decoder in UNITY with a NAR unit decoder. We adopted the decoder architecture of FastSpeech2 并将其扩展到分散的单位生成。UNITY2 开始从分字符长度到字符长度等级采样 T2U 编码器输出,然后扩展到单位长度。 . The architecture is shown in detail in Supplementary Information section . X2T fine-tuning 73 74 四八 我们用X-eng和eng-X S2ST翻译数据的组合对S2ST任务进行了精心调整,总计为121,000小时,我们冻结了与X2T模型相符的模型重量,仅对T2U组件进行了精心调整,以确保模型在上一阶段精心调整的任务上的性能保持不变。 自动和人为评估 BLASER 2.0 布拉格 2.0 (ref. )是BLASER的新版本 , which works with both speech and text modalities, and hence being modality-agnostic. Like the first version, our approach leverages the similarity between input and output sentence embeddings. The new version uses SONAR embeddings Supplementary Information section ,支持57种语音语言和202种文本语言(本文提交时由SONAR覆盖语言),并可扩展到未来的编码器,用于共享相同嵌入空间的新语言或模式。 24 75 III.3.1 更具体地说,在BLASER 2.0中,我们将源输入、任何S2ST、S2TT 或 T2TT 模型的翻译输出和参考语音部分或文本转换为SONAR嵌入引擎。 人类评价 Apart from automatic metrics such as (ASR) BLEU and BLASER 2.0, we used human metrics such as XSTS , which measures semantic similarity between a source and target translation, and a standard Mean Opinion Score (as standardized in Recommendation ITU-T P.800, henceforth MOS), which measures (1) naturalness, (2) sound quality and (3) clarity of audio generations to evaluate our models. To obtain more robust language-level scores, we also incorporate a calibration set and calibration methodology, the same used to evaluate the NLLB models 除了 XSTS,我们还获得了 MOS 评估,以了解目标语音中的其他音频质量方面。 . 26 6 v1 的 可靠性 We built a replicable noise-robustness evaluation benchmark based on FLEURS (noisy FLEURS), which covers 102 languages, two speech tasks (S2TT and ASR), and various noise types (natural noises and music). To create simulated noisy audios, we sampled audio clips from MUSAN 在“噪音”和“音乐”类别中,我们将它们与原来的FLEURS语音音相混合,在不同的信号与噪音比率(SNR): 10, 5, 0, −5, −10, −15 和 −20。我们以 BLEU-SNR 曲线(对于 S2TT)或 WER-SNR 曲线(对于 ASR)来比较模型,这说明了语音输入的噪音水平增加(即 SNR 下降)时模型性能下降的程度。 76 我们遵循Ref。 为了评估模型强度与扬声器变异,通过使用发音级质量指标计算平均组平均分数和组次变量系数,而不是使用BLEU作为质量指标,我们使用 chrF,该指标在发音级别具有更好的稳定性。计算两种强度指标不需要明确的扬声器子组标签。我们将评估样本和相应的发音级别 chrF分数按内容(转录)组合,然后计算了平均组次平均分数 chrFMS和平均组次变量系数 CoefVarMS,定义如下: 47 哪里 两个指标是互补的:chrFMS提供了一个标准化的质量指标,与传统的体级指标不同,它考虑了扬声器的变异,而CoefVarMS提供了一个标准化的质量变异指标在扬声器的变异中。 G 负责的人 毒性检测 Inspired by ASR-BLEU, this work proposes using ASR-ETOX as a new metric to detect added toxicity in speech and evaluate added toxicity for S2ST ability of SEAMLESSM4T. Essentially, this metric follows a cascaded framework by first deploying a standard ASR module (that is, the same that is used for ASR-BLEU as defined in Supplementary Table ), then the toxicity detection module, ETOX ,使用毒性200单词列表 对于 S2TT,翻译的输出可以直接用 ETOX 进行评估。在两种情况下(S2ST 和 S2TT),我们在发音或句子级别测量了增加的毒性。我们首先计算了评估数据集中的每个输入和相应的输出的毒性检测。然后,我们比较了它们,并将一个案例计算为仅当输出值超过输入时添加的毒性。 , MuTox the same number of languages by the zero-shot property. However, accounting for validated quality, we report MuTox only the languages that have been benchmarked . Again in both cases (S2ST and S2TT), we measured added toxicity at the utterance or sentence level. In this case, a sentence contains added toxicity if MuTox scores is >0.9 in the output and <0.5 in the input. We have experimentally validated these thresholds for several languages with human bilingual speakers for several pairs of languages. For S2TT, we computed MuTox in transcribed speech and target text. For S2ST, we computed MuTox in source and target speech. 2 27 6 29 29 For toxicity mitigation, we implement two techniques for the mitigation of added toxicity. Before training, we filter out training pairs with imbalanced toxicity. Furthermore, we use Mintox 在推断时间. 特别是,主工作流会生成无限制搜索的翻译假设。 然后,毒性分类器在这个假设上运行。 如果没有毒性被检测到,我们会提供在输出中检测到的翻译假设。 但是,如果在输出中检测到毒性,我们会在输入上运行分类器。 如果毒性不平衡(即在输入中没有毒性被检测到),我们会重新运行与缓解的翻译,即 BEAMFILTERING 步骤。 这个 BEAMFILTERING 是以输入中不应该出现在输出中的多标记表达式来处理,并将它们从雷射搜索假设中排除出来的。 请注意,我们在输入中没有毒性(换句话说,我们不处理输入中有毒性 11 We used two datasets to analyse added toxicity. First, we deployed FLEURS to better align with our human evaluation effort and other evaluative components of this work. Furthermore, we used the English-only HOLISTICBIAS framework , which has been shown to trigger true added toxicity in previous studies . In this work, we extend HOLISTICBIAS to speech by applying the default English TTS model from MMS . 77 27 23 Speech extension of MULTILINGUAL HOLISTICBIAS To compare the performances across modalities (S2ST and S2TT), we extended MULTILINGUAL HOLISTICBIAS to speech ( ). We used this generated TTS data as input for S2TT and S2ST and as a reference for S2ST. We conducted the translations in two directions: eng–X and X–eng. Concretely, in X–eng, we translated both masculine and feminine versions of the speech. It is worth noting that some target languages are not available in the SEAMLESSM4T S2ST model, so we performed translations on only 17 languages for the S2ST task in the eng–X direction. For S2TT in eng–X, we have all languages included in the MULTILINGUAL HOLISTICBIAS dataset ( 作为参考,我们实验中使用的完整语言列表可在补充表中找到。 . 23 https://github.com/facebookresearch/fairseq/tree/main/examples/mms#tts-1 n 26 In terms of evaluation metrics for S2TT, we used chrF. For S2ST, we used ASRchrf (the transcription is done by WHISPER-LARGE and WHISPER-MEDIUM 对于 eng–X 和 X–eng,分别, chrF 与 S2TT 相同地计算,但在 S2ST 中,预测和参考的文本都被正常化) 和 BLASER 2.0 时,值得注意的是,在 BLASER 2.0 上进行评估时,我们只包括了 14 种语言(arb, cat, deu, eng, fra, nld, por, ron, rus, spa, swe, tha, ukr 和 urd)为 eng–X 方向(从生成的 TTS 数据和我们的 S2ST 模型中可用的语言的重叠)。 20 Data availability To make our work available to the community, we provide open source of the following data sets, models and code at GitHub ( ): (1) SEAMLESSM4T models, including model weights for SEAMLESSM4T-LARGE (2.3B parameters) and SEAMLESSM4T-MEDIUM (1.2B parameters), as well as their inference code and fine-tuning recipes powered by our new modelling toolkit FAIRSEQ2 ( ); (2)创建对齐语音数据的工具,包括元数据,以重塑SEAMLESSALIGN基于STOPES的无过滤470,000小时管道( )创建类似SEAMLESSALIGN和SONAR的语音编码器在37种语言和文本编码器在200种语言的对齐( ) and (3) a text-free S2ST automatic evaluation model, BLASER 2.0, inclusive of model weights and inference scripts. https://github.com/facebookresearch/seamless_communication https://github.com/facebookresearch/fairseq2 https://github.com/facebookresearch/stopes https://github.com/facebookresearch/SONAR 参考 Lavie, A. et al. Janus-III: speech-to-speech translation in multiple languages. In Vol. 1, 99–102 (IEEE, 1997). Proc. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing Wahlster, W. (ed.) (Springer, 2000). Verbmobil: Foundations of Speech-to-Speech Translation Nakamura, S. et al. The atr multilingual speech-to-speech translation system. , 365–376 (2006). IEEE Trans. Audio Speech Lang. Process. 14 Article Google Scholar Jia, Y. et al. Direct speech-to-speech translation with a sequence-to-sequence model. In , 1123–1127 (ISCA, 2019). Proc. Interspeech 2019 Lee, A. et al. Direct speech-to-speech translation with discrete units. In (eds Muresan, S. et al.) Vol. 1, 3327–3339 (Association for Computational Linguistics, 2022). Proc. 60th Annual Meeting of the Association for Computational Linguistics NLLB Team Scaling neural machine translation to 200 languages. , 841–846 (2024). Nature 630 Article CAS Google Scholar Agarwal, M. et al. Findings of the IWSLT 2023 evaluation campaign. In (eds Salesky, E. et al.) 1–61 (Association for Computational Linguistics, 2023). Proc. 20th International Conference on Spoken Language Translation (IWSLT 2023) Duquenne, P.-A., Schwenk, H. & Sagot, B. SONAR: sentence-level multimodal and language-agnostic representations. Preprint at (2023). https://arxiv.org/abs/2308.11466 Popović, M. chrF: character n-gram F-score for automatic MT evaluation. In (eds Bojar, O. et al.) 392–395 (Association for Computational Linguistics, 2015). Proc. Tenth Workshop on Statistical Machine Translation Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In (eds Isabelle, P. et al.) 311–318 (Association for Computational Linguistics, 2002). Proc. 40th Annual Meeting of the Association for Computational Linguistics Costa-jussà, M. R., Dale, D., Elbayad, M. & Yu, B. Added toxicity mitigation at inference time for multimodal and massively multilingual translation. In (eds Scarton, C. et al.) Vol. 1, 360–372 (European Association for Machine Translation, 2024). Proc. 25th Annual Conference of the European Association for Machine Translation ACTFL. World-readiness standards for learning languages. (2023). https://www.actfl.org/educator-resources/world-readiness-standards-for-learning-languages Desplanques, B., Thienpondt, J. & Demuynck, K. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In (eds Meng, H. et al.) 3830–3834 (ISCA, 2020). Proc. Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event Valk, J. & Alumäe, T. VOXLINGUA107: a dataset for spoken language recognition. In 652–658 (IEEE, 2021). Proc. 2021 IEEE Spoken Language Technology Workshop, Chen, M., Heffernan, K., Çelebi, O., Mourachko, A. & Schwenk, H. xSIM++: An improved proxy to bitext mining performance for low-resource languages. In (eds Rogers, A. et al.) Vol. 2, 101–109 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Heffernan, K., Çelebi, O. & Schwenk, H. Bitext mining using distilled sentence representations for low-resource languages. In , 2101–2112 (eds Goldberg, Y. et al.) (Association for Computational Linguistics, 2022). Findings of the Association for Computational Linguistics: EMNLP 2022 Feng, F., Yang, Y., Cer, D., Arivazhagan, N. & Wang, W. Language-agnostic BERT sentence embedding. In (eds Muresan, S. et al.) Vol. 1, 878–891 (Association for Computational Linguistics, 2022). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Conneau, A. et al. FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech. In , 798–805 (IEEE, 2023). 2022 IEEE Spoken Language Technology Workshop (SLT) Zhang, Z.-H. et al. Speak foreign languages with your own voice: cross-lingual neural codec language modeling. Preprint at (2023). https://arxiv.org/abs/2303.03926 Radford, A. et al. Robust speech recognition via large-scale weak supervision. In , article no. 1182, 28492–28518 (ACM, 2023). Proceedings of the 40th International Conference on Machine Learning Rubenstein, P. K. et al. AudioPaLM: a large language model that can speak and listen. Preprint at (2023). https://arxiv.org/abs/2306.12925 Casanova, E. et al. YourTTS: towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In , Vol. 162, 2709–2720 (PMLR, 2022). Proc. 39th International Conference on Machine Learning Pratap, V. et al. Scaling speech technology to 1,000+ languages. , 1–52 (2024). J. Mach. Learn. Res. 25 Dale, D. & Costa-jussà, M. R. Blaser 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation. In (eds Al-Onaizan, Y. et al.) 16075–16085 (Association for Computational Linguistics, 2024). Findings of the Association for Computational Linguistics: EMNLP 2024 Chen, P.-J. et al. Speech-to-speech translation for a real-world unwritten language. In (eds Rogers, A. et al.) 4969–4983 (Association for Computational Linguistics, 2023). Findings of the Association for Computational Linguistics: ACL 2023 Licht, D. et al. Consistent human evaluation of machine translation across language pairs. In (eds Duh, K. & Guzmán, F.) Vol. 1, 309–321 (Association for Machine Translation in the Americas, 2022). Proc. 15th Biennial Conference of the Association for Machine Translation in the Americas Costa-jussà, M. et al. Toxicity in multilingual machine translation at scale. In , 9570–9586 (Association for Computational Linguistics, 2023). Findings of the Association for Computational Linguistics: EMNLP 2023 Ghosh, S., Lepcha, S., Sakshi, S., Shah, R. R. & Umesh, S. DeToxy: a large-scale multimodal dataset for toxicity classification in spoken utterances. In 2022, 5185–5189 (2022); . Proc. Interspeech https://api.semanticscholar.org/CorpusID:247940269 Costa-jussà, M. R. et al. MuTox: universal multilingual audio-based toxicity dataset and zero-shot detector. In (eds Ku, L.-W. et al.) 5725–5734 (Association for Computational Linguistics, 2024). Findings of the Association for Computational Linguistics: ACL 2024 Stanovsky, G., Smith, N. A. & Zettlemoyer, L. Evaluating gender bias in machine translation. In (eds Korhonen, A. et al.) 1679–1684 (Association for Computational Linguistics, 2019). Proc. 57th Annual Meeting of the Association for Computational Linguistics Levy, S., Lazar, K. & Stanovsky, G. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In (eds Moens, M. F. et al.) 2470–2480 (Association for Computational Linguistics, 2021). Findings of the Association for Computational Linguistics: EMNLP 2021 Costa-jussà, M. R. et al. Interpreting gender bias in neural machine translation: Multilingual architecture matters. , 11855–11863 (2022). Proc. AAAI Conf. Artif. Intell. 36 Google Scholar Costa-jussà, M. R., Basta, C. & Gállego, G. I. Evaluating gender bias in speech translation. In (eds Calzolari, N. et al.) 2141–2147 (European Language Resources Association, 2022). Proceedings of the Thirteenth Language Resources and Evaluation Conference Bentivogli, L. et al. Gender in danger? Evaluating speech translation technology on the MuST-SHE corpus. In (eds Jurafsky, D. et al.) 6923–6933 (Association for Computational Linguistics, 2020). Proc. 58th Annual Meeting of the Association for Computational Linguistics Costa-jussà, M. et al. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In , 14141–14156 (Association for Computational Linguistics, 2023). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Zuckerman, E. The polyglot internet. (2008). https://ethanzuckerman.com/the-polyglot-internet/ Belekar, A., Sunka, S., Bhawar, N. & Bagade, S. Voice based e-mail for the visually impaired. , 8–12 (2020). Int. J. Comput. Appl. 175 Google Scholar Wang, S., Cooper, N. & Eby, M. From human-centered to social-centered artificial intelligence: Assessing ChatGPT’s impact through disruptive events. (2024). Big Data Soc. https://doi.org/10.1177/20539517241290220 Koenecke, A. et al. Racial disparities in automated speech recognition. , 7684–7689 (2020). Proc. Natl Acad. Sci. USA 117 Article ADS CAS PubMed PubMed Central Google Scholar Ngueajio, M. K. & Washington, G. Hey ASR system! Why aren’t you more inclusive? Automatic speech recognition systems’ bias and proposed bias mitigation techniques. A literature review. In (eds Chen, J. Y. C. et al.) Vol. 13518, 421–440 (Springer, 2022). Proc. International Conference on Human-Computer Interaction Elbow, P. The shifting relationships between speech and writing. , 283–303 (1985). Coll. Compos. Commun. 36 Article Google Scholar Trilla, A. & Alias, F. Sentence-based sentiment analysis for expressive text-to-speech. , 223–233 (2013). IEEE Trans. Audio Speech Lang. Process. 21 Article Google Scholar Iranzo-Sánchez, J., Civera, J. & Juan, A. From simultaneous to streaming machine translation by leveraging streaming history. In , 6972–6985 (Association for Computational Linguistics, 2022). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Rybakov, O. et al. Streaming Parrotron for on-device speech-to-speech conversion. Preprint at (2022). https://arxiv.org/abs/2210.13761 Koehn, P. Europarl: A parallel corpus for statistical machine translation. In , 79–86 (2005). Proceedings of Machine Translation Summit X: Papers Ziemski, M., Junczys-Dowmunt, M. & Pouliquen, B. The United Nations Parallel Corpus v1.0. In (eds Calzolari, N. et al.) 3530–3534 (European Language Resources Association, 2016). Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16) Wang, C., Pino, J., Wu, A. & Gu, J. CoVoST: a diverse multilingual speech-to-text translation corpus. In (eds Calzolari, N. et al.) 4197–4203 (European Language Resources Association, 2020). Proceedings of the Twelfth Language Resources and Evaluation Conference Wang, C., Wu, A., Gu, J. & Pino, J. CoVoST 2 and Massively Multilingual Speech Translation. In , 2247–2251 (ISCA, 2021). Proc. Interspeech 2021 Salesky, E. et al. The multilingual TEDx corpus for speech recognition and translation. In , 3655–3659 (ISCA, 2021). Proc. Interspeech 2021 Zhang, Y. et al. Google USM: scaling automatic speech recognition beyond 100 languages. Preprint at (2023). https://arxiv.org/abs/2303.01037 Schwenk, H. Filtering and mining parallel data in a joint multilingual space. In (eds Gurevych, I. & Miyao, Y.) Vol. 2, 228–234 (Association for Computational Linguistics, 2018). Proc. 56th Annual Meeting of the Association for Computational Linguistics Artetxe, M. & Schwenk, H. Margin-based parallel corpus mining with multilingual sentence embeddings. In (eds Korhonen, A. et al.) 3197–3203 (Association for Computational Linguistics, 2019). Proc. 57th Annual Meeting of the Association for Computational Linguistics Artetxe, M. & Schwenk, H. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. , 597–610 (2019). Trans. Assoc. Comput. Linguist. 7 Article Google Scholar Duquenne, P.-A., Gong, H. & Schwenk, H. Multimodal and multilingual embeddings for large-scale speech mining. In (eds Ranzato, M. et al.) Vol. 34, 15748–15761 (Curran Associates, 2021). Advances in Neural Information Processing Systems Duquenne, P.-A. et al. SpeechMatrix: a large-scale mined corpus of multilingual speech-to-speech translations. In (eds Rogers, A. et al.) Vol. 1, 16251–16269 (2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Silero. Silero VAD: pre-trained enterprise-grade voice activity detector (VAD), number detector and language classifier. (2021). GitHub https://github.com/snakers4/silero-vad Khurana, S., Laurent, A. & Glass, J. SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation. , 1493–1504 (2022). IEEE J. Sel. Top. Signal Process. 16 Douze, M. et al. The Faiss library. Preprint at (2024). https://arxiv.org/abs/2401.08281 Inaguma, H. et al. UnitY: two-pass direct speech-to-speech translation with discrete units. In (eds Rogers, A. et al.) Vol. 1, 15655–15680 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Tjandra, A., Sakti, S. & Nakamura, S. Speech-to-speech translation between untranscribed unknown languages. In , 593–600 (IEEE, 2019). 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Babu, A. et al. XLS-R: self-supervised Cross-lingual Speech Representation Learning at Scale. In , 2278–2282 (ISCA, 2022). Proc. Interspeech 2022 Gong, H. et al. Multilingual speech-to-speech translation into multiple target languages. Preprint at (2023). https://arxiv.org/abs/2307.08655 Chung, Y.-A. et al. w2v-BERT: combining contrastive learning and masked language modeling for self-supervised speech pre-training. In , 244–250 (IEEE, 2021). 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Chiu, C.-C., Qin, J., Zhang, Y., Yu, J. & Wu, Y. Self-supervised learning with random-projection quantizer for speech recognition. In , 3915–3924 (PMLR, 2022). Proc. 39th International Conference on Machine Learning Gulati, A. et al. Conformer: convolution-augmented transformer for speech recognition. In , 5036–5040 (ISCA, 2020). Proc. Interspeech 2020 Kudo, T. & Richardson, J. SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. In (eds Blanco, E. & Lu, W.) 66–71 (Association for Computational Linguistics, 2018). Proc. 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations Sennrich, R., Haddow, B. & Birch, A. Neural machine translation of rare words with subword units. In (eds Erk, K. & Smith, N. A.) Vol. 1, 1715–1725 (Association for Computational Linguistics, 2016). Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Andrews, P. et al. stopes - modular machine translation pipelines. In (eds Che, W. & Shutova, E.) 258–265 (Association for Computational Linguistics, 2022). Proc. 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations Jia, Y. et al. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In , 7180–7184 (IEEE, 2019). 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Pino, J., Xu, Q., Ma, X., Dousti, M. J. & Tang, Y. Self-training for end-to-end speech translation. In , 1476–1480 (2020). Proc. Interspeech 2020 Vaswani, A. et al. Attention is all you need. In (eds Guyon, I. et al.) Vol. 30 (Curran Associates, 2017). Advances in Neural Information Processing Systems Zhao, J., Yang, H., Haffari, G. & Shareghi, E. M-Adapter: modality adaptation for end-to-end speech-to-text translation. In , 111–115 (ISCA, 2022). Proc. Interspeech 2022 Ren, Y. et al. FastSpeech 2: fast and high-quality end-to-end text to speech. In (2021). Proc. 2021 International Conference on Learning Representations Shih, K. J. et al. RAD-TTS: parallel flow-based tts with robust alignment learning and diverse synthesis. In (2021). ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models Chen, M. et al. BLASER: A text-free speech-to-speech translation evaluation metric. In (eds Rogers, A. et al.) Vol. 1, 9064–9079 (Association for Computational Linguistics, 2023). Proc. 61st Annual Meeting of the Association for Computational Linguistics Snyder, D., Chen, G. & Povey, D. MUSAN: a music, speech, and noise corpus. Preprint at (2015). https://arxiv.org/abs/1510.08484 Smith, E. M., Hall, M., Kambadur, M., Presani, E. & Williams, A. “I’m sorry to hear that”: finding new biases in language models with a holistic descriptor dataset. In (eds Goldberg, Y. et al.) 9180–9211 (Association for Computational Linguistics, 2022). Proc. 2022 Conference on Empirical Methods in Natural Language Processing 认可 我们感谢S. Edunov和A. Fan帮助塑造该项目的早期阶段;S. Bhosale,V. Goswami,F. Hernandez和Y. Zhang为他们在构建更强的音频模型进行基准支持;M. Seejoor和M. Duppenthaler为他们对BLASER 1.0的贡献;K. Klyushkin为他帮助构建更好的体验;A. Moghevnikov为他对FAIRSEQ2和SONAR推断的贡献;A. Ni和X. Zhang为他们在构建更强的音频模型的帮助;N. Seejoor和M. Duppenthaler为他们在建立演示的帮助;V. Goswami,S. Huhan,B. Acun-Uyan和C.-J. Wu为他帮助提供新的效率优化建议;B. Al 本文在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下可用。 本文在 CC by 4.0 Deed (Attribution 4.0 International) 许可证下可用。