Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work
2.1 Text to Vocal Generation
2.2 Text to Motion Generation
2.3 Audio to Motion Generation


RapVerse Dataset
3.1 Rap-Vocal Subset
3.2 Rap-Motion Subset


Method
4.1 Problem Formulation
4.2 Motion VQ-VAE Tokenizer
4.3 Vocal2unit Audio Tokenizer
4.4 General Auto-regressive Modeling


Experiments
5.1 Experimental Setup
5.2 Main Results Analysis and 5.3 Ablation Study


Conclusion and References Related Work
2.1 Text to Vocal Generation
2.2 Text to Motion Generation
2.3 Audio to Motion Generation Related Work 2.1 Text to Vocal Generation 2.1 Text to Vocal Generation 2.2 Text to Motion Generation 2.2 Text to Motion Generation 2.3 Audio to Motion Generation 2.3 Audio to Motion Generation RapVerse Dataset
3.1 Rap-Vocal Subset
3.2 Rap-Motion Subset RapVerse Dataset 3.1 Rap-Vocal Subset 3.1 Rap-Vocal Subset 3.2 Rap-Motion Subset 3.2 Rap-Motion Subset Method
4.1 Problem Formulation
4.2 Motion VQ-VAE Tokenizer
4.3 Vocal2unit Audio Tokenizer
4.4 General Auto-regressive Modeling Method 4.1 Problem Formulation 4.1 Problem Formulation 4.2 Motion VQ-VAE Tokenizer 4.2 Motion VQ-VAE Tokenizer 4.3 Vocal2unit Audio Tokenizer 4.3 Vocal2unit Audio Tokenizer 4.4 General Auto-regressive Modeling 4.4 General Auto-regressive Modeling Experiments
5.1 Experimental Setup
5.2 Main Results Analysis and 5.3 Ablation Study Experiments 5.1 Experimental Setup 5.1 Experimental Setup 5.2 Main Results Analysis and 5.3 Ablation Study 5.2 Main Results Analysis and 5.3 Ablation Study Conclusion and References Conclusion and References Conclusion and References A. Appendix A. Appendix Given a piece of lyrics text, our goal is to generate rap-style vocals and whole-body motions, including body movements, hand gestures, and facial expressions that resonate with the lyrics. With the help of our RapVerse dataset, we propose a novel framework that not only represents texts, vocals, and motions as unified token forms but also integrates token modeling in a unified model. As illustrated in Fig. 3, our model consists of multiple tokenizers for motion (Sec. 4.2) and vocal (Sec. 4.3) token conversions, as well as a general Large Text-Motion-Audio Foundation Model (Sec. 4.4) that targets for audio token synthesize and motion token creation, based on rap lyrics. 4.1 Problem Formulation 4.2 Motion VQ-VAE Tokenizer 4.3 Vocal2unit Audio Tokenizer Overall, we leverage the self-supervised framework [45] in speech resynthesis domain to learn vocal representations from the audio sequences. Specifically, we train a Vocal2unit audio tokenizer to build a discrete tokenized representation for the human singing voice. The vocal tokenizer consists of three encoders and a vocoder. The encoders include three different parts: (1) the semantic encoder; (2) the F0 encoder; and (3) the singer encoder. We will introduce each component of the model separately. 4.4 General Auto-regressive Modeling After optimizing via this training objective, our model learns to predict the next token, which can be decoded into different modality features. This process is similar to text word generation in language models, while the “word” in our method such as <face_02123>, does not have explicit semantic information, but can be decoded into continuous modality features. face_02123 Inference and Decoupling. In the inference stage, we use different start tokens to specify which modality to generate. The textual input is encoded as features to guide token inference. We also adopt a top-k algorithm to control the diversity of the generated content by adjusting the temperature, as generating vocals and motions based on lyrics is a creation process with multiple possible answers. After token prediction, a decoupling algorithm is used to process output tokens to make sure tokens from different modalities are separated and temporally aligned. These discrete tokens will be further decoded into text-aligned vocals and motions Inference and Decoupling Authors:
(1) Jiaben Chen, University of Massachusetts Amherst;
(2) Xin Yan, Wuhan University;
(3) Yihang Chen, Wuhan University;
(4) Siyuan Cen, University of Massachusetts Amherst;
(5) Qinwei Ma, Tsinghua University;
(6) Haoyu Zhen, Shanghai Jiao Tong University;
(7) Kaizhi Qian, MIT-IBM Watson AI Lab;
(8) Lie Lu, Dolby Laboratories;
(9) Chuang Gan, University of Massachusetts Amherst. Authors: Authors (1) Jiaben Chen, University of Massachusetts Amherst; (2) Xin Yan, Wuhan University; (3) Yihang Chen, Wuhan University; (4) Siyuan Cen, University of Massachusetts Amherst; (5) Qinwei Ma, Tsinghua University; (6) Haoyu Zhen, Shanghai Jiao Tong University; (7) Kaizhi Qian, MIT-IBM Watson AI Lab; (8) Lie Lu, Dolby Laboratories; (9) Chuang Gan, University of Massachusetts Amherst. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Text-to-Rap AI Turns Lyrics Into Vocals, Gestures, and Facial Expressions

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Multimodal Dataset for Synthesizing Rap Vocals and 3D Motion

A Single Prompt Will Have This AI Rapping and Dancing

The RapVerse Dataset: A New Benchmark in Text-to-Music and Motion Generation

A Multimodal Dataset for Synthesizing Rap Vocals and 3D Motion

This AI Turns Lyrics Into Fully Synced Song and Dance Performances

A Multimodal Dataset for Synthesizing Rap Vocals and 3D Motion

A Single Prompt Will Have This AI Rapping and Dancing

The RapVerse Dataset: A New Benchmark in Text-to-Music and Motion Generation

A Multimodal Dataset for Synthesizing Rap Vocals and 3D Motion

This AI Turns Lyrics Into Fully Synced Song and Dance Performances

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps