Author:
(1) Mingda Chen. Table of Links Abstract


Acknowledgements


1 INTRODUCTION


1.1 Overview


1.2 Contributions


2 BACKGROUND


2.1 Self-Supervised Language Pretraining


2.2 Naturally-Occurring Data Structures


2.3 Sentence Variational Autoencoder


2.4 Summary


3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING


3.1 Improving Language Representation Learning via Sentence Ordering Prediction


3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training


3.3 Summary


4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA


4.1 Learning Entity Representations from Hyperlinks


4.2 Learning Discourse-Aware Sentence Representations from Document Structures


4.3 Learning Concept Hierarchies from Document Categories


4.4 Summary


5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY


5.1 Disentangling Semantics and Syntax in Sentence Representations


5.2 Controllable Paraphrase Generation with a Syntactic Exemplar


5.3 Summary


6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS


6.1 Long-Form Data-to-Text Generation


6.2 Long-Form Text Summarization


6.3 Story Generation with Constraints


6.4 Summary


7 CONCLUSION


APPENDIX A - APPENDIX TO CHAPTER 3


APPENDIX B - APPENDIX TO CHAPTER 6


BIBLIOGRAPHY CHAPTER 1 - INTRODUCTION Written language is ubiquitous. Humans use writing to communicate ideas and store knowledge in their daily lives. These activities naturally produce traces of human intelligence, resulting in abundant, freely-available textual data: e.g., Wikipedia,[1] Reddit,[2] and Fandom,[3] among others. These data often contain sophisticated knowledge expressed with complex language structures. For example, for encyclopedias, there usually are dedicated structures for connecting pieces of information scattered around different places for the convenience of readers (e.g., hyperlinks that point the same person or events mentioned in other documents to the same place for disambiguation). Aside from explicit structures, corpora have rich implicit structures. For example, the data pair in bilingual text shares the same semantic meaning but differs in syntactic forms. The implicit difference between the data pair allows us to disentangle the semantic and syntactic information implied in the data structure. Despite the rich structures, recent advances in NLP have been driven by deep neural models trained on a massive amount of plain text, which often strips away the knowledge and structure from the input. This thesis research approaches to better drive supervision from various naturally-occurring textual resources. In particular, we (1) improve ways of transforming plain text into training signals; (2) propose approaches to exploit the rich structures in Wikipedia and paraphrases; and (3) create evaluation benchmarks from fan-contributed websites to reflect real-world challenges. Below we briefly introduce these three areas and summarize our contributions. This paper is available on arxiv under CC 4.0 license. [1] https://www.wikipedia.org/, an online collaborative encyclopedia. [2] https://www.reddit.com/, an online forum for discussion and web content rating. [3] https://www.fandom.com/, a fan-contributed encyclopedia of movies, films, and other media. Author: (1) Mingda Chen. Author: Author: (1) Mingda Chen. Table of Links Abstract Acknowledgements 1 INTRODUCTION 1.1 Overview 1.2 Contributions 2 BACKGROUND 2.1 Self-Supervised Language Pretraining 2.2 Naturally-Occurring Data Structures 2.3 Sentence Variational Autoencoder 2.4 Summary 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING 3.1 Improving Language Representation Learning via Sentence Ordering Prediction 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training 3.3 Summary 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA 4.1 Learning Entity Representations from Hyperlinks 4.2 Learning Discourse-Aware Sentence Representations from Document Structures 4.3 Learning Concept Hierarchies from Document Categories 4.4 Summary 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY 5.1 Disentangling Semantics and Syntax in Sentence Representations 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar 5.3 Summary 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS 6.1 Long-Form Data-to-Text Generation 6.2 Long-Form Text Summarization 6.3 Story Generation with Constraints 6.4 Summary 7 CONCLUSION APPENDIX A - APPENDIX TO CHAPTER 3 APPENDIX B - APPENDIX TO CHAPTER 6 BIBLIOGRAPHY Abstract Abstract Abstract Acknowledgements Acknowledgements Acknowledgements 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 1.1 Overview 1.1 Overview 1.1 Overview 1.2 Contributions 1.2 Contributions 1.2 Contributions 2 BACKGROUND 2 BACKGROUND 2 BACKGROUND 2 BACKGROUND 2.1 Self-Supervised Language Pretraining 2.1 Self-Supervised Language Pretraining 2.1 Self-Supervised Language Pretraining 2.2 Naturally-Occurring Data Structures 2.2 Naturally-Occurring Data Structures 2.2 Naturally-Occurring Data Structures 2.3 Sentence Variational Autoencoder 2.3 Sentence Variational Autoencoder 2.3 Sentence Variational Autoencoder 2.4 Summary 2.4 Summary 2.4 Summary 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING 3.1 Improving Language Representation Learning via Sentence Ordering Prediction 3.1 Improving Language Representation Learning via Sentence Ordering Prediction 3.1 Improving Language Representation Learning via Sentence Ordering Prediction 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training 3.3 Summary 3.3 Summary 3.3 Summary 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA 4.1 Learning Entity Representations from Hyperlinks 4.1 Learning Entity Representations from Hyperlinks 4.1 Learning Entity Representations from Hyperlinks 4.2 Learning Discourse-Aware Sentence Representations from Document Structures 4.2 Learning Discourse-Aware Sentence Representations from Document Structures 4.2 Learning Discourse-Aware Sentence Representations from Document Structures 4.3 Learning Concept Hierarchies from Document Categories 4.3 Learning Concept Hierarchies from Document Categories 4.3 Learning Concept Hierarchies from Document Categories 4.4 Summary 4.4 Summary 4.4 Summary 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY 5.1 Disentangling Semantics and Syntax in Sentence Representations 5.1 Disentangling Semantics and Syntax in Sentence Representations 5.1 Disentangling Semantics and Syntax in Sentence Representations 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar 5.3 Summary 5.3 Summary 5.3 Summary 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS 6.1 Long-Form Data-to-Text Generation 6.1 Long-Form Data-to-Text Generation 6.1 Long-Form Data-to-Text Generation 6.2 Long-Form Text Summarization 6.2 Long-Form Text Summarization 6.2 Long-Form Text Summarization 6.3 Story Generation with Constraints 6.3 Story Generation with Constraints 6.3 Story Generation with Constraints 6.4 Summary 6.4 Summary 6.4 Summary 7 CONCLUSION 7 CONCLUSION 7 CONCLUSION 7 CONCLUSION APPENDIX A - APPENDIX TO CHAPTER 3 APPENDIX A - APPENDIX TO CHAPTER 3 APPENDIX A - APPENDIX TO CHAPTER 3 APPENDIX B - APPENDIX TO CHAPTER 6 APPENDIX B - APPENDIX TO CHAPTER 6 APPENDIX B - APPENDIX TO CHAPTER 6 BIBLIOGRAPHY BIBLIOGRAPHY BIBLIOGRAPHY CHAPTER 1 - INTRODUCTION Written language is ubiquitous. Humans use writing to communicate ideas and store knowledge in their daily lives. These activities naturally produce traces of human intelligence, resulting in abundant, freely-available textual data: e.g., Wikipedia,[1] Reddit,[2] and Fandom,[3] among others. These data often contain sophisticated knowledge expressed with complex language structures. For example, for encyclopedias, there usually are dedicated structures for connecting pieces of information scattered around different places for the convenience of readers (e.g., hyperlinks that point the same person or events mentioned in other documents to the same place for disambiguation). Aside from explicit structures, corpora have rich implicit structures. For example, the data pair in bilingual text shares the same semantic meaning but differs in syntactic forms. The implicit difference between the data pair allows us to disentangle the semantic and syntactic information implied in the data structure. Despite the rich structures, recent advances in NLP have been driven by deep neural models trained on a massive amount of plain text, which often strips away the knowledge and structure from the input. This thesis research approaches to better drive supervision from various naturally-occurring textual resources. In particular, we (1) improve ways of transforming plain text into training signals; (2) propose approaches to exploit the rich structures in Wikipedia and paraphrases; and (3) create evaluation benchmarks from fan-contributed websites to reflect real-world challenges. Below we briefly introduce these three areas and summarize our contributions. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [1] https://www.wikipedia.org/, an online collaborative encyclopedia. [2] https://www.reddit.com/, an online forum for discussion and web content rating. [3] https://www.fandom.com/, a fan-contributed encyclopedia of movies, films, and other media.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Leveraging Natural Supervision for Language Representation Learning and Generation: Introduction

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

5 Ways to Measure Privacy Risks in Text Data

Leveraging Natural Supervision: Appendix A - Appendix to Chapter 3

Leveraging Natural Supervision: Naturally-Occurring Data Structures

Leveraging Natural Supervision for Language Representation Learning and Generation: Overview

Leveraging Natural Supervision for Language Representation Learning and Generation: Contributions

5 Ways to Measure Privacy Risks in Text Data

Leveraging Natural Supervision: Appendix A - Appendix to Chapter 3

Leveraging Natural Supervision: Naturally-Occurring Data Structures

Leveraging Natural Supervision for Language Representation Learning and Generation: Overview

Leveraging Natural Supervision for Language Representation Learning and Generation: Contributions

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps