Improving Self-Supervision for Language Pretraining: Summary

Table of Links

• Abstract

• Acknowledgements

  
  

• 1 INTRODUCTION

• 1.1 Overview

• 1.2 Contributions

  
  

• 2 BACKGROUND

• 2.1 Self-Supervised Language Pretraining

• 2.2 Naturally-Occurring Data Structures

• 2.3 Sentence Variational Autoencoder

• 2.4 Summary

  
  

• 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING

• 3.1 Improving Language Representation Learning via Sentence Ordering Prediction

• 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training

• 3.3 Summary

  
  

• 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA

• 4.1 Learning Entity Representations from Hyperlinks

• 4.2 Learning Discourse-Aware Sentence Representations from Document Structures

• 4.3 Learning Concept Hierarchies from Document Categories

• 4.4 Summary

  
  

• 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY

• 5.1 Disentangling Semantics and Syntax in Sentence Representations

• 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar

• 5.3 Summary

  
  

• 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS

• 6.1 Long-Form Data-to-Text Generation

• 6.2 Long-Form Text Summarization

• 6.3 Story Generation with Constraints

• 6.4 Summary

  
  

• 7 CONCLUSION

  
  

• APPENDIX A - APPENDIX TO CHAPTER 3

• APPENDIX B - APPENDIX TO CHAPTER 6

• BIBLIOGRAPHY

3.3 Summary

In this chapter, we described the SOP loss for pretraining language models and a pretraining technique for improving model performance on in-context few-shot learning. In Section 3.1, we compared SOP to NSP on both intrinsic and extrinsic tasks, finding that the two losses encode different information. We showed that the models pretrained with SOP achieve state-of-the-art performance across various benchmarks. In Section 3.2, we benchmarked 4 self-supervised tasks on 13 tasks, finding that similar to human-annotated datasets, self-supervision can also lead to improved downstream task performance. Our analysis uncovered several factors that contributed the improvements, including dataset sizes for the self-supervised tasks, the few-shot templates, and the semantic similarities between the training and evaluation tasks. In addition, we also experimented with concatenating forward and backward language modeling losses to achieve bidirectional training for language models, finding that our proposed approaches showed better performance than the unidirectional language modeling loss but worse than masked language modeling on downstream tasks (see Appendix A.1 for more details).