Learning Semantic Knowledge from Wikipedia: Summary

Table of Links

• Abstract

• Acknowledgements

  
  

• 1 INTRODUCTION

• 1.1 Overview

• 1.2 Contributions

  
  

• 2 BACKGROUND

• 2.1 Self-Supervised Language Pretraining

• 2.2 Naturally-Occurring Data Structures

• 2.3 Sentence Variational Autoencoder

• 2.4 Summary

  
  

• 3 IMPROVING SELF-SUPERVISION FOR LANGUAGE PRETRAINING

• 3.1 Improving Language Representation Learning via Sentence Ordering Prediction

• 3.2 Improving In-Context Few-Shot Learning via Self-Supervised Training

• 3.3 Summary

  
  

• 4 LEARNING SEMANTIC KNOWLEDGE FROM WIKIPEDIA

• 4.1 Learning Entity Representations from Hyperlinks

• 4.2 Learning Discourse-Aware Sentence Representations from Document Structures

• 4.3 Learning Concept Hierarchies from Document Categories

• 4.4 Summary

  
  

• 5 DISENTANGLING LATENT REPRESENTATIONS FOR INTERPRETABILITY AND CONTROLLABILITY

• 5.1 Disentangling Semantics and Syntax in Sentence Representations

• 5.2 Controllable Paraphrase Generation with a Syntactic Exemplar

• 5.3 Summary

  
  

• 6 TAILORING TEXTUAL RESOURCES FOR EVALUATION TASKS

• 6.1 Long-Form Data-to-Text Generation

• 6.2 Long-Form Text Summarization

• 6.3 Story Generation with Constraints

• 6.4 Summary

  
  

• 7 CONCLUSION

  
  

• APPENDIX A - APPENDIX TO CHAPTER 3

• APPENDIX B - APPENDIX TO CHAPTER 6

• BIBLIOGRAPHY

4.4 Summary

In this chapter, we described approaches to exploiting various naturally-occurring structures on Wikipedia. In Section 4.1, we used hyperlinks as natural supervision for two kinds of entity representations: CER and DER. For CER, we asked models to predict entity descriptions given context sentences in which the entities appear. For DER, we asked models to predict mention texts in the context sentences. Our proposed approaches were evaluated on a benchmark for entity representations and showed promising results.

In Section 4.2, we used article structures to train sentence encoders. The article structures are cast as multi-task learning objectives, encouraging the sentence-level models to encode information with respect to the broader context in which it situates. We evaluated the models on a discourse-related benchmark, finding that using the losses beyond sentences helped model performance on the discourse tasks.

In Section 4.3, we used Wikipedia category graphs to induce knowledge related to textual entailment. We treated the parent-child relations in Wikipedia category graphs as the entailment relations and built a training dataset for textual entailment. We found that training on our proposed datasets improves model performance on low-resource textual entailment tasks and we obtained similar improvements when extending our approaches to multilingual settings.