Author: (1) Shravan Cheekati, Georgia Institute of Technology (shravan.cheekati@gatech.edu). Author: Author: (1) Shravan Cheekati, Georgia Institute of Technology (shravan.cheekati@gatech.edu). Table of Links Introduction Related Work Methodology Experiments Conclusion and References Introduction Introduction Related Work Related Work Methodology Methodology Experiments Experiments Conclusion and References Conclusion and References 1. Introduction Transformer models have revolutionized the field of natural language processing (NLP) and computer vision (CV) in recent years. Since the introduction of the Transformer architecture by Vaswani et al. [11], these models have achieved state-of-the-art performance on a wide range of tasks, such as machine translation, sentiment analysis, and image classification [3, 4, 7]. The success of Transformers can be attributed to their ability to capture long-range dependencies and their scalability to large amounts of data [11]. However, the training of Transformer models is resource-intensive and time-consuming, requiring significant computational power and energy consumption [10]. To address this issue, various techniques have been proposed to optimize the training process and reduce the computational requirements of Transformer models [9,12]. One promising approach is the early-bird ticket hypothesis, which suggests that subnetworks capable of matching the performance of fully-trained networks can be identified early in the training process [5]. This hypothesis has been successfully applied to CNNs, leading to significant resource optimization and cost reduction in their training [1, 13]. However, the applicability of the early-bird ticket hypothesis to Transformer models has not been extensively explored. In this research, we investigate the early-bird ticket hypothesis in Transformer models, focusing on vision transformers and language models. By identifying early-bird tickets in these architectures, we aim to optimize the training process and reduce the computational requirements, making Transformer models more accessible and efficient. 2. Related Work The early-bird ticket hypothesis was first introduced by Frankle et al. [5] in the context of CNNs. They discovered that subnetworks capable of matching the performance of fully-trained networks could be identified early in the training process. This finding has led to the development of various techniques to identify and exploit early-bird tickets in CNNs [1, 13]. In the domain of Transformers, there have been limited explorations of the early-bird ticket hypothesis. One notable work is EarlyBERT by Kovaleva et al. [2], which investigated the applicability of the early-bird ticket hypothesis to BERT. They found that early-bird tickets exist in BERT and can be used to optimize the fine-tuning process. However, their work focused solely on BERT and did not provide a comparative analysis across different Transformer architectures. Other works have explored various techniques to optimize the training and inference of Transformer models. For example, Michel et al. [8] proposed a method to prune attention heads in Transformers, reducing the computational requirements while maintaining performance. Sanh et al. [9] introduced DistilBERT, a distilled version of BERT that achieves comparable performance with fewer parameters and faster inference times. Despite these efforts, the potential speedup and resource optimization achievable through the early-bird ticket hypothesis in Transformers have not been fully explored. Many existing works rely on the slow and rigorous process of the train-prune-retrain methodology [6], which can be time-consuming and resource-intensive. In this research, we aim to address these limitations by investigating the early-bird ticket hypothesis across different Transformer architectures, including vision transformers and language models. We explore efficient methods to identify early-bird tickets and evaluate their performance in comparison to fully-trained models. Our goal is to provide insights into the applicability of the early-bird ticket hypothesis in Transformers and contribute to the development of more efficient training strategies for these powerful models. Author: (1) Shravan Cheekati, Georgia Institute of Technology (shravan.cheekati@gatech.edu). Author: Author: (1) Shravan Cheekati, Georgia Institute of Technology (shravan.cheekati@gatech.edu). This paper is available on arxiv under CC BY-SA 4.0 DEED license. This paper is available on arxiv under CC BY-SA 4.0 DEED license. available on arxiv available on arxiv
