EleutherAI in collaboration with INCITE Project, AAI CERC lab at the Université de Montréal, have introduced Hi-NOLIN, an open-source English-Hindi bilingual model.

EleutherAI Launches Open-Source English-Hindi Bilingual Model, Hi-NOLIN

In a new paper DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention, a research team from DeepSpeed of Microsoft presents the DeepSpeed-VisualChat framework, which is designed to optimize LLMs by incorporating ...

Microsoft’s DeepSpeed-VisualChat: Breaking Boundaries in Multi-Modal Language Models

Cutting-edge large language models (LLMs) have displayed remarkable capabilities in various text-based tasks, but there is a growing desire to extend their abilities beyond the confines of text-only processing.

In a new paper FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields, a research team from KAIST and Scatter Lab introduces FaceCLIPNeRF, a novel text-driven pipeline that enable high-quality face manipulation using ...

Tag: Artificial Intelligence

In the fast-paced world of technology investment, one question looms large: Can Nvidia sustain its incredible growth and high margins? Over the past two quarters, Nvidia (NASDAQ: NVDA) has displayed remarkable performance,

AMD and Intel Forge Alliances with FAANG Giants to Challenge Nvidia’s AI Dominance

But the more interesting attribute of both AMD's and Intel's software stacks is that each has a portability feature that aims to enable software developers to take programming code written in CUDA and port that code over to their other platforms for use on other hardware, with as little recoding as possible.

How AMD and Intel Are Teaming Up With FAANG to Take On Nvidia

The scaling rule of language models has produced success like never before. These huge language models have gotten novel emerging capabilities in addition to demonstrating tremendous superiority over earlier paradigms for many disciplines when trained on ...

This AI Paper Unveils OpenBA: An Open-Sourced 15B Parameter Bilingual Model Outperforming Predecessors and Aiding Chinese-centric NLP Advancements

This article explores these challenges, drawing insights from a recent paper on DeepSpeed-VisualChat and discussing the limitations and prospects of GPT models. The paper titled 'DeepSpeed ...

Challenges and Future of Multi-Modal Conversations in AI

To revolutionize collaboration with generative AI, Bard has introduced its most advanced model yet. This innovation promises to be a game-changer, allowing users to tailor responses to their specific needs seamlessly. Whether it’s drafting a trip ...

Bard Unveils Enhanced Capabilities: Integrating with Gmail, Drive, and Other Google Apps

In 2022, the DeepSpeed team made a number of breakthroughs to improve mixture of experts (MoE) models, making them more efficient, faster, and less costly. Specifically, they were able to reduce training cost by 5x, reduce MoE parameter size by up to 3.7x ...

Research @ Microsoft 2022: A look back at a year of accelerating … – Microsoft

DeepSpeed

deepspeed

Authors:
(1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors;
(2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors;
(3) Sahal Shaji, Mohamed bin Zayed University of AI;
(4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Hisham Cholakkal, Mohamed bin Zayed University of AI;
(7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University;
(8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University;
(9) Ming-Hsuan Yang, University of California - Merced and Google Research;
(10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 7 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Work
3. Method
4. Data Annotation Pipeline
5. Experiments
6. Conclusion and References Supplementary Material (Part 1) A. Additional Implementation Details
B. Additional Downstream Tasks
C. Additional Qualitative Results Supplementary Material (Part 2) D. Dataset Visualization
E. Limitations and Future Work
F. Ethics and Societal Impact We provide supplementary material for a deeper understanding and more analysis related to the main paper, arranged as follows: Additional implementation details (Appendix A)
Additional downstream tasks (Appendix B
Additional qualitative results (Appendix C)
Dataset visualizations (Appendix D)
Limitations and future work (Appendix E)
Ethics and societal impact (Appendix F) A. Additional Implementation Details A.1. Evaluation Metrics Mask Recall: To quantify region-specific grounding, we propose a ‘mask recall’ metric, utilizing a two-tiered validation approach. Initially, predicted masks are mapped to ground-truth masks via a one-to-one set assignment, followed by IoU computation for these pairs. Pairs surpassing a 0.5 IoU threshold proceed to a textual similarity assessment using BERT. A pair is considered a true positive (TP) only if both IoU and BERT similarity exceed their 0.5 thresholds; otherwise, it is classified as a false positive (FP). The mask recall is subsequently calculated using the standard formula, normalizing the number of TPs by the total ground-truth mask count. A.2. Model Architecture and Training In all of our experiments, we use Vicuna LLM [60] with 7B parameters. The design of region encoder is motivated from GPT4RoI [57] and grounding image encoder and pixel decoder are inspired from LISA [21]. The V-L and L-P layers are implemented using 2 layer MLP with GELU activation as in LLaVA-v1.5 [28]. We use PyTorch to implement our GLaMM and use Deepspeed zero-2 optimization during training. Specifically, our model is trained using two types of losses: auto-regressive cross-entropy loss for text generation and a linear combination of per-pixel binary crossentropy loss and DICE loss for segmentation. During training, the global image encoder and grounding image encoder are kept frozen and the region encoder, projection layers (VL and L-P) and the pixel decoder are fully finetuned, while the LLM is LORA finetuned with α = 8. Our codes and pretrained models will be publicly released. A.2.1 Pretraining on GranD During pretraining GLaMM is trained on GranD dataset for referring expression segmentation, region-level captioning, image-level captioning and grounded conversation generation (GCG) tasks simultaneously. We use a batch size of 160 and train for a total of 35K iterations during pretraining. We use LORA-8 for efficiently adapting the LLM and initialize the pretraining from GPT4RoI [57] for faster convergence. In the experiment tables in Section. 5, we refer to this model as GLaMM (ZS) which is obtained after pretraining on GranD. A.3. Finetuning on Downstream Tasks We finetune GLaMM on multiple downstream tasks including GCG, referring expression segmentation, region-level captioning and image-level captioning. For GCG, we finetune our model on GranDf dataset. A batch size of 160 is used and the model is trained for 5K iterations in total. It is worth noting that GranDf dataset is a combination of multiple open-source datasets that we repurposed for GCG task using GPT4 [34]. Please refer to Appendix. D for the prompts designed to query GPT4 for constructing GranDf dataset, along with the dataset visualizations. For referring expressions segmentation, we finetune GLaMM on refCOCO, refCOCO+ and refCOCOg datasets. We represent this model as GLaMM (FT) in Tab. 4. Similarly, for region-level captioning, GLaMM (FT) is finetuned on refCOCOg and Visual Genome datasets. For imagelevel captioning, we fine tune GLaMM on LLaVA-Instruct150K [29] dataset. For LLaVA-bench, the model is finetuned on LLaVA-Instruct-80K [29] instruction set. We use eight NVIDIA A100-40GB GPUs in all of our pretraining and finetuning experiments. A.4. Automated Dataset Annotation Pipeline Our automated annotation pipeline incorporates diverse state-of-the-art models at various levels. For Level-1, we use Tag2Text [14] and RAM [58] for image tagging, CoDETR [62], EVAv02 [7], OWL-ViT [33], and POMP [40] for object localization, GRiT [48] and GPT4RoI [57] for attribute generation, and MiDAS [39] for depth estimation. Level-2 leverages BLIP-2 [24] and LLaVA-v1.5 [28, 29] for scene descriptions and landmark categorization, SpaCy [11] for phrase extraction, and MDETR [15] for phrase grounding. For both Level-3 and Level-4, we use Vicuna-v1.5 [60] with 13B parameters, supplemented with in-context examples. Please refer to Appendix A.4 for further details on implementation and LLM prompts used across different pipeline levels. We design a fully automated dataset annotation pipeline using multiple hierarchical levels in the visual domain to construct GranD dataset. The segmentation masks for most of the regions are obtained from SAM [18] annotations by comparing our detected labeled regions with SAMprovided class-agnostic regions. For the remaining regions that do not match with any of the SAM regions, we run SAM model with a bounding box query to obtain masks. Our automated annotation pipeline utilizes only opensource models and incorporates a feedback loop using the chain of thoughts prompting via LLM. As it does not require feedback from the human in the loop, it can be scaled to generate dense noisy labels for a larger number of images, which can then be used to pretrain a larger LMM. Given the availability of enough compute power, this could be a step towards building a larger generic large multi-modal model. We will release our GranD dataset along with the implementation of our automated dataset annotation pipeline for further research. Below we present the LLM prompts we use at different levels of our automated dataset annotation pipeline. A.4.1 LLM Prompts and In-context Learning Landmark categorization: We use LLaVA-v1.5-13B [28] model to assign landmark categories to each image. Please refer to Tab. 7 for primary and fine categories used. Dense Captioning: We arrange objects, attributes and relationships hierarchically to construct a visual scene graph, that is used to query Vicuna-v1.5-13B [60] model along with in-context examples to generate dense captions. The designed prompt is shown in Fig. 6 (a). Extra Context: We query Vicuna-v1.5-13B model to generate additional context about the visual scene. The prompt designed for this purpose is shown in Fig. 6 (b). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 7 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 7 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 7 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 7 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method 4. Data Annotation Pipeline 5. Experiments 6. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method 3. Method 4. Data Annotation Pipeline 4. Data Annotation Pipeline 5. Experiments 5. Experiments 6. Conclusion and References 6. Conclusion and References Supplementary Material (Part 1) Supplementary Material (Part 1) A. Additional Implementation Details B. Additional Downstream Tasks C. Additional Qualitative Results A. Additional Implementation Details A. Additional Implementation Details B. Additional Downstream Tasks B. Additional Downstream Tasks C. Additional Qualitative Results C. Additional Qualitative Results Supplementary Material (Part 2) Supplementary Material (Part 2) D. Dataset Visualization E. Limitations and Future Work F. Ethics and Societal Impact D. Dataset Visualization D. Dataset Visualization E. Limitations and Future Work E. Limitations and Future Work F. Ethics and Societal Impact F. Ethics and Societal Impact We provide supplementary material for a deeper understanding and more analysis related to the main paper, arranged as follows: Additional implementation details (Appendix A) Additional downstream tasks (Appendix B Additional qualitative results (Appendix C) Dataset visualizations (Appendix D) Limitations and future work (Appendix E) Ethics and societal impact (Appendix F) Additional implementation details (Appendix A) Additional downstream tasks (Appendix B Additional qualitative results (Appendix C) Dataset visualizations (Appendix D) Limitations and future work (Appendix E) Ethics and societal impact (Appendix F) A. Additional Implementation Details A.1. Evaluation Metrics Mask Recall: To quantify region-specific grounding, we propose a ‘mask recall’ metric, utilizing a two-tiered validation approach. Initially, predicted masks are mapped to ground-truth masks via a one-to-one set assignment, followed by IoU computation for these pairs. Pairs surpassing a 0.5 IoU threshold proceed to a textual similarity assessment using BERT. A pair is considered a true positive (TP) only if both IoU and BERT similarity exceed their 0.5 thresholds; otherwise, it is classified as a false positive (FP). The mask recall is subsequently calculated using the standard formula, normalizing the number of TPs by the total ground-truth mask count. Mask Recall: A.2. Model Architecture and Training In all of our experiments, we use Vicuna LLM [60] with 7B parameters. The design of region encoder is motivated from GPT4RoI [57] and grounding image encoder and pixel decoder are inspired from LISA [21]. The V-L and L-P layers are implemented using 2 layer MLP with GELU activation as in LLaVA-v1.5 [28]. We use PyTorch to implement our GLaMM and use Deepspeed zero-2 optimization during training. Specifically, our model is trained using two types of losses: auto-regressive cross-entropy loss for text generation and a linear combination of per-pixel binary crossentropy loss and DICE loss for segmentation. During training, the global image encoder and grounding image encoder are kept frozen and the region encoder, projection layers (VL and L-P) and the pixel decoder are fully finetuned, while the LLM is LORA finetuned with α = 8. Our codes and pretrained models will be publicly released. A.2.1 Pretraining on GranD A.2.1 Pretraining on GranD During pretraining GLaMM is trained on GranD dataset for referring expression segmentation, region-level captioning, image-level captioning and grounded conversation generation (GCG) tasks simultaneously. We use a batch size of 160 and train for a total of 35K iterations during pretraining. We use LORA-8 for efficiently adapting the LLM and initialize the pretraining from GPT4RoI [57] for faster convergence. In the experiment tables in Section. 5, we refer to this model as GLaMM (ZS) which is obtained after pretraining on GranD. A.3. Finetuning on Downstream Tasks We finetune GLaMM on multiple downstream tasks including GCG, referring expression segmentation, region-level captioning and image-level captioning. For GCG, we finetune our model on GranD f dataset. A batch size of 160 is used and the model is trained for 5K iterations in total. It is worth noting that GranD f dataset is a combination of multiple open-source datasets that we repurposed for GCG task using GPT4 [34]. Please refer to Appendix. D for the prompts designed to query GPT4 for constructing GranDf dataset, along with the dataset visualizations. f f For referring expressions segmentation, we finetune GLaMM on refCOCO, refCOCO+ and refCOCOg datasets. We represent this model as GLaMM (FT) in Tab. 4. Similarly, for region-level captioning, GLaMM (FT) is finetuned on refCOCOg and Visual Genome datasets. For imagelevel captioning, we fine tune GLaMM on LLaVA-Instruct150K [29] dataset. For LLaVA-bench, the model is finetuned on LLaVA-Instruct-80K [29] instruction set. We use eight NVIDIA A100-40GB GPUs in all of our pretraining and finetuning experiments. A.4. Automated Dataset Annotation Pipeline Our automated annotation pipeline incorporates diverse state-of-the-art models at various levels. For Level-1, we use Tag2Text [14] and RAM [58] for image tagging, CoDETR [62], EVAv02 [7], OWL-ViT [33], and POMP [40] for object localization, GRiT [48] and GPT4RoI [57] for attribute generation, and MiDAS [39] for depth estimation. Level-2 leverages BLIP-2 [24] and LLaVA-v1.5 [28, 29] for scene descriptions and landmark categorization, SpaCy [11] for phrase extraction, and MDETR [15] for phrase grounding. For both Level-3 and Level-4, we use Vicuna-v1.5 [60] with 13B parameters, supplemented with in-context examples. Please refer to Appendix A.4 for further details on implementation and LLM prompts used across different pipeline levels. We design a fully automated dataset annotation pipeline using multiple hierarchical levels in the visual domain to construct GranD dataset. The segmentation masks for most of the regions are obtained from SAM [18] annotations by comparing our detected labeled regions with SAMprovided class-agnostic regions. For the remaining regions that do not match with any of the SAM regions, we run SAM model with a bounding box query to obtain masks. Our automated annotation pipeline utilizes only opensource models and incorporates a feedback loop using the chain of thoughts prompting via LLM. As it does not require feedback from the human in the loop, it can be scaled to generate dense noisy labels for a larger number of images, which can then be used to pretrain a larger LMM. Given the availability of enough compute power, this could be a step towards building a larger generic large multi-modal model. We will release our GranD dataset along with the implementation of our automated dataset annotation pipeline for further research. Below we present the LLM prompts we use at different levels of our automated dataset annotation pipeline. A.4.1 LLM Prompts and In-context Learning A.4.1 LLM Prompts and In-context Learning Landmark categorization : We use LLaVA-v1.5-13B [28] model to assign landmark categories to each image. Please refer to Tab. 7 for primary and fine categories used. Landmark categorization Dense Captioning: We arrange objects, attributes and relationships hierarchically to construct a visual scene graph, that is used to query Vicuna-v1.5-13B [60] model along with in-context examples to generate dense captions. The designed prompt is shown in Fig. 6 (a). Dense Captioning: Extra Context: We query Vicuna-v1.5-13B model to generate additional context about the visual scene. The prompt designed for this purpose is shown in Fig. 6 (b). Extra Context: This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

UAE Researchers Spill the Beans on How Their AI Comprehends Images in Detail

Is ByteDance’s Lumi an AI Experiment Disguised as an AI Creator Community?

ByteDance's Lumi የ AI ሙከራ እንደ AI ፈጣሪ ማህበረሰብ ተደብቋል?

¿Es Lumi de ByteDance un experimento de IA disfrazado de comunidad de creadores de IA?

Èske Lumi ByteDance a se yon eksperyans AI degize kòm yon kominote kreyatè AI?

Արդյո՞ք ByteDance-ի Lumi-ն AI փորձ է, որը քողարկված է որպես AI Ստեղծողների համայնք:

ByteDance の Lumi は AI クリエイター コミュニティを装った AI 実験なのか?

ByteDance의 Lumi는 AI 크리에이터 커뮤니티로 위장한 AI 실험일까요?

ByteDance’s Lumi ezali Expérience ya AI oyo e déguisé lokola Communauté ya Créateur ya AI?

Lumi ຂອງ ByteDance ເປັນການທົດລອງ AI ປອມຕົວເປັນຊຸມຊົນຜູ້ສ້າງ AI ບໍ?

Adakah Lumi ByteDance merupakan Eksperimen AI yang Menyamar sebagai Komuniti Pencipta AI?

Je, Lumi ya ByteDance ni Jaribio la AI Limefichwa kama Jumuiya ya Watayarishi wa AI?

Оё Lumi byteDance як озмоиши AI аст, ки ҳамчун ҷомеаи Офарандаи AI пинҳон карда шудааст?

Ngaba i-ByteDance's Lumi liLingelo le-AI eliguqulelwe njengoluntu lwabadali be-AI?

Authors:
(1) Timothy R. McIntosh;
(2) Teo Susnjak;
(3) Tong Liu;
(4) Paul Watters;
(5) Malka N. Halgamuge. Table of Links Abstract and Introduction Background: Evolution of Generative AI The Current Generative AI Research Taxonomy Innovative Horizon of MOE Speculated Capabilities of Q* Projected Capabilities of AGI Impact Analysis on Generative AI Research Taxonomy Emergent Research Priorities in Generative AI Practical Implications and Limitations of Generative AI Technologies Impact of Generative AI on Preprints Across Disciplines Conclusions, Disclaimer, and References IV. INNOVATIVE HORIZON OF MOE The MoE model architecture represents a pioneering advancement in transformer-based language models, offering unparalleled scalability and efficiency (Fig. 4). As evidenced by recent models like the 1.6 trillion parameter Switch Transformer [285] and the 8x7B parameter Mixtra [286], MoEbased designs are rapidly redefining the frontiers of model scale and performance across diverse language tasks. A. Core Concept and Structure MoE models represent a significant innovation in neural network design, offering enhanced scalability and efficiency in training and inference [287], [288], [110]. At their core, MoE models utilize a sparsity-driven architecture by replacing dense layers with sparse MoE layers comprising multiple expert networks, where each expert is dedicated to a specific subset of the training data or task, and a trainable gating mechanism dynamically allocates input tokens to these experts, thereby optimizing computational resources and effectively adapting to the task’s complexity [94], [21], [110]. MoE models demonstrate a substantial advantage in terms of pretraining speed, outperforming dense models [94], [287]. However, they face challenges in fine-tuning and require substantial memory for inference due to the necessity of loading all experts into Video Random Access Memory (VRAM) [289], [290], [110]. The structure of MoE involves alternating transformer layers with router layers containing gating networks for expert routing, leading to an architecture that allows significant parameter scaling and advanced specialization in problem-solving [291], [21]. A distinguishing characteristic of MoE models is their flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency [289], [292]. The Sparsely-Gated Mixture-of-Experts Layer, a key component of these models, comprises numerous simple feed-forward expert networks and a trainable gating network responsible for expert selection, which can facilitate the dynamic and sparse activation of experts for each input instance, maintaining high computational efficiency [293], [294], [110]. Recent advancements in MoE models, such as those in the Switch Transformer, have highlighted the significant benefits of intelligent routing, when the router’s ability to intelligently route tokens to appropriate experts confers considerable advantages to MoE models, allowing them to scale up model sizes while keeping compute time constant [295], [296], [297]. Experimental evidence suggests that routers learn to route inputs according to data clusters, demonstrating their potential in real-world applications [295], [289]. The core concept and structure of MoE models lie in their dynamic routing and specialization capabilities, offering promising avenues for scaling up neural networks and enhancing their efficiency and adaptability in various tasks, but the robustness of the router must be protected against adversarial attacks [289], [298]. B. Training and Inference Efficiency MoE models, notably Mixtral 8x7B, are renowned for their superior pretraining speed compared to dense models, yet they face hurdles in fine-tuning and demand considerable VRAM for inference, owing to the requirement of loading all experts [289], [290], [110]. Recent advancements in MoE architecture have resulted in notable training cost efficiencies, especially in encoder-decoder models, with evidence showing cost savings of up to fivefold in certain contexts when compared to dense models [21], [289], [298], [287]. Innovations like DeepSpeedMoE [287] offered new architectural designs and model compression, decreasing the MoE model size by approximately 3.7x and optimizing inference to achieve up to 7.3x better latency and cost efficiency. The progression in distributed MoE training and inference, notably with innovations like Lina [299], has effectively tackled the all-to-all communication bottleneck by enhancing tensor partitioning, which not only improves all-to-all communication and training step time, but also optimizes resource scheduling during inference, leading to a substantial reduction in training step time by up to 1.73 times and lowering the 95th percentile inference time by an average of 1.63 times compared to existing systems. These developments have marked a crucial shift in the large model landscape, from dense to sparse MoE models, expanding the potential applications of AI by training higher-quality models with fewer resources. C. Load Balancing and Router Optimization Effective load balancing is essential in MoE models to guarantee a uniform distribution of computational load among experts, with the router network in MoE layers, responsible for selecting the appropriate experts for processing specific tokens, playing a pivotal role in achieving this balance, which is fundamental to the stability and overall performance of MoE models [293], [289], [288], [300], [110]. Developments in router Zloss regularization techniques plays a crucial role in addressing expert imbalance in MoE models by fine-tuning the gating mechanism, ensuring a more equitable workload distribution across experts and fostering a stable training environment, thereby enhancing model performance and reducing training time and computational overhead [301], [302]. Concurrently, the integration of expert capacity management strategies, emerges as a crucial approach in MoE models to regulate the processing abilities of individual experts by setting thresholds on the number of tokens each can handle, effectively averting bottlenecks and ensuring a more efficient and streamlined model operation, leading to improved training processes and heightened performance during complex computational tasks [293], [303], [289]. D. Parallelism and Serving Techniques Recent developments in MoE models highlighted their efficiency in parallelism and serving techniques, significantly influencing large-scale neural networks. DeepSpeed-MoE, for instance, introduces advanced parallelism modes like data parallelism, tensor-slicing for non-expert parameters, and expert parallelism for expert parameters, enhancing model efficiency, as their approach optimizes both latency and throughput in MoE model inference, offering scalable solutions in production environments using multiple Graphics Processing Unit (GPU) devices [287]. MoE models, versatile in applications like multilingual tasks and coding, demonstrated impressive capabilities in handling complex tasks due to their ensemblelike structure within a single framework [304], [305], [306]. Notably, models like Mixtral and Switch Transformer, with over 1.6 trillion parameters, achieved computational efficiency equivalent to a 10 billion-parameter dense model, because they benefited from the sublinear scaling of MoE compute versus model size, leading to substantial accuracy gains within fixed compute budgets [21], [289], [287], [110]. Moreover, DeepSpeed-MoE included model compression techniques, reducing model size by up to 3.7x while maintaining accuracy, and an end-to-end MoE training and inference solution, part of the DeepSpeed library, which was instrumental in serving large-scale MoE models with enhanced speed and costefficiency [287]. These innovations open new directions in AI, shifting from dense to sparse MoE models, where training and deploying higher-quality models with fewer resources become more widely achievable. E. Future Directions and Applications Emerging research on MoE architectures could focus on advancing sparse fine-tuning techniques, exploring instruction tuning methods, and improving routing algorithms to fully utilize performance and efficiency gains. As models scale over one billion parameters, MoE represents a paradigm shift for vastly expanding capabilities across scientific, medical, creative, and real-world applications. Frontier work could also aim to refine auto-tuning of hyperparameters during finetuning to optimize accuracy, calibration, and safety. MoE research continues to push model scale limits while maintaining specialization for transfer learning. Adaptive sparse access allows coordinating thousands of experts to cooperate on tasks ranging from reasoning to open domain dialogue. Continued analysis of routing mechanisms seeks to balance load across experts and minimize redundant computation. As the AI community further investigates MoE methods at scale, these models hold promise for new breakthroughs in language, code generation, reasoning, and multimodal applications. There is great interest in evaluating implications across education, healthcare, financial analysis, and other fields. Outcomes may yield insights not only into model optimization but also for understanding principles behind combinatorial generalization. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Timothy R. McIntosh; (2) Teo Susnjak; (3) Tong Liu; (4) Paul Watters; (5) Malka N. Halgamuge. Authors: Authors: (1) Timothy R. McIntosh; (2) Teo Susnjak; (3) Tong Liu; (4) Paul Watters; (5) Malka N. Halgamuge. Table of Links Abstract and Introduction Abstract and Introduction Background: Evolution of Generative AI Background: Evolution of Generative AI The Current Generative AI Research Taxonomy The Current Generative AI Research Taxonomy Innovative Horizon of MOE Innovative Horizon of MOE Speculated Capabilities of Q* Speculated Capabilities of Q* Projected Capabilities of AGI Projected Capabilities of AGI Impact Analysis on Generative AI Research Taxonomy Impact Analysis on Generative AI Research Taxonomy Emergent Research Priorities in Generative AI Emergent Research Priorities in Generative AI Practical Implications and Limitations of Generative AI Technologies Practical Implications and Limitations of Generative AI Technologies Impact of Generative AI on Preprints Across Disciplines Impact of Generative AI on Preprints Across Disciplines Conclusions, Disclaimer, and References Conclusions, Disclaimer, and References IV. INNOVATIVE HORIZON OF MOE The MoE model architecture represents a pioneering advancement in transformer-based language models, offering unparalleled scalability and efficiency (Fig. 4). As evidenced by recent models like the 1.6 trillion parameter Switch Transformer [285] and the 8x7B parameter Mixtra [286], MoEbased designs are rapidly redefining the frontiers of model scale and performance across diverse language tasks. A. Core Concept and Structure A. Core Concept and Structure MoE models represent a significant innovation in neural network design, offering enhanced scalability and efficiency in training and inference [287], [288], [110]. At their core, MoE models utilize a sparsity-driven architecture by replacing dense layers with sparse MoE layers comprising multiple expert networks, where each expert is dedicated to a specific subset of the training data or task, and a trainable gating mechanism dynamically allocates input tokens to these experts, thereby optimizing computational resources and effectively adapting to the task’s complexity [94], [21], [110]. MoE models demonstrate a substantial advantage in terms of pretraining speed, outperforming dense models [94], [287]. However, they face challenges in fine-tuning and require substantial memory for inference due to the necessity of loading all experts into Video Random Access Memory (VRAM) [289], [290], [110]. The structure of MoE involves alternating transformer layers with router layers containing gating networks for expert routing, leading to an architecture that allows significant parameter scaling and advanced specialization in problem-solving [291], [21]. A distinguishing characteristic of MoE models is their flexibility in managing large datasets, capable of amplifying model capacity by over a thousand times while only experiencing minor reductions in computational efficiency [289], [292]. The Sparsely-Gated Mixture-of-Experts Layer, a key component of these models, comprises numerous simple feed-forward expert networks and a trainable gating network responsible for expert selection, which can facilitate the dynamic and sparse activation of experts for each input instance, maintaining high computational efficiency [293], [294], [110]. Recent advancements in MoE models, such as those in the Switch Transformer, have highlighted the significant benefits of intelligent routing, when the router’s ability to intelligently route tokens to appropriate experts confers considerable advantages to MoE models, allowing them to scale up model sizes while keeping compute time constant [295], [296], [297]. Experimental evidence suggests that routers learn to route inputs according to data clusters, demonstrating their potential in real-world applications [295], [289]. The core concept and structure of MoE models lie in their dynamic routing and specialization capabilities, offering promising avenues for scaling up neural networks and enhancing their efficiency and adaptability in various tasks, but the robustness of the router must be protected against adversarial attacks [289], [298]. B. Training and Inference Efficiency B. Training and Inference Efficiency MoE models, notably Mixtral 8x7B, are renowned for their superior pretraining speed compared to dense models, yet they face hurdles in fine-tuning and demand considerable VRAM for inference, owing to the requirement of loading all experts [289], [290], [110]. Recent advancements in MoE architecture have resulted in notable training cost efficiencies, especially in encoder-decoder models, with evidence showing cost savings of up to fivefold in certain contexts when compared to dense models [21], [289], [298], [287]. Innovations like DeepSpeedMoE [287] offered new architectural designs and model compression, decreasing the MoE model size by approximately 3.7x and optimizing inference to achieve up to 7.3x better latency and cost efficiency. The progression in distributed MoE training and inference, notably with innovations like Lina [299], has effectively tackled the all-to-all communication bottleneck by enhancing tensor partitioning, which not only improves all-to-all communication and training step time, but also optimizes resource scheduling during inference, leading to a substantial reduction in training step time by up to 1.73 times and lowering the 95th percentile inference time by an average of 1.63 times compared to existing systems. These developments have marked a crucial shift in the large model landscape, from dense to sparse MoE models, expanding the potential applications of AI by training higher-quality models with fewer resources. C. Load Balancing and Router Optimization C. Load Balancing and Router Optimization Effective load balancing is essential in MoE models to guarantee a uniform distribution of computational load among experts, with the router network in MoE layers, responsible for selecting the appropriate experts for processing specific tokens, playing a pivotal role in achieving this balance, which is fundamental to the stability and overall performance of MoE models [293], [289], [288], [300], [110]. Developments in router Zloss regularization techniques plays a crucial role in addressing expert imbalance in MoE models by fine-tuning the gating mechanism, ensuring a more equitable workload distribution across experts and fostering a stable training environment, thereby enhancing model performance and reducing training time and computational overhead [301], [302]. Concurrently, the integration of expert capacity management strategies, emerges as a crucial approach in MoE models to regulate the processing abilities of individual experts by setting thresholds on the number of tokens each can handle, effectively averting bottlenecks and ensuring a more efficient and streamlined model operation, leading to improved training processes and heightened performance during complex computational tasks [293], [303], [289]. D. Parallelism and Serving Techniques D. Parallelism and Serving Techniques Recent developments in MoE models highlighted their efficiency in parallelism and serving techniques, significantly influencing large-scale neural networks. DeepSpeed-MoE, for instance, introduces advanced parallelism modes like data parallelism, tensor-slicing for non-expert parameters, and expert parallelism for expert parameters, enhancing model efficiency, as their approach optimizes both latency and throughput in MoE model inference, offering scalable solutions in production environments using multiple Graphics Processing Unit (GPU) devices [287]. MoE models, versatile in applications like multilingual tasks and coding, demonstrated impressive capabilities in handling complex tasks due to their ensemblelike structure within a single framework [304], [305], [306]. Notably, models like Mixtral and Switch Transformer, with over 1.6 trillion parameters, achieved computational efficiency equivalent to a 10 billion-parameter dense model, because they benefited from the sublinear scaling of MoE compute versus model size, leading to substantial accuracy gains within fixed compute budgets [21], [289], [287], [110]. Moreover, DeepSpeed-MoE included model compression techniques, reducing model size by up to 3.7x while maintaining accuracy, and an end-to-end MoE training and inference solution, part of the DeepSpeed library, which was instrumental in serving large-scale MoE models with enhanced speed and costefficiency [287]. These innovations open new directions in AI, shifting from dense to sparse MoE models, where training and deploying higher-quality models with fewer resources become more widely achievable. E. Future Directions and Applications E. Future Directions and Applications Emerging research on MoE architectures could focus on advancing sparse fine-tuning techniques, exploring instruction tuning methods, and improving routing algorithms to fully utilize performance and efficiency gains. As models scale over one billion parameters, MoE represents a paradigm shift for vastly expanding capabilities across scientific, medical, creative, and real-world applications. Frontier work could also aim to refine auto-tuning of hyperparameters during finetuning to optimize accuracy, calibration, and safety. MoE research continues to push model scale limits while maintaining specialization for transfer learning. Adaptive sparse access allows coordinating thousands of experts to cooperate on tasks ranging from reasoning to open domain dialogue. Continued analysis of routing mechanisms seeks to balance load across experts and minimize redundant computation. As the AI community further investigates MoE methods at scale, these models hold promise for new breakthroughs in language, code generation, reasoning, and multimodal applications. There is great interest in evaluating implications across education, healthcare, financial analysis, and other fields. Outcomes may yield insights not only into model optimization but also for understanding principles behind combinatorial generalization. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Surveying the Evolution and Future Trajectory of Generative AI - Innovative Horizon of MOE

Authors:
(1) Liang Wang, Microsoft Corporation, and Correspondence to (wangliang@microsoft.com);
(2) Nan Yang, Microsoft Corporation, and correspondence to (nanya@microsoft.com);
(3) Xiaolong Huang, Microsoft Corporation;
(4) Linjun Yang, Microsoft Corporation;
(5) Rangan Majumder, Microsoft Corporation;
(6) Furu Wei, Microsoft Corporation and Correspondence to (fuwei@microsoft.com). Table of Links Abstract and 1 Introduction 2 Related Work 3 Method 3.1 Synthetic Data Generation 3.2 Training 4 Experiments 4.1 Statistics of the Synthetic Data 4.2 Model Fine-tuning and Evaluation 4.3 Main Results 4.4 Multilingual Retrieval 5 Analysis 5.1 Is Contrastive Pre-training Necessary? 5.2 Extending to Long Text Embeddings and 5.3 Analysis of Training Hyperparameters 6 Conclusion and References A Implementation Details B Test Set Contamination Analysis C Prompts for Synthetic Data Generation D Instructions for Training and Evaluation 4.2 Model Fine-tuning and Evaluation The pretrained Mistral-7b [19] checkpoint is fine-tuned for 1 epoch using the loss in Equation 2. We follow the training recipe from RankLLaMA [24] and utilize LoRA [17] with rank 16. To further reduce GPU memory requirement, techniques including gradient checkpointing, mixed precision training, and DeepSpeed ZeRO-3 are applied. For the training data, we utilize both the generated synthetic data and a collection of 13 public datasets, yielding approximately 1.8M examples after sampling. More details are available in Appendix A. To provide a fair comparison with some previous work, we also report results when the only labeled supervision is the MS-MARCO passage ranking [5] dataset. We evaluate the trained model on the MTEB benchmark [28]. Note that the retrieval category in MTEB corresponds to the 15 publicly available datasets in the BEIR benchmark [42]. Evaluation of one model takes about 3 days on 8 V100 GPUs due to the need to encode a large number of documents. Although our model can accommodate sequence length beyond 512, we only evaluate on the first 512 tokens for efficiency. Official metrics are reported for each category. For more details about the evaluation protocol, please refer to the original papers [28, 42]. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Liang Wang, Microsoft Corporation, and Correspondence to (wangliang@microsoft.com); (2) Nan Yang, Microsoft Corporation, and correspondence to (nanya@microsoft.com); (3) Xiaolong Huang, Microsoft Corporation; (4) Linjun Yang, Microsoft Corporation; (5) Rangan Majumder, Microsoft Corporation; (6) Furu Wei, Microsoft Corporation and Correspondence to (fuwei@microsoft.com). Authors: Authors: (1) Liang Wang, Microsoft Corporation, and Correspondence to (wangliang@microsoft.com); (2) Nan Yang, Microsoft Corporation, and correspondence to (nanya@microsoft.com); (3) Xiaolong Huang, Microsoft Corporation; (4) Linjun Yang, Microsoft Corporation; (5) Rangan Majumder, Microsoft Corporation; (6) Furu Wei, Microsoft Corporation and Correspondence to (fuwei@microsoft.com). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Method 3.1 Synthetic Data Generation 3.1 Synthetic Data Generation 3.2 Training 3.2 Training 4 Experiments 4.1 Statistics of the Synthetic Data 4.1 Statistics of the Synthetic Data 4.2 Model Fine-tuning and Evaluation 4.2 Model Fine-tuning and Evaluation 4.3 Main Results 4.3 Main Results 4.4 Multilingual Retrieval 4.4 Multilingual Retrieval 5 Analysis 5.1 Is Contrastive Pre-training Necessary? 5.1 Is Contrastive Pre-training Necessary? 5.2 Extending to Long Text Embeddings and 5.3 Analysis of Training Hyperparameters 5.2 Extending to Long Text Embeddings and 5.3 Analysis of Training Hyperparameters 6 Conclusion and References 6 Conclusion and References A Implementation Details A Implementation Details B Test Set Contamination Analysis B Test Set Contamination Analysis C Prompts for Synthetic Data Generation C Prompts for Synthetic Data Generation D Instructions for Training and Evaluation D Instructions for Training and Evaluation 4.2 Model Fine-tuning and Evaluation The pretrained Mistral-7b [19] checkpoint is fine-tuned for 1 epoch using the loss in Equation 2. We follow the training recipe from RankLLaMA [24] and utilize LoRA [17] with rank 16. To further reduce GPU memory requirement, techniques including gradient checkpointing, mixed precision training, and DeepSpeed ZeRO-3 are applied. For the training data, we utilize both the generated synthetic data and a collection of 13 public datasets, yielding approximately 1.8M examples after sampling. More details are available in Appendix A. To provide a fair comparison with some previous work, we also report results when the only labeled supervision is the MS-MARCO passage ranking [5] dataset. We evaluate the trained model on the MTEB benchmark [28]. Note that the retrieval category in MTEB corresponds to the 15 publicly available datasets in the BEIR benchmark [42]. Evaluation of one model takes about 3 days on 8 V100 GPUs due to the need to encode a large number of documents. Although our model can accommodate sequence length beyond 512, we only evaluate on the first 512 tokens for efficiency. Official metrics are reported for each category. For more details about the evaluation protocol, please refer to the original papers [28, 42]. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

Improving Text Embeddings with
Large Language Models: Model Fine-tuning and Evaluation

Authors:
(1) Suriya Gunasekar, Microsoft Research;
(2) Yi Zhang, Microsoft Research;
(3) Jyoti Aneja, Microsoft Research;
(4) Caio C´esar Teodoro Mendes, Microsoft Research;
(5) Allie Del Giorno, Microsoft Research;
(6) Sivakanth Gopi, Microsoft Research;
(7) Mojan Javaheripi, Microsoft Research;
(8) Piero Kauffmann, Microsoft Research;
(9) Gustavo de Rosa, Microsoft Research;
(10) Olli Saarikivi, Microsoft Research;
(11) Adil Salim, Microsoft Research;
(12) Shital Shah, Microsoft Research;
(13) Harkirat Singh Behl, Microsoft Research;
(14) Xin Wang, Microsoft Research;
(15) S´ebastien Bubeck, Microsoft Research;
(16) Ronen Eldan, Microsoft Research;
(17) Adam Tauman Kalai, Microsoft Research;
(18) Yin Tat Lee, Microsoft Research;
(19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction
2 Training details and the importance of high-quality data
2.1 Filtering of existing code datasets using a transformer-based classifier
2.2 Creation of synthetic textbook-quality datasets
2.3 Model architecture and training
3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries
4 Evaluation on unconventional problems with LLM grading
5 Data pruning for unbiased performance evaluation
5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis
6 Conclusion and References
A Additional examples for Section 3
B Limitation of phi-1
C Examples for Section 5 2.3 Model architecture and training We use a decoder only transformer [VSP+ 17] model using the FlashAttention implementation of multihead attention (MHA) [DFE+ 22]. We also use MHA and MLP layers in parallel configuration following some recent models like CodeGen [NPH+ 22], PaLM [CND+ 22], and GPT-NeoX [BBH+ 22]. The architecture for our 1.3B parameter phi-1 model consists of 24 layers, hidden dimension of 2048, MLP-inner dimension of 8192, and 32 attention heads of dimension 64 each. The smaller 350M parameter phi1-small model consists of 20 layers, hidden dimension of 1024, MLP-inner dimension of 4096, and 16 attention heads of dimension 64 each. We also use a rotary position embedding [SLP+ 21] with rotary dimension 32. These architectural choices were adopted from [NPH+ 22]. We also use the same tokenizer as codegen-350M-mono [NPH+ 22]. Aside from FlashAttention, our models do not use other techniques like Fill-In-the-Middle (FIM) [BJT+ 22], or Multi-Query-Attention (MQA) [RSR+ 20] that could further boost performance and efficiency [LAZ+ 23]. For both pretraining and finetuning, we concatenate our respective datasets into a single dimensional array with “⟨∣endoftext∣⟩” token used for separating the files. We train our models on sequence length of 2048 sliced from our dataset array with next-token prediction loss. We use fp16 training with AdamW optimizer, linear-warmup-linear-decay learning rate schedule, and attention and residual dropout of 0.1. We train on 8 Nvidia-A100 GPUs using deepspeed. Our pretrained base model phi-1-base was obtained in under 4 days of training. Finetuning to obtain phi-1 used an additional 7 hours on the same hardware. Pretraining. phi-1-base was trained on the CodeTextbook dataset (filtered code-language corpus and synthetic textbooks). We use effective batch size 1024 (including data parallelism and gradient accumulation), maximum learning rate 1e-3 with warmup over 750 steps, and weight decay 0.1, for a total of 36,000 steps. We use the checkpoint at 24,000 steps as our phi-1-base – this is equivalent to ∼ 8 epochs on our CodeTextbook dataset for a total of little over 50B total training tokens. Despite the small size and computation, this model already achieves a 29% accuracy on HumanEval. Finetuning. phi-1 is obtained by finetuning phi-1-base on the CodeExercises dataset. For finetuning, we use the same setup as pretraining, but different hyperparameters: we use effective batchsize of 256, maximum learning rate 1e-4 with 50 steps of warmup, and weight decay 0.01. We train for total of 6,000 steps and pick the best checkpoint (saved every 1000 steps). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Authors: Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References A Additional examples for Section 3 B Limitation of phi-1 C Examples for Section 5 Abstract and 1. Introduction Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References 6 Conclusion and References A Additional examples for Section 3 A Additional examples for Section 3 B Limitation of phi-1 B Limitation of phi-1 C Examples for Section 5 C Examples for Section 5 2.3 Model architecture and training We use a decoder only transformer [VSP+ 17] model using the FlashAttention implementation of multihead attention (MHA) [DFE+ 22]. We also use MHA and MLP layers in parallel configuration following some recent models like CodeGen [NPH+ 22], PaLM [CND+ 22], and GPT-NeoX [BBH+ 22]. The architecture for our 1.3B parameter phi-1 model consists of 24 layers, hidden dimension of 2048, MLP-inner dimension of 8192, and 32 attention heads of dimension 64 each. The smaller 350M parameter phi1-small model consists of 20 layers, hidden dimension of 1024, MLP-inner dimension of 4096, and 16 attention heads of dimension 64 each. We also use a rotary position embedding [SLP+ 21] with rotary dimension 32. These architectural choices were adopted from [NPH+ 22]. We also use the same tokenizer as codegen-350M-mono [NPH+ 22]. Aside from FlashAttention, our models do not use other techniques like Fill-In-the-Middle (FIM) [BJT+ 22], or Multi-Query-Attention (MQA) [RSR+ 20] that could further boost performance and efficiency [LAZ+ 23]. For both pretraining and finetuning, we concatenate our respective datasets into a single dimensional array with “⟨∣endoftext∣⟩” token used for separating the files. We train our models on sequence length of 2048 sliced from our dataset array with next-token prediction loss. We use fp16 training with AdamW optimizer, linear-warmup-linear-decay learning rate schedule, and attention and residual dropout of 0.1. We train on 8 Nvidia-A100 GPUs using deepspeed. Our pretrained base model phi-1-base was obtained in under 4 days of training. Finetuning to obtain phi-1 used an additional 7 hours on the same hardware. Pretraining. phi-1-base was trained on the CodeTextbook dataset (filtered code-language corpus and synthetic textbooks). We use effective batch size 1024 (including data parallelism and gradient accumulation), maximum learning rate 1e-3 with warmup over 750 steps, and weight decay 0.1, for a total of 36,000 steps. We use the checkpoint at 24,000 steps as our phi-1-base – this is equivalent to ∼ 8 epochs on our CodeTextbook dataset for a total of little over 50B total training tokens. Despite the small size and computation, this model already achieves a 29% accuracy on HumanEval. Pretraining. phi-1-base Finetuning. phi-1 is obtained by finetuning phi-1-base on the CodeExercises dataset. For finetuning, we use the same setup as pretraining, but different hyperparameters: we use effective batchsize of 256, maximum learning rate 1e-4 with 50 steps of warmup, and weight decay 0.01. We train for total of 6,000 steps and pick the best checkpoint (saved every 1000 steps). Finetuning. phi-1 phi-1-base This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Textbooks are All You Need: Model Architecture and Training

An Architect’s Guide to the Top 10 Tools Needed to Build the Modern Data Lake

আধুনিক ডেটা লেক তৈরির জন্য প্রয়োজনীয় শীর্ষ 10টি সরঞ্জামের জন্য একজন স্থপতির গাইড

Ein Architektenhandbuch zu den 10 wichtigsten Tools zum Aufbau eines modernen Data Lake

Guía para arquitectos sobre las 10 herramientas principales necesarias para crear un lago de datos moderno

Guide de l’architecte sur les 10 principaux outils nécessaires à la création du lac de données moderne

आधुनिक डेटा लेक के निर्माण के लिए आवश्यक शीर्ष 10 उपकरणों के लिए एक आर्किटेक्ट गाइड

現代のデータレイクを構築するために必要なトップ 10 ツールに関するアーキテクト ガイド

Um guia de arquiteto para as 10 principais ferramentas necessárias para construir o moderno Data Lake

Руководство архитектора по 10 лучшим инструментам, необходимым для создания современного озера данных

Modern Veri Gölünü Oluşturmak İçin Gereken En İyi 10 Araca Yönelik Bir Mimarın Rehberi

Hướng dẫn của Kiến trúc sư về 10 Công cụ hàng đầu cần thiết để Xây dựng Hồ dữ liệu hiện đại

架构师指南：构建现代数据湖所需的十大工具

Large Language Models on Memory-Constrained Devices Using Flash Memory: Conclusion & Discussion

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66 Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraChat • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. • UltraFeedback We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • MT-Bench • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). • AlpacaEval We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66

Zephyr: Direct Distillation of LM Alignment: Experimental Details

Highlight: Check out my GPU VRAM Calculator I’ve always been curious about the GPU VRAM required for training and fine-tuning transformer-based language models. What factors influence VRAM consumption? How does it vary with different model settings? I dug into the topic and conducted my measurements. Other great resources include Stas Bekman’s section from his ML Engineering book, which is the core inspiration for Hugging Face’s model memory anatomy article. Also, check out Eleuther’s blog, which also covers compute costs. Quick note: This post doesn’t investigate the memory usage of quantized models and PEFT fine-tuning techniques like LoRA or QLoRA. Prerequisites for experiments When we talk about RAM, we often use GB (10^9 bytes) and GiB (2^30 bytes) interchangeably. But in reality, we’re dealing with GiB. Take the Nvidia 3090’s “24 GB VRAM” – it’s actually 24 GiB, or about 25.76 GB. To keep things clear, I’ll stick with MiB and GiB. To measure VRAM usage accurately, we need to delete the variable, run garbage collection, clear the CUDA cache, and then measure the VRAM difference. Here’s an example: x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB The ipyexperiments Python package automates this after each cell execution, which is pretty convenient. Before assessing memory usage, it’s important to perform warm-up steps, essentially running the same code twice, to load CUDA kernels that weren’t loaded during the initial setup. Also, we should disable the cache in the decoder, which is used during inference to prevent re-computation of hidden states [1]. Mixed precision training Understanding mixed precision training is key, as it’s commonly used in pretraining and finetuning. Normally, model parameters are stored in float32 format, taking up 4 bytes per parameter. Mixed precision training uses float16, halving the calculation time and reducing the size of activations. But why “mixed”? The training isn’t entirely in half precision. Lower precision can lead to imprecise weight updates or even gradients turning to zero. So, in mixed precision training, the master copy of the weights is kept and updated in fp32, and before each forward pass, these weights are copied into fp16 format. For a deeper dive into mixed precision, check out this fast.ai documentation, which includes a detailed illustration, and Aleksey Bilogur’s blog, which offers practical PyTorch code examples. Handling multi-GPU scenarios What if a model doesn’t fit on a single GPU? There are two scenarios: Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto". Learn more in the accelerate docs.
Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation. Breaking down the components Memory consumption consists of the following components: Train

Inference



CUDA Kernels

✅

✅



Parameters

✅

✅



Activations

✅

✅



Gradients

✅

❌



Optimizer States

✅

❌



Outputs

✅

✅ An interesting aspect of PyTorch is its approach to memory allocation. Essentially, PyTorch rarely releases memory once it’s been allocated. For instance, during the forward pass, activations are calculated and stored in memory. Even after these activations are no longer needed following the backward pass, the memory they occupy isn’t released. This strategy is adopted to avoid the overhead associated with frequent memory allocation calls [3]. CUDA Kernels Upon first using the GPU, CUDA kernels will allocate between 300 MiB to 2000 MiB. This can vary based on GPU, driver, and PyTorch versions. It could be measured by initializing any small tensor and moving it to GPU: x = torch.ones(1).cuda() Parameters When measuring the amount of memory that will be used by parameters, it is important to understand the difference between parameters and buffers. Parameters are the actual weights that are being trained and updated by the optimizer. They could be retrieved by calling model.parameters(). Apart from parameters, fixed tensors exist, which are needed in some computations but do not need to be updated. These are called buffers and may be retrieved by calling model.buffers(). One example of buffers is precomputed positional encodings [4]. So, in this section, under ‘parameters,’ I assume ‘parameters’ + ‘buffers.’ During inference, the memory needed for parameters is straightforward — it’s just the number of parameters multiplied by the number of bytes per parameter. You are specifying the number of bytes per parameter when loading a model like .from_pretrained(..., torch_dtype=torch.float16). For instance, a 7B-parameter model like Mistral, when loaded in half-precision (float16), would take 7.51 × 10**9 × 2 bytes, equating to 14324 MiB. When training as usual, in full precision, 4 bytes per parameter are occupied, mixed precision training is more common though, in this case, we have to maintain both half precision (for forward pass, 2 bytes per param) and full precision model weights (for applying updates to them, 4 bytes per param), so in total it takes 6 bytes per param. Activations ‘Activations’ refer to the intermediate outputs essential for backpropagation. They are usually the memory bottleneck in transformer training, especially since their size scales quadratically with sequence length (we have to store the output of a softmax(Q×K.T) which has Batch Size × Number of Attention Heads × Sequence Length ** 2 shape). There are good estimations of activations size per layer in the “Reducing Activation Recomputation in Large Transformer Models” paper in section 4.1 although for each model activations will differ. For example, in the mentioned paper they also count dropout masks whereas newer architectures like Llama don’t use dropout at all. During training, we store all layer activations for backprop, but in inference, we only keep the current (single) layer’s activations. We can reduce activation size on training in the cost of training speed (slowdown around 20%) by discarding the activations during the forward pass and recalculating them when needed during the backward pass; this is called gradient checkpointing. Gradients Gradients are always stored at full precision, taking 4 bytes per parameter. Optimizer states Optimizers like Adam and SGD have their own memory needs. SGD with momentum and Adam both store a moving average of gradients for each parameter in full precision. Additionally, Adam keeps a moving average of squared gradients. First Moments

Second Moments

Bytes per Param



SGD

❌

❌

0



SGD w momentum

✅

❌

4



ADAM

✅

✅

8 Outputs Finally, the output tensors (Batch Size × Sequence Length × Vocabulary Size) are almost always in float32. This remains true even if the model was loaded at a lower precision because the model itself casts outputs to float32 most of the time [5] [6]. While training, we also need to store probabilities F.softmax(logits, dim=-1) which are the same size as the output tensor. Problems In my experiments with measuring VRAM usage in the notebook, I am facing some persistent mismatch between what my experiments show and the calculated figures, particularly regarding the size of activations during the training’s forward pass. So, there is still something to figure out! Acknowledgments Thanks to Stas Bekman for helping me shape my understanding and Quentin Anthony’s Python gist for VRAM calculation. Highlight : Check out my GPU VRAM Calculator Highlight GPU VRAM Calculator I’ve always been curious about the GPU VRAM required for training and fine-tuning transformer-based language models. What factors influence VRAM consumption? How does it vary with different model settings? I dug into the topic and conducted my measurements. Other great resources include Stas Bekman’s section from his ML Engineering book , which is the core inspiration for Hugging Face’s model memory anatomy article . Also, check out Eleuther’s blog , which also covers compute costs. Stas Bekman’s section from his ML Engineering book Hugging Face’s model memory anatomy article Eleuther’s blog Quick note: This post doesn’t investigate the memory usage of quantized models and PEFT fine-tuning techniques like LoRA or QLoRA. Prerequisites for experiments When we talk about RAM, we often use GB (10^9 bytes) and GiB (2^30 bytes) interchangeably. But in reality, we’re dealing with GiB. Take the Nvidia 3090’s “24 GB VRAM” – it’s actually 24 GiB, or about 25.76 GB. To keep things clear, I’ll stick with MiB and GiB. To measure VRAM usage accurately, we need to delete the variable, run garbage collection, clear the CUDA cache, and then measure the VRAM difference. Here’s an example: x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB x = torch.Tensor(4, 8192, 32000).cuda()
total_vram = get_vram()
del x; gc.collect(); torch.cuda.empty_cache()
x_vram = total_vram - get_vram()
# 4000 MiB The ipyexperiments Python package automates this after each cell execution, which is pretty convenient. ipyexperiments Before assessing memory usage, it’s important to perform warm-up steps, essentially running the same code twice, to load CUDA kernels that weren’t loaded during the initial setup. Also, we should disable the cache in the decoder, which is used during inference to prevent re-computation of hidden states [1] . cache [1] Mixed precision training Understanding mixed precision training is key, as it’s commonly used in pretraining and finetuning. Normally, model parameters are stored in float32 format, taking up 4 bytes per parameter. Mixed precision training uses float16, halving the calculation time and reducing the size of activations. But why “mixed”? The training isn’t entirely in half precision. Lower precision can lead to imprecise weight updates or even gradients turning to zero. So, in mixed precision training, the master copy of the weights is kept and updated in fp32, and before each forward pass, these weights are copied into fp16 format. For a deeper dive into mixed precision, check out this fast.ai documentation , which includes a detailed illustration, and Aleksey Bilogur’s blog , which offers practical PyTorch code examples. fast.ai documentation Aleksey Bilogur’s blog Handling multi-GPU scenarios What if a model doesn’t fit on a single GPU? There are two scenarios: Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto". Learn more in the accelerate docs. Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation. Inference: Use model parallelism to distribute layers across GPUs. This is done automatically in transformers with device_map="auto" . Learn more in the accelerate docs . device_map="auto" accelerate docs Training: Distribute layers, optimizer states, and gradients across GPUs. Depending on your setup, you might use different DeepSpeed ZeRO stages or FSDP [2] for full sharding. The more you shard, the slower training will be because of a communication overhead. For a comparison of multi-GPU training approaches, check out Hugging Face’s documentation . DeepSpeed ZeRO stages FSDP [2] Hugging Face’s documentation Breaking down the components Memory consumption consists of the following components: Train

Inference



CUDA Kernels

✅

✅



Parameters

✅

✅



Activations

✅

✅



Gradients

✅

❌



Optimizer States

✅

❌



Outputs

✅

✅ Train

Inference CUDA Kernels

✅

✅ Parameters

✅

✅ Activations

✅

✅ Gradients

✅

❌ Optimizer States

✅

❌ Outputs

✅

✅ Train Inference Train Train Inference Inference CUDA Kernels ✅ ✅ CUDA Kernels CUDA Kernels ✅ ✅ ✅ ✅ Parameters ✅ ✅ Parameters Parameters ✅ ✅ ✅ ✅ Activations ✅ ✅ Activations Activations ✅ ✅ ✅ ✅ Gradients ✅ ❌ Gradients Gradients ✅ ✅ ❌ ❌ Optimizer States ✅ ❌ Optimizer States Optimizer States ✅ ✅ ❌ ❌ Outputs ✅ ✅ Outputs Outputs ✅ ✅ ✅ ✅ An interesting aspect of PyTorch is its approach to memory allocation. Essentially, PyTorch rarely releases memory once it’s been allocated. For instance, during the forward pass, activations are calculated and stored in memory. Even after these activations are no longer needed following the backward pass, the memory they occupy isn’t released. This strategy is adopted to avoid the overhead associated with frequent memory allocation calls [3] . [3] CUDA Kernels Upon first using the GPU, CUDA kernels will allocate between 300 MiB to 2000 MiB. This can vary based on GPU, driver, and PyTorch versions. It could be measured by initializing any small tensor and moving it to GPU: x = torch.ones(1).cuda() x = torch.ones(1).cuda() Parameters When measuring the amount of memory that will be used by parameters, it is important to understand the difference between parameters and buffers. Parameters are the actual weights that are being trained and updated by the optimizer. They could be retrieved by calling model.parameters() . Apart from parameters, fixed tensors exist, which are needed in some computations but do not need to be updated. These are called buffers and may be retrieved by calling model.buffers() . One example of buffers is precomputed positional encodings [4] . So, in this section, under ‘parameters,’ I assume ‘parameters’ + ‘buffers.’ model.parameters() model.buffers() [4] During inference, the memory needed for parameters is straightforward — it’s just the number of parameters multiplied by the number of bytes per parameter. You are specifying the number of bytes per parameter when loading a model like .from_pretrained(..., torch_dtype=torch.float16) . For instance, a 7B-parameter model like Mistral, when loaded in half-precision (float16), would take 7.51 × 10**9 × 2 bytes, equating to 14324 MiB. .from_pretrained(..., torch_dtype=torch.float16) When training as usual, in full precision, 4 bytes per parameter are occupied, mixed precision training is more common though, in this case, we have to maintain both half precision (for forward pass, 2 bytes per param) and full precision model weights (for applying updates to them, 4 bytes per param), so in total it takes 6 bytes per param. Activations ‘Activations’ refer to the intermediate outputs essential for backpropagation. They are usually the memory bottleneck in transformer training, especially since their size scales quadratically with sequence length (we have to store the output of a softmax(Q×K.T) which has Batch Size × Number of Attention Heads × Sequence Length ** 2 shape). There are good estimations of activations size per layer in the “Reducing Activation Recomputation in Large Transformer Models” paper in section 4.1 although for each model activations will differ. For example, in the mentioned paper they also count dropout masks whereas newer architectures like Llama don’t use dropout at all. softmax(Q×K.T) “Reducing Activation Recomputation in Large Transformer Models” Llama During training, we store all layer activations for backprop, but in inference, we only keep the current (single) layer’s activations. We can reduce activation size on training in the cost of training speed (slowdown around 20%) by discarding the activations during the forward pass and recalculating them when needed during the backward pass; this is called gradient checkpointing . gradient checkpointing Gradients Gradients are always stored at full precision, taking 4 bytes per parameter. Optimizer states Optimizers like Adam and SGD have their own memory needs. SGD with momentum and Adam both store a moving average of gradients for each parameter in full precision. Additionally, Adam keeps a moving average of squared gradients. First Moments

Second Moments

Bytes per Param



SGD

❌

❌

0



SGD w momentum

✅

❌

4



ADAM

✅

✅

8 First Moments

Second Moments

Bytes per Param SGD

❌

❌

0 SGD w momentum

✅

❌

4 ADAM

✅

✅

8 First Moments Second Moments Bytes per Param First Moments First Moments Second Moments Second Moments Bytes per Param Bytes per Param SGD ❌ ❌ 0 SGD SGD ❌ ❌ ❌ ❌ 0 0 SGD w momentum ✅ ❌ 4 SGD w momentum SGD w momentum ✅ ✅ ❌ ❌ 4 4 ADAM ✅ ✅ 8 ADAM ADAM ✅ ✅ ✅ ✅ 8 8 Outputs Finally, the output tensors (Batch Size × Sequence Length × Vocabulary Size) are almost always in float32. This remains true even if the model was loaded at a lower precision because the model itself casts outputs to float32 most of the time [5] [6] . [5] [6] While training, we also need to store probabilities F.softmax(logits, dim=-1) which are the same size as the output tensor. F.softmax(logits, dim=-1) Problems In my experiments with measuring VRAM usage in the notebook , I am facing some persistent mismatch between what my experiments show and the calculated figures, particularly regarding the size of activations during the training’s forward pass. So, there is still something to figure out! in the notebook Acknowledgments Thanks to Stas Bekman for helping me shape my understanding and Quentin Anthony’s Python gist for VRAM calculation . Stas Bekman gist for VRAM calculation

Stories, News & Mentions about DEEPSPEED

Deepspeed’s Stories on HackerNoon

Deepspeed's News & Mentions