Our solution is to generate donations to cancer charities, around the world, from regular web searches that our users are performing

Sonmit

ChipNeMo: Domain-Adapted LLMs for Chip Design: Evaluations

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: মূল্যায়ন

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Bewertungen

ChipNeMo: LLM adaptados al dominio para diseño de chips: evaluaciones

ChipNeMo : LLM adaptés au domaine pour la conception de puces : évaluations

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: मूल्यांकन

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 評価

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 평가

ChipNeMo: LLMs adaptados ao domínio para design de chips: avaliações

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: оценки

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Değerlendirmeler

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Đánh giá

ChipNeMo：适用于芯片设计的领域适应性法学硕士 (LLM)：评估

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix II. DATASET A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix II. DATASET A. DAPT Dataset A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Chip Design Datasets: Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Public Datasets: Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. Data Blend: B. SFT Instruction Data B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Dataset

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: ডেটাসেট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Datensatz

ChipNeMo: LLM adaptados al dominio para diseño de chips: conjunto de datos

ChipNeMo : LLM adaptés au domaine pour la conception de puces : ensemble de données

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: डेटासेट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: データセット

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 데이터 세트

ChipNeMo: LLMs adaptados ao domínio para design de chips: conjunto de dados

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: набор данных

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Veri Kümesi

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Bộ dữ liệu

ChipNeMo：适用于芯片设计的领域适配法学硕士 (LLM)：数据集

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VI. DISCUSSION A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VI. DISCUSSION A. Considerations for Domain Adaptation A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Training and inference trade-off: • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Uniqueness of use case: • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • Availability of domain data: • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. • End use case diversity: B. Performance Gap B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 1) Data Collection: 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 2) Base Model: 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 3) Training: 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. 4) Retrieval: C. Agent-Based Design Methodologies C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Discussion

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: আলোচনা

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Diskussion

ChipNeMo: LLM adaptados al dominio para diseño de chips: discusión

ChipNeMo : LLM adaptés au domaine pour la conception de puces : discussion

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चर्चा

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ディスカッション

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 토론

ChipNeMo: LLMs adaptados ao domínio para design de chips: discussão

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: обсуждение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Tartışma

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Thảo luận

ChipNeMo：适用于芯片设计的领域法学硕士：讨论

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo. We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data;


ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo . We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. ChipNeMo Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. e5_small_unsupervised Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: ChipNemo Domain Adaptation Methods

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: চিপনেমো ডোমেন অ্যাডাপ্টেশন মেথড

ChipNeMo: Domänenangepasste LLMs für Chipdesign: ChipNemo-Domänenanpassungsmethoden

ChipNeMo: LLM adaptados al dominio para diseño de chips: métodos de adaptación de dominio de ChipNemo

ChipNeMo : LLM adaptés au domaine pour la conception de puces : méthodes d'adaptation de domaine ChipNemo

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चिपनेमो डोमेन अनुकूलन विधियां

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ChipNemo ドメイン適応法

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: ChipNemo 도메인 적응 방법

ChipNeMo: LLMs adaptados ao domínio para design de chips: Métodos de adaptação de domínio ChipNemo

ChipNeMo: Адаптированные к предметной области LLM для проектирования микросхем: Методы адаптации предметной области ChipNemo

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: ChipNemo Etki Alanı Uyarlama Yöntemleri

ChipNeMo: LLM thích ứng với miền cho thiết kế chip: Các phương pháp thích ứng miền ChipNemo

ChipNeMo：芯片设计领域适配法学硕士：ChipNemo 领域适配方法

ChipNeMo: Domain-Adapted LLMs for Chip Design: Appendix

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: পরিশিষ্ট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Anhang

ChipNeMo: LLM adaptados al dominio para diseño de chips: Apéndice

ChipNeMo : LLM adaptés au domaine pour la conception de puces : annexe

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: परिशिष्ट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 付録

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 부록

ChipNeMo: LLMs adaptados ao domínio para design de chips: Apêndice

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: Приложение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Ek

ChipNeMo: LLM thích ứng theo miền cho thiết kế chip: Phụ lục

ChipNeMo：适用于芯片设计的领域法学硕士：附录

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VIII. CONCLUSIONS The authors would like to thank: NVIDIA IT teams for their support on NVBugs integration; NVIDIA Hardware Security team for their support on security issues; NVIDIA NeMo teams for their support and guidance on training and inference of ChipNeMo models; NVIDIA Infrastructure teams for supporting the GPU training and inference resources for the project; NVIDIA Hardware design teams for their support and insight. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VIII. CONCLUSIONS The authors would like to thank: NVIDIA IT teams for their support on NVBugs integration; NVIDIA Hardware Security team for their support on security issues; NVIDIA NeMo teams for their support and guidance on training and inference of ChipNeMo models; NVIDIA Infrastructure teams for supporting the GPU training and inference resources for the project; NVIDIA Hardware design teams for their support and insight. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Conclusions

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: উপসংহার

ChipNeMo: Domänenangepasste LLMs für den Chipentwurf: Schlussfolgerungen

ChipNeMo: LLM adaptados al dominio para el diseño de chips: Conclusiones

ChipNeMo : LLM adaptés au domaine pour la conception de puces : conclusions

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: निष्कर्ष

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 結論

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 결론

ChipNeMo: LLMs adaptados ao domínio para design de chips: conclusões

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: выводы

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Sonuçlar

ChipNeMo: LLM thích ứng với tên miền cho thiết kế chip: Kết luận

ChipNeMo：适用于芯片设计的领域适应性法学硕士：结论

ChipNeMo: Domain-Adapted LLMs for Chip Design: Abstract and Intro

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: অ্যাবস্ট্রাক্ট এবং ইন্ট্রো

ChipNeMo: Fachangepasste LLMs für Chipdesign: Abstract und Einführung

ChipNeMo: LLM adaptados al dominio para diseño de chips: resumen e introducción

ChipNeMo : LLM adaptés au domaine pour la conception de puces : résumé et introduction

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: सार और परिचय

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 概要と概要

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 개요 및 소개

ChipNeMo: LLMs adaptados ao domínio para design de chips: resumo e introdução

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: Краткое содержание и введение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Özet ve Giriş

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Tóm tắt và giới thiệu

ChipNeMo：芯片设计领域适应性法学硕士：摘要和简介

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix IV. LLM APPLICATIONS We conducted a survey of potential LLM applications within our design teams and categorized them into four buckets: code generation, question & answer, analysis and reporting, and triage. Code generation refers to LLM generating design code, testbenches, assertions, internal tools scripts, etc.; Q & A refers to an LLM answering questions about designs, tools, infrastructures, etc.; Analysis and reporting refers to an LLM analyzing data and providing reports; triage refers to an LLM helping debug design or tool problems given logs and reports. We selected one key application from each category to study in this work, except for the triage category which we leave for further research. The motivation and technical details of each application are given below. A. Engineering Assistant Chatbot This application aims to help design engineers with answers to their architecture, design, verification, and build questions, which could significantly improve their overall productivity without impacting the productivity of others. It is observed that design engineers often enjoy brainstorming, designing hardware, and writing code, but can be slowed down waiting for answers on design knowledge they lack. Design productivity can also be enhanced by avoiding having engineers write code based on mistaken assumptions or debugging code that they are unfamiliar with. Internal studies have shown that up to 60% of a typical chip designer’s time is spent in debug or checklist related tasks across a range of topics including design specifications, testbench construction, architecture definition, and tools or infrastructure. Experts on these issues are often spread around the globe in a multinational company, such that it is not always convenient to find immediate help. Therefore, an engineering assistant chatbot based on knowledge extracted from internal design documents, code, any recorded data about designs and technical communications such as emails and corporate instant communications, etc. could help significantly improve design productivity. We implemented this application with the domain-adapted RAG method mentioned in Section III-D. B. EDA Script Generation Another common task in an industrial chip design flow is writing EDA scripts to accomplish a variety of tasks such as design implementation, introspection and transformation. These scripts often leverage both tool-specific and custom internal script libraries. Learning these libraries, navigating tool documentation, and writing and debugging these scripts, can take up a significant amount of engineering time. LLMs have proven adept at small scale code generation on a wide array of tasks [32] and therefore customizing these models to accelerate engineer productivity in this domain specific task is a natural fit. In this work we focus on generating two different types of scripts from natural language task descriptions. The first are scripts which leverage Tool1, an internal python library for design editing and analysis. The second are Tcl scripts that use the command interface provided by Tool2, which is a leading industrial static timing analysis tool. In order to build our domain-specific fine-tuning dataset for this task, production scripts for both tools were collected from design experts. We observed that our DAPT models can generate reasonable inline comments for the code. This enabled us to use these models to improve the quality of collected scripts by generating additional inline comments. Human experts later verified and corrected these comments and created an associated prompt. These prompts and code pairs make up the data used for DSFT in the format discussed in Section III-C. To provide and collect feedback in the most meaningful way, we spent significant effort building the flow shown in Fig. 4 where engineers can both query the model and run generated code through the same interface. This allows us to be confident in the correctness of generated code as well as provide accurate feedback by allowing engineers to see how many corrections they might need to get a functioning script. We support Tool1 and Tool2 integration by establishing interactive connections to tool servers. Additionally, we provide a user feedback form, allowing us to compare different models and glean valuable insights from user feedback. This valuable information can aid us in further refining our models. C. Bug Summarization and Analysis Tracking the reporting, triage, debug and resolution of various features and bugs across stages of the production flow is a time-consuming process. Engineering managers spend a lot of time reviewing internal issue tracking databases to build understanding of the state of the project and help speed their execution. Therefore, a tool that is able to look at all supporting information and quickly summarize both technical and managerial data as well as suggest next steps would boost team productivity. We focus on using LLMs to generate three different outputs - one focused on technical details, one on managerial details and one recommending task assignment. To study these tasks we used NVIDIA’s internal bug database, NVBugs. This database is used for bug reporting, tracking and resolution as well as general task and feature tracking across the company. We expect ChipNeMo models to perform well on this task as a large amount of bug data was included in the DAPT dataset. Additionally, we built a domain-specific SFT dataset for this task that includes examples of the bug summarizing and task assignment tasks. Often, bug descriptions contain large snippets of log files or code dumps along with long comment histories. In such cases, the bug text is too large for our LLM context windows. To work around this, we implemented two solutions. First, we found and replaced long path names with shorter aliases to allow the model to associate paths that occur in multiple places in the bug without needing to process the entire string. Second, we split the summarization task into an incremental task where the model is tasked with accumulating data across multiple summary and bug data chunks. We use a hierarchical approach where the bug is first separated into chunks that fit into the context window. Those chunks are then summarized and the summaries are accumulated then separated into chunks. This process is repeated until the entire set of summaries fits into a single context window and a single summary is generated. We use this same approach independent of the LLM used for summarization. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix IV. LLM APPLICATIONS We conducted a survey of potential LLM applications within our design teams and categorized them into four buckets: code generation, question & answer, analysis and reporting , and triage . Code generation refers to LLM generating design code, testbenches, assertions, internal tools scripts, etc.; Q & A refers to an LLM answering questions about designs, tools, infrastructures, etc.; Analysis and reporting refers to an LLM analyzing data and providing reports; triage refers to an LLM helping debug design or tool problems given logs and reports. We selected one key application from each category to study in this work, except for the triage category which we leave for further research. The motivation and technical details of each application are given below. code generation, question & answer, analysis and reporting triage triage A. Engineering Assistant Chatbot A. Engineering Assistant Chatbot This application aims to help design engineers with answers to their architecture, design, verification, and build questions, which could significantly improve their overall productivity without impacting the productivity of others. It is observed that design engineers often enjoy brainstorming, designing hardware, and writing code, but can be slowed down waiting for answers on design knowledge they lack. Design productivity can also be enhanced by avoiding having engineers write code based on mistaken assumptions or debugging code that they are unfamiliar with. Internal studies have shown that up to 60% of a typical chip designer’s time is spent in debug or checklist related tasks across a range of topics including design specifications, testbench construction, architecture definition, and tools or infrastructure. Experts on these issues are often spread around the globe in a multinational company, such that it is not always convenient to find immediate help. Therefore, an engineering assistant chatbot based on knowledge extracted from internal design documents, code, any recorded data about designs and technical communications such as emails and corporate instant communications, etc. could help significantly improve design productivity. We implemented this application with the domain-adapted RAG method mentioned in Section III-D. B. EDA Script Generation B. EDA Script Generation Another common task in an industrial chip design flow is writing EDA scripts to accomplish a variety of tasks such as design implementation, introspection and transformation. These scripts often leverage both tool-specific and custom internal script libraries. Learning these libraries, navigating tool documentation, and writing and debugging these scripts, can take up a significant amount of engineering time. LLMs have proven adept at small scale code generation on a wide array of tasks [32] and therefore customizing these models to accelerate engineer productivity in this domain specific task is a natural fit. In this work we focus on generating two different types of scripts from natural language task descriptions. The first are scripts which leverage Tool1, an internal python library for design editing and analysis. The second are Tcl scripts that use the command interface provided by Tool2, which is a leading industrial static timing analysis tool. In order to build our domain-specific fine-tuning dataset for this task, production scripts for both tools were collected from design experts. We observed that our DAPT models can generate reasonable inline comments for the code. This enabled us to use these models to improve the quality of collected scripts by generating additional inline comments. Human experts later verified and corrected these comments and created an associated prompt. These prompts and code pairs make up the data used for DSFT in the format discussed in Section III-C. To provide and collect feedback in the most meaningful way, we spent significant effort building the flow shown in Fig. 4 where engineers can both query the model and run generated code through the same interface. This allows us to be confident in the correctness of generated code as well as provide accurate feedback by allowing engineers to see how many corrections they might need to get a functioning script. We support Tool1 and Tool2 integration by establishing interactive connections to tool servers. correctness Additionally, we provide a user feedback form, allowing us to compare different models and glean valuable insights from user feedback. This valuable information can aid us in further refining our models. C. Bug Summarization and Analysis C. Bug Summarization and Analysis Tracking the reporting, triage, debug and resolution of various features and bugs across stages of the production flow is a time-consuming process. Engineering managers spend a lot of time reviewing internal issue tracking databases to build understanding of the state of the project and help speed their execution. Therefore, a tool that is able to look at all supporting information and quickly summarize both technical and managerial data as well as suggest next steps would boost team productivity. We focus on using LLMs to generate three different outputs - one focused on technical details, one on managerial details and one recommending task assignment. To study these tasks we used NVIDIA’s internal bug database, NVBugs. This database is used for bug reporting, tracking and resolution as well as general task and feature tracking across the company. We expect ChipNeMo models to perform well on this task as a large amount of bug data was included in the DAPT dataset. Additionally, we built a domain-specific SFT dataset for this task that includes examples of the bug summarizing and task assignment tasks. Often, bug descriptions contain large snippets of log files or code dumps along with long comment histories. In such cases, the bug text is too large for our LLM context windows. To work around this, we implemented two solutions. First, we found and replaced long path names with shorter aliases to allow the model to associate paths that occur in multiple places in the bug without needing to process the entire string. Second, we split the summarization task into an incremental task where the model is tasked with accumulating data across multiple summary and bug data chunks. We use a hierarchical approach where the bug is first separated into chunks that fit into the context window. Those chunks are then summarized and the summaries are accumulated then separated into chunks. This process is repeated until the entire set of summaries fits into a single context window and a single summary is generated. We use this same approach independent of the LLM used for summarization. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: LLM Applications

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: এলএলএম অ্যাপ্লিকেশন

ChipNeMo: Fachangepasste LLMs für Chipdesign: LLM-Anwendungen

ChipNeMo: LLM adaptados al dominio para diseño de chips: aplicaciones LLM

ChipNeMo : LLM adaptés au domaine pour la conception de puces : applications LLM

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: एलएलएम अनुप्रयोग

ChipNeMo: チップ設計のためのドメイン適応型 LLM: LLM アプリケーション

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: LLM 애플리케이션

ChipNeMo: LLMs adaptados ao domínio para design de chips: aplicações LLM

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: приложения LLM

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Yüksek Lisans Uygulamaları

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Ứng dụng LLM

ChipNeMo：适用于芯片设计的领域法学硕士：法学硕士应用

SONMIT

sonmit.com

-- employees

Since n.d.

COMPANY RANKING

SONMIT

EVERGREEN INDEX #9977

Representing this company? Click here to claim and customize this page!

Sonmit WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SONMIT

sonmit.com

-- employees

Since n.d.

COMPANY RANKING

SONMIT

EVERGREEN INDEX #9977

RELATED COMPANIESRANK

STORIES ABOUT SONMIT

Representing this company? Click here to claim and customize this page!

Sonmit WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!