WHAT DO WE DO Bohni is an ad-tech platform offering thousands of verified wrapped vehicles for displaying brand advertisements in every area across mumbai. With telematics vehicle tracking, impression data, and live fleet analytics, brands will have all the information to measure the campaign performance. Our Mission MAKING ADVERTISING "ACCESSIBLE ACCOUNTABLE AFFORDABLE"

Bohni

Throughout the vegetable and animal worlds the sexual functions are periodic

Stipe Pleic, Bosnia's 'first Viking,' has followed in Ragnar's footsteps, making weapons by hand, decorating the walls of his house with swords and shields, and dressing like the character. In this thread, our community discusses this case and whether or not they'll go total immersion in a characters' lifestyle. This Slogging thread by Mónica Freitas, Sara Pinto, Jack Boreham and Limarc Ambalina occurred in slogging's official #random channel, and has been edited for readability. Mónica Freitas Feb 1, 2022, 5:25 PM A 21st century Ragnar? https://www.euronews.com/culture/2022/01/31/meet-the-bosnian-viking-living-as-his-favourite-fictional-character Mónica Freitas Feb 1, 2022, 5:26 PM "With his thick grey beard, plaited hair and an arsenal of medieval weapons, Bosnia's 'first Viking' relaxes by hurling axes in the attic of his stone cottage. Over the past five years, Stipe Pleic has undergone a steady transformation from builder to Nordic warrior in homage to the popular 'Vikings'. The metamorphosis began after Pleic returned home to Tomislavgrad in southern Bosnia after years working on construction sites in Germany." https://www.euronews.com/culture/tag/television Mónica Freitas Feb 1, 2022, 5:32 PM Stipe Pleic has followed in Ragnar's footsteps, making weapons by hand (from axes, spears, bows to shields, with some pieces sold for up to €300), decorating the walls of his house with swords and shields, and dressing as the character (tunic and matching trousers). He even replicated a Drakkar (a flat-bottomed boat capable of crossing oceans)! Mónica Freitas Feb 1, 2022, 5:33 PM Jack Boreham, Limarc Ambalina, Abeer, Sara Pinto, Blake Cram, Linh Smooke, Daniel Guzman what do you guys think of this? Would you ever go full emersion on a character? Which character would you choose? Sara Pinto Feb 2, 2022, 1:00 PM Mónica Freitas, oh wow, that takes commitment! And money! I don't think I could do it. I would miss some things about the current world. But if I was to do it, I would choose something like Marie Antoinette in the movie, before the tragic ending, of course, haha. There's something about the corsets and the big hair that I like. What about you? Jack Boreham Feb 2, 2022, 1:52 PM I would, I would live out my last days as Darth Vader, in a cape and everything 😂hahah Limarc Ambalina Feb 2, 2022, 4:42 PM This guy is my Hero. I don’t think we can call him a Viking. Limarc Ambalina Feb 2, 2022, 4:42 PM I think for me to say this is a real-life Viking, he needs to survive off the land and by pillaging (stealing), lol. 😂 3 Mónica Freitas Feb 3, 2022, 3:46 PM Sara Pinto, corsets are great, but they prevent you from breathing, at least the ones Marie Antoinette used. I'd like to live as a medieval princess or a greek oracle. And yes, I'd do it all for the garments and the mystery, haha! Mónica Freitas Feb 3, 2022, 3:47 PM Jack Boreham, halloween costume unlocked! I can imagine you trying to use the force on the remote control, haha. Mónica Freitas Feb 3, 2022, 3:49 PM Limarc Ambalina, maybe that's going to be his next step 😂 What/who would you be? Limarc Ambalina Feb 4, 2022, 2:13 AM Vikings (the show) I think is super underrated, and I really love this guy just for his passion for it. I don't think I'd ever want to become a Viking though, their lifespans were too short (at least I assume so when most of your life is raiding and pillaging). 🔥 1 Limarc Ambalina Feb 4, 2022, 2:14 AM "And in his garden, he has built a replica of a Drakkar – a flat-bottomed boat capable of crossing oceans – which he occasionally sails on a nearby lake." I think with the hype this guy has drawn, he could make a killing giving tours and selling his stuff on Etsy 😄 Limarc Ambalina Feb 4, 2022, 2:14 AM Sara Pinto Feb 4, 2022, 9:07 PM Mónica Freitas, they may not let you breathe properly, but boy do they look stylish haha. Great choice! The garments are always worth it, haha! Mónica Freitas Feb 4, 2022, 9:22 PM Limarc Ambalina, yup! I can't get over Lagertha. I still have to check out the latest seasons. Unfortunately, Netflix only has until the 4th one. Mónica Freitas Feb 4, 2022, 9:22 PM Limarc Ambalina, don't give him any ideas. I can see him opening a Viking-inspired hostel someday. Maybe this is worth a pitch! Mónica Freitas Feb 4, 2022, 9:23 PM Sara Pinto, the things we do fashion 😂 if only we could travel back in time! I'd love to see the Greek temples in their prime years... Sara Pinto Feb 7, 2022, 11:25 AM Mónica Freitas, the worst part is that we may change what we wear and what we use, but the time won't go back for us to have the full experience. :smiling_face_with_tear: 1 Mónica Freitas Feb 7, 2022, 4:04 PM Sara Pinto, that's the real downside. Though I'm not sure I'd like to have the full women experience of the past. Not keen on having my rights stripped away. Sara Pinto Feb 8, 2022, 12:01 PM Mónica Freitas, that's true. Maybe sticking with the garments and some parts of the lifestyle is not that bad haha. This guy can be really on to something. Mónica Freitas Feb 8, 2022, 1:50 PM Sara Pinto, I'm curious to know if there are more people following in his footsteps. Many people still do LARP, so maybe it's not too far-fetched to think there are more Stipe Pleic's out there. 🔥 1

Bosnian Man Chooses to Live as a 21st Century Ragnar

Film Trailer Generation via Task Decomposition: Conclusions and References

টাস্ক ডিকম্পোজিশনের মাধ্যমে ফিল্ম ট্রেলার জেনারেশন: উপসংহার এবং রেফারেন্স

Filmtrailer-Generierung durch Aufgabenzerlegung: Schlussfolgerungen und Referenzen

Generación de avances de películas mediante descomposición de tareas: conclusiones y referencias

Génération de bandes-annonces de films via la décomposition de tâches : conclusions et références

टास्क डिकंपोज़िशन के माध्यम से फ़िल्म ट्रेलर निर्माण: निष्कर्ष और संदर्भ

タスク分解による映画予告編生成: 結論と参考文献

작업 분해를 통한 영화 예고편 생성: 결론 및 참고 자료

Geração de trailer de filme por meio de decomposição de tarefas: conclusões e referências

Генерация трейлеров к фильму посредством декомпозиции задач: выводы и ссылки

Görev Ayrıştırma Yoluyla Film Fragmanı Oluşturma: Sonuçlar ve Referanslar

Tạo đoạn giới thiệu phim thông qua phân tách nhiệm vụ: Kết luận và tài liệu tham khảo

通过任务分解生成电影预告片：结论与参考文献

ChipNeMo: Domain-Adapted LLMs for Chip Design: Evaluations

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: মূল্যায়ন

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Bewertungen

ChipNeMo: LLM adaptados al dominio para diseño de chips: evaluaciones

ChipNeMo : LLM adaptés au domaine pour la conception de puces : évaluations

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: मूल्यांकन

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 評価

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 평가

ChipNeMo: LLMs adaptados ao domínio para design de chips: avaliações

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: оценки

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Değerlendirmeler

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Đánh giá

ChipNeMo：适用于芯片设计的领域适应性法学硕士 (LLM)：评估

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix II. DATASET A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix II. DATASET A. DAPT Dataset A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Chip Design Datasets: Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Public Datasets: Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. Data Blend: B. SFT Instruction Data B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Dataset

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: ডেটাসেট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Datensatz

ChipNeMo: LLM adaptados al dominio para diseño de chips: conjunto de datos

ChipNeMo : LLM adaptés au domaine pour la conception de puces : ensemble de données

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: डेटासेट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: データセット

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 데이터 세트

ChipNeMo: LLMs adaptados ao domínio para design de chips: conjunto de dados

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: набор данных

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Veri Kümesi

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Bộ dữ liệu

ChipNeMo：适用于芯片设计的领域适配法学硕士 (LLM)：数据集

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VI. DISCUSSION A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VI. DISCUSSION A. Considerations for Domain Adaptation A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Training and inference trade-off: • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Uniqueness of use case: • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • Availability of domain data: • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. • End use case diversity: B. Performance Gap B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 1) Data Collection: 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 2) Base Model: 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 3) Training: 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. 4) Retrieval: C. Agent-Based Design Methodologies C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Discussion

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: আলোচনা

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Diskussion

ChipNeMo: LLM adaptados al dominio para diseño de chips: discusión

ChipNeMo : LLM adaptés au domaine pour la conception de puces : discussion

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चर्चा

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ディスカッション

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 토론

ChipNeMo: LLMs adaptados ao domínio para design de chips: discussão

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: обсуждение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Tartışma

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Thảo luận

ChipNeMo：适用于芯片设计的领域法学硕士：讨论

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo. We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data;


ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo . We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. ChipNeMo Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. e5_small_unsupervised Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: ChipNemo Domain Adaptation Methods

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: চিপনেমো ডোমেন অ্যাডাপ্টেশন মেথড

ChipNeMo: Domänenangepasste LLMs für Chipdesign: ChipNemo-Domänenanpassungsmethoden

ChipNeMo: LLM adaptados al dominio para diseño de chips: métodos de adaptación de dominio de ChipNemo

ChipNeMo : LLM adaptés au domaine pour la conception de puces : méthodes d'adaptation de domaine ChipNemo

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चिपनेमो डोमेन अनुकूलन विधियां

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ChipNemo ドメイン適応法

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: ChipNemo 도메인 적응 방법

ChipNeMo: LLMs adaptados ao domínio para design de chips: Métodos de adaptação de domínio ChipNemo

ChipNeMo: Адаптированные к предметной области LLM для проектирования микросхем: Методы адаптации предметной области ChipNemo

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: ChipNemo Etki Alanı Uyarlama Yöntemleri

ChipNeMo: LLM thích ứng với miền cho thiết kế chip: Các phương pháp thích ứng miền ChipNemo

ChipNeMo：芯片设计领域适配法学硕士：ChipNemo 领域适配方法

ChipNeMo: Domain-Adapted LLMs for Chip Design: Appendix

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: পরিশিষ্ট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Anhang

ChipNeMo: LLM adaptados al dominio para diseño de chips: Apéndice

ChipNeMo : LLM adaptés au domaine pour la conception de puces : annexe

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: परिशिष्ट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 付録

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 부록

ChipNeMo: LLMs adaptados ao domínio para design de chips: Apêndice

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: Приложение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Ek

ChipNeMo: LLM thích ứng theo miền cho thiết kế chip: Phụ lục

ChipNeMo：适用于芯片设计的领域法学硕士：附录

Bohni

bohni.io

11-50 employees

Since 2021

COMPANY RANKING

BOHNI

EVERGREEN INDEX #2219

Bohni WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Bohni

bohni.io

11-50 employees

Since 2021

COMPANY RANKING

BOHNI

EVERGREEN INDEX #2219

RELATED COMPANIESRANK

STORIES ABOUT BOHNI

Bohni WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!