WHAT DO WE DO Bohni is an ad-tech platform offering thousands of verified wrapped vehicles for displaying brand advertisements in every area across mumbai. With telematics vehicle tracking, impression data, and live fleet analytics, brands will have all the information to measure the campaign performance. Our Mission MAKING ADVERTISING "ACCESSIBLE ACCOUNTABLE AFFORDABLE"

Bohni

bohni

Throughout the vegetable and animal worlds the sexual functions are periodic

Stipe Pleic, Bosnia's 'first Viking,' has followed in Ragnar's footsteps, making weapons by hand, decorating the walls of his house with swords and shields, and dressing like the character. In this thread, our community discusses this case and whether or not they'll go total immersion in a characters' lifestyle. This Slogging thread by Mónica Freitas, Sara Pinto, Jack Boreham and Limarc Ambalina occurred in slogging's official #random channel, and has been edited for readability. Mónica Freitas Feb 1, 2022, 5:25 PM A 21st century Ragnar? https://www.euronews.com/culture/2022/01/31/meet-the-bosnian-viking-living-as-his-favourite-fictional-character Mónica Freitas Feb 1, 2022, 5:26 PM "With his thick grey beard, plaited hair and an arsenal of medieval weapons, Bosnia's 'first Viking' relaxes by hurling axes in the attic of his stone cottage. Over the past five years, Stipe Pleic has undergone a steady transformation from builder to Nordic warrior in homage to the popular 'Vikings'. The metamorphosis began after Pleic returned home to Tomislavgrad in southern Bosnia after years working on construction sites in Germany." https://www.euronews.com/culture/tag/television Mónica Freitas Feb 1, 2022, 5:32 PM Stipe Pleic has followed in Ragnar's footsteps, making weapons by hand (from axes, spears, bows to shields, with some pieces sold for up to €300), decorating the walls of his house with swords and shields, and dressing as the character (tunic and matching trousers). He even replicated a Drakkar (a flat-bottomed boat capable of crossing oceans)! Mónica Freitas Feb 1, 2022, 5:33 PM Jack Boreham, Limarc Ambalina, Abeer, Sara Pinto, Blake Cram, Linh Smooke, Daniel Guzman what do you guys think of this? Would you ever go full emersion on a character? Which character would you choose? Sara Pinto Feb 2, 2022, 1:00 PM Mónica Freitas, oh wow, that takes commitment! And money! I don't think I could do it. I would miss some things about the current world. But if I was to do it, I would choose something like Marie Antoinette in the movie, before the tragic ending, of course, haha. There's something about the corsets and the big hair that I like. What about you? Jack Boreham Feb 2, 2022, 1:52 PM I would, I would live out my last days as Darth Vader, in a cape and everything 😂hahah Limarc Ambalina Feb 2, 2022, 4:42 PM This guy is my Hero. I don’t think we can call him a Viking. Limarc Ambalina Feb 2, 2022, 4:42 PM I think for me to say this is a real-life Viking, he needs to survive off the land and by pillaging (stealing), lol. 😂 3 Mónica Freitas Feb 3, 2022, 3:46 PM Sara Pinto, corsets are great, but they prevent you from breathing, at least the ones Marie Antoinette used. I'd like to live as a medieval princess or a greek oracle. And yes, I'd do it all for the garments and the mystery, haha! Mónica Freitas Feb 3, 2022, 3:47 PM Jack Boreham, halloween costume unlocked! I can imagine you trying to use the force on the remote control, haha. Mónica Freitas Feb 3, 2022, 3:49 PM Limarc Ambalina, maybe that's going to be his next step 😂 What/who would you be? Limarc Ambalina Feb 4, 2022, 2:13 AM Vikings (the show) I think is super underrated, and I really love this guy just for his passion for it. I don't think I'd ever want to become a Viking though, their lifespans were too short (at least I assume so when most of your life is raiding and pillaging). 🔥 1 Limarc Ambalina Feb 4, 2022, 2:14 AM "And in his garden, he has built a replica of a Drakkar – a flat-bottomed boat capable of crossing oceans – which he occasionally sails on a nearby lake." I think with the hype this guy has drawn, he could make a killing giving tours and selling his stuff on Etsy 😄 Limarc Ambalina Feb 4, 2022, 2:14 AM Sara Pinto Feb 4, 2022, 9:07 PM Mónica Freitas, they may not let you breathe properly, but boy do they look stylish haha. Great choice! The garments are always worth it, haha! Mónica Freitas Feb 4, 2022, 9:22 PM Limarc Ambalina, yup! I can't get over Lagertha. I still have to check out the latest seasons. Unfortunately, Netflix only has until the 4th one. Mónica Freitas Feb 4, 2022, 9:22 PM Limarc Ambalina, don't give him any ideas. I can see him opening a Viking-inspired hostel someday. Maybe this is worth a pitch! Mónica Freitas Feb 4, 2022, 9:23 PM Sara Pinto, the things we do fashion 😂 if only we could travel back in time! I'd love to see the Greek temples in their prime years... Sara Pinto Feb 7, 2022, 11:25 AM Mónica Freitas, the worst part is that we may change what we wear and what we use, but the time won't go back for us to have the full experience. :smiling_face_with_tear: 1 Mónica Freitas Feb 7, 2022, 4:04 PM Sara Pinto, that's the real downside. Though I'm not sure I'd like to have the full women experience of the past. Not keen on having my rights stripped away. Sara Pinto Feb 8, 2022, 12:01 PM Mónica Freitas, that's true. Maybe sticking with the garments and some parts of the lifestyle is not that bad haha. This guy can be really on to something. Mónica Freitas Feb 8, 2022, 1:50 PM Sara Pinto, I'm curious to know if there are more people following in his footsteps. Many people still do LARP, so maybe it's not too far-fetched to think there are more Stipe Pleic's out there. 🔥 1

Bosnian Man Chooses to Live as a 21st Century Ragnar

Film Trailer Generation via Task Decomposition: Conclusions and References

টাস্ক ডিকম্পোজিশনের মাধ্যমে ফিল্ম ট্রেলার জেনারেশন: উপসংহার এবং রেফারেন্স

Filmtrailer-Generierung durch Aufgabenzerlegung: Schlussfolgerungen und Referenzen

Generación de avances de películas mediante descomposición de tareas: conclusiones y referencias

Génération de bandes-annonces de films via la décomposition de tâches : conclusions et références

टास्क डिकंपोज़िशन के माध्यम से फ़िल्म ट्रेलर निर्माण: निष्कर्ष और संदर्भ

タスク分解による映画予告編生成: 結論と参考文献

작업 분해를 통한 영화 예고편 생성: 결론 및 참고 자료

Geração de trailer de filme por meio de decomposição de tarefas: conclusões e referências

Генерация трейлеров к фильму посредством декомпозиции задач: выводы и ссылки

Görev Ayrıştırma Yoluyla Film Fragmanı Oluşturma: Sonuçlar ve Referanslar

Tạo đoạn giới thiệu phim thông qua phân tách nhiệm vụ: Kết luận và tài liệu tham khảo

通过任务分解生成电影预告片：结论与参考文献

ChipNeMo: Domain-Adapted LLMs for Chip Design: Evaluations

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: মূল্যায়ন

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Bewertungen

ChipNeMo: LLM adaptados al dominio para diseño de chips: evaluaciones

ChipNeMo : LLM adaptés au domaine pour la conception de puces : évaluations

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: मूल्यांकन

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 評価

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 평가

ChipNeMo: LLMs adaptados ao domínio para design de chips: avaliações

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: оценки

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Değerlendirmeler

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Đánh giá

ChipNeMo：适用于芯片设计的领域适应性法学硕士 (LLM)：评估

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix II. DATASET A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix II. DATASET A. DAPT Dataset A. DAPT Dataset During Domain-Adaptive Pre-Training (DAPT), we assemble a dataset from a combination of NVIDIA-proprietary chip design specific data sources and publicly available datasets. Chip Design Datasets: Our internal dataset consists of a diverse range of text sources pertinent to chip design, spanning design, verification, infrastructure, and internal documentation. Table I provides a breakdown of the data collected after filtering, and the corresponding number of tokens using the LLaMA2 tokenizer. We construct the dataset by gathering all relevant internal data, then filtering by file type, based on filename extensions and distinguishing between machine-generated and human-written content. Although we evaluated on three specific use cases, we did not specifically limit the dataset to sources known to be relevant to these use cases since we believed that incorporating additional domain knowledge would improve performance. After collection, cleaning, and filtering, the internal data training corpus has 23.1 billion tokens. Further details of the data collection process are covered in Appendix A. Chip Design Datasets: Public Datasets: We augment the chip design specific data with a sample of publicly available data from various sources, a common practice in the development of foundational large language models. Our approach was to reuse public training data from other language models, with the stipulation that it must be publicly accessible and compatible with open sourcing. These datasets exhibit a high degree of correlation with the pretraining data used in LLaMA2 [5], with the intention of preserving general knowledge and natural language capabilities during DAPT. The public datasets used by ChipNeMo can be categorized into two groups, natural language and code. For the natural language component, we draw from Wikipedia data [17], as it is widely regarded for its high data quality. For code, we leverage GitHub data [18], focusing on programming languages also present in our internal data chip design dataset such as C++, Python, and Verilog. To ensure that the overall dataset is representative of pre-training distributions, we perform a subsampling operation that results in approximately 9.2% of the total training tokens being sampled from these public datasets, with a balanced representation of natural language and code. Public Datasets: Data Blend: A significant proportion of the domain data we gathered is comprised of unannotated code from diverse origins. In an effort to enhance the model’s comprehension of domain-specific knowledge, we conducted downsampling of code data while concurrently upsampling natural language data, specifically design documentation, over a span of 2 to 4 training epochs. We also increased the representation of data that we deemed more pertinent to downstream applications, such as human-written EDA tool scripts. Furthermore, we incorporated publicly available domain data for 1 epoch. Details of the token distribution for training are shown in Table I. Data Blend: B. SFT Instruction Data B. SFT Instruction Data During Supervised Fine-Tuning (SFT), we employ a general chat SFT instruction dataset that is accessible for commercial use. The dataset is comprised largely of publicly available instruction following datasets including OASST [19], FLAN [20], P3 [21] and a small amount of a broad domain proprietary dataset comprising various topics such as brainstorming, open-ended question answering, rewriting, summarization etc. It’s important to note that the SFT instruction data we discuss here is focused on general natural language tasks and does not contain any information or tasks related to the downstream use cases in chip design. In total, this dataset comprises 128,000 training samples. Additionally, we meticulously assembled a domain-specific instruction dataset for aligning the model to downstream use cases. These examples have been meticulously crafted by subject matter experts and are formatted as single-turn questions and answers. Table II depicts the quantity of our domainspecific instruction dataset. It’s worth noting that the total number of training samples in the domain-specific instruction dataset is quite small when compared to the extensive amount of generative chat instruction data. C. AutoEval C. AutoEval In order to quickly and quantitatively assess the accuracy of various models, we established evaluation criteria structured as multiple-choice question-and-answer formats for each use case, designed to closely align with established benchmarks, such as MMLU [22]. In the process of formulating these multiplechoice questions, collaboration with domain experts was pivotal. The goal was to ensure that each question included at least one complex answer choice, thereby posing a challenge to individuals with limited domain expertise. Careful attention was also given to prevent any inadvertent contamination of the questions with data from our domain-specific SFT. In addition to the per-use-case benchmarks, an additional benchmark was created for general circuit design knowledge, covering both analog and digital design topics. The number of multiple-choice questions for evaluation benchmark are shown in Table III. When we report results on the above benchmarks, we take average results obtained from five distinct runs to mitigate the effects of variance and noise in the testing process. Each iteration employs a set of 5-shot examples, with variations introduced across each individual runs. In addition to these domain-specific evaluation benchmarks, we also include commonly-used publicly available LLM academic benchmarks. Furthermore, we measure the model’s code generation capabilities, by evaluating HumanEval [23] for Python and VerilogEval [12] for Verilog. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Dataset

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: ডেটাসেট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Datensatz

ChipNeMo: LLM adaptados al dominio para diseño de chips: conjunto de datos

ChipNeMo : LLM adaptés au domaine pour la conception de puces : ensemble de données

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: डेटासेट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: データセット

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 데이터 세트

ChipNeMo: LLMs adaptados ao domínio para design de chips: conjunto de dados

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: набор данных

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Veri Kümesi

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Bộ dữ liệu

ChipNeMo：适用于芯片设计的领域适配法学硕士 (LLM)：数据集

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VI. DISCUSSION A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VI. DISCUSSION A. Considerations for Domain Adaptation A. Considerations for Domain Adaptation Although domain-adapted ChipNeMo models achieve significant improvements over their corresponding foundation models, we also observe that the larger LLaMA2 70B can sometimes achieve similar accuracy as ChipNeMo, as seen in Figures 8, 9, and 10. Recent work has leveraged these powerful models to perform chip design tasks. However, it is important to consider the cost-efficiency benefits gained from the use of a smaller model. Pope et al. demonstrate that inference costs on an 8B model are 8- 12x lower than on a 62B model for equal latency targets [34]. Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [35]. Our ChipNeMo 13B model can be loaded within the memory of a single A100 GPU without any quantization, unlike the LLaMA2 70B model. This leads to significant inference speed increases under normal GPU operation, which can be traded off for significant inference cost reduction should the GPU be underclocked. Thus, when deciding between the use of a larger general-purpose model versus a smaller specialized model in a production environment the following criteria must be considered: • Training and inference trade-off: Smaller domain adapted models can match the accuracy of larger general purpose models. While domain adaptation incurs additional up-front costs, the use of smaller models leads to significantly reduced operating costs. • Training and inference trade-off: • Uniqueness of use case: As can be seen from Figures 6, 9, and 10, domain adapted models show the most improvement on tasks that are rarely present in the public domain, such as writing code in proprietary languages or libraries. Indeed, our data shows that even when they are provided with hand-picked contexts, large general-purpose models have difficulty matching the accuracy of domain adapted models in such scenarios. • Uniqueness of use case: • Availability of domain data: Domain adaption works best when there is large amount of training data, i.e. billions of training tokens. This is often the case for large corporations and projects which have accumulated a large amount of internal documents and code, but not necessarily true for smaller businesses or projects. • Availability of domain data: • End use case diversity: It is possible to fine-tune a general-purpose model for a particular task, but domain adapted models are suited for a diverse set of tasks in a domain. Although we only demonstrate three use cases for ChipNeMo models in this work, it can be readily re-used for other use cases with sufficient SFT data. • End use case diversity: B. Performance Gap B. Performance Gap Although ChipNeMo achieves impressive results in our selected applications as shown in Appendix E, the evaluation results for all applications still show a considerate gap with human expert performance. We are considering the following approaches to bridge this performance gap: 1) Data Collection: We can expand the DAPT dataset to include more internal proprietary data. In addition, we plan to add more task specific instruction sets for SFT as evidence shown task specific SFT improves the evaluation results meaningfully. 1) Data Collection: 2) Base Model: We expect better and larger base models can improve performance, such as LLaMA2 70B. We can also explore applying DAPT to code-specific base models such as Code LLaMA [32] for code generation tasks. 2) Base Model: 3) Training: We also plan to conduct reinforcement learning from human feedback (RLHF) [36] over the ChipNeMo chat model to make it more versatile. We plan to leverage pretrained reward models trained over general purpose datasets. We also plan to conduct long-context training [37] to overcome the challenge where long context is needed, e.g. in the bug summarization application. In general, longer context support would help improve retrieval-based methods for chat assistance as well as code generation. 3) Training: 4) Retrieval: We will further investigate better RAG methods for both the engineering assistant chatbot and EDA script generation. For the engineering assistant chatbot, we can create different data stores for different application areas. We can also integrate enterprise search engines with RAG to find relevant context for a diverse set of problems. For code generation, we can investigate automated retrieval of context from existing code and documentation. 4) Retrieval: C. Agent-Based Design Methodologies C. Agent-Based Design Methodologies The use cases we experimented in this work are straightforward applications of the prompt and response capability of LLMs. Agents refer to the use of an LLM to choose a sequence of actions to take, where an LLM is acting as a reasoning engine to drive outside tools. Chip design processes involve many existing EDA tools and methodologies. We believe some of these methodologies can be driven by agents powered by domain-adapted LLMs such as ChipNeMo models. We plan to work on agent-based design methodologies for verification and optimization in the future. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Discussion

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: আলোচনা

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Diskussion

ChipNeMo: LLM adaptados al dominio para diseño de chips: discusión

ChipNeMo : LLM adaptés au domaine pour la conception de puces : discussion

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चर्चा

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ディスカッション

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 토론

ChipNeMo: LLMs adaptados ao domínio para design de chips: discussão

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: обсуждение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Tartışma

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Thảo luận

ChipNeMo：适用于芯片设计的领域法学硕士：讨论

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo. We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data;


ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix III. CHIPNEMO DOMAIN ADAPTATION METHODS ChipNeMo implements multiple domain adaptation techniques to adapt LLMs to the chip design domain. These techniques include custom tokenizers for chip design data, domain adaptive pretraining with large corpus of domain data, supervised-fine-tuning with domain specific tasks, and retrievalaugmented generation with a fine-tuned retrieval model. We will illustrate the details of each technique in this section. A. Tokenizer A. Tokenizer When adapting a pre-trained tokenizer, the main goals are to improve tokenization efficiency on domain-specific data, maintain efficiency and language model performance on general datasets, and minimize the effort for retraining/fine-tuning. To achieve this, we’ve developed a four-step approach: • Step 1: Training a tokenizer from scratch using domain specific data. • Step 2: From the vocabulary of the new tokenizer, identifying tokens that are absent in the general-purpose tokenizer and are rarely found in general-purpose datasets. • Step 3: Expanding the general-purpose tokenizer with the newly identified tokens at Step 2. • Step 4: Initializing the embeddings of the new tokens by utilizing the general-purpose tokenizer. Specifically for Step 4, when a new token is encountered, it is tokenized using the pretrained general-purpose tokenizer. The embedding of the new token is determined by averaging the embeddings of the tokens generated by the general-purpose tokenizer [24], and the output layer weights initialized to zero. Step 2 helps maintain the performance of the pre-trained LLM on general datasets by selectively introducing new tokens that are infrequently encountered in general-purpose datasets. And Step 4 reduces the effort required for retraining/finetuning the LLM via initialization of the embeddings of new tokens guided by the general-purpose tokenizer. B. Domain Adaptive Pretraining B. Domain Adaptive Pretraining In our study, we apply DAPT on pretrained foundation base models LLaMA2 7B/13B. Each DAPT model is initialized using the weights of their corresponding pretrained foundational base models. We name our DAPT models ChipNeMo . We employ tokenizer augmentation as depicted in Section III-A and initialize embedding weight accordingly [24]. We conduct further pretraining on domain-specific data by employing the standard autoregressive language modeling objective. All model training procedures are conducted using the NVIDIA NeMo framework [25], incorporating techniques such as tensor parallelism [26] and flash attention [27] for enhanced efficiency. ChipNeMo Figure 2 illustrates the training loss of ChipNeMo under the specified hyperparameters. We do observe spikes in the training loss. In contrast to the hypothesis in [28], we postulate that in our scenario, these spikes can be attributed to “bad data” since these irregularities seem to consistently occur in similar training steps for the same model, even across different model sizes. We chose not to address this issue, as these anomalies did not appear to significantly impede subsequent training steps (with no noticeable degradation in validation loss), possibly due to our application of a low learning rate. C. Supervised Fine-Tuning C. Supervised Fine-Tuning After DAPT, we perform model alignment with supervised fine-tuning (SFT). We adopt the identical hyperparameter training configuration as DAPT for all models, with the exception of using a reduced global batch size of 128. All SFT data is structured according to the chat template below: <extra_id_0>system\n{system} <extra_id_1>User\n{user_utterance} <extra_id_1>Assistant\n{chipnemo_response} … We employ an autoregressive optimization objective, implementing a strategy where losses associated with tokens originating from the system and user prompts are masked [5]. This approach ensures that during backpropagation, our focus is exclusively directed towards the optimization of answer tokens. We combine our domain SFT dataset, comprising approximately 1.1k samples, with the more extensive general chat SFT dataset of 128k samples. We then engaged in fine-tuning for a single epoch after applying a random shuffle to the data. We conducted experiments involving augmentation of the domain-specific SFT dataset for more than one epoch. However, it became evident that the model rapidly exhibited signs of overfitting when presented with in-domain questions, often repeating irrelevant answers from the domain SFT dataset. Additionally, we conducted an additional SFT using solely the general chat dataset, excluding any domain-specific SFT data. For clarity, we designate all our ChipNeMo models as follows: ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: Models fine-tuned with both domain and general chat data; ChipNeMo-Chat: ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): Models fine-tuned with general chat data exclusively. ChipNeMo-Chat (noDSFT): We also experimented with DAPT directly on a chat aligned model, such as the LLaMA2-Chat model. We found that DAPT significantly degraded the model’s alignment, making the resulting model useless for downstream tasks. D. Retrieval-Augmented Generation D. Retrieval-Augmented Generation It is well known that LLMs can generate inaccurate text, so-called hallucination [29]. Although the phenomenon is not completely understood, we still must mitigate hallucinations since they are particularly problematic in an engineering assistant chatbot context, where accuracy is critical. Our proposal is to leverage the retrieval augmented generation (RAG) method. RAG tries to retrieve relevant passages from a database to be included in the prompt together with the question, which grounds the LLM to produce more accurate answers. We find that using a domain adapted language model for RAG significantly improves answer quality on our domain specific questions. Also, we find that fine-tuning an off-the-shelf unsupervised pre-trained dense retrieval model with a modest amount of domain specific training data significantly improves retrieval accuracy. Our domain-adapted RAG implementation diagram is illustrated on Figure 3. We created our domain adapted retrieval model by fine-tuning the e5_small_unsupervised model [30] with 3000 domain specific auto-generated samples using the Tevatron framework [31]. The sample generation and training process are covered in Appendix C. e5_small_unsupervised Even with the significant gains that come with fine-tuning a retrieval model, the fact remains that retrieval still struggles with queries that do not map directly to passages in the document corpus or require more context not present in the passage. Unfortunately, these queries are also more representative of queries that will be asked by engineers in real situations. Combining retrieval with a domain adapted language model is one way to address this issue. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: ChipNemo Domain Adaptation Methods

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: চিপনেমো ডোমেন অ্যাডাপ্টেশন মেথড

ChipNeMo: Domänenangepasste LLMs für Chipdesign: ChipNemo-Domänenanpassungsmethoden

ChipNeMo: LLM adaptados al dominio para diseño de chips: métodos de adaptación de dominio de ChipNemo

ChipNeMo : LLM adaptés au domaine pour la conception de puces : méthodes d'adaptation de domaine ChipNemo

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: चिपनेमो डोमेन अनुकूलन विधियां

ChipNeMo: チップ設計のためのドメイン適応型 LLM: ChipNemo ドメイン適応法

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: ChipNemo 도메인 적응 방법

ChipNeMo: LLMs adaptados ao domínio para design de chips: Métodos de adaptação de domínio ChipNemo

ChipNeMo: Адаптированные к предметной области LLM для проектирования микросхем: Методы адаптации предметной области ChipNemo

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: ChipNemo Etki Alanı Uyarlama Yöntemleri

ChipNeMo: LLM thích ứng với miền cho thiết kế chip: Các phương pháp thích ứng miền ChipNemo

ChipNeMo：芯片设计领域适配法学硕士：ChipNemo 领域适配方法

ChipNeMo: Domain-Adapted LLMs for Chip Design: Appendix

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: পরিশিষ্ট

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Anhang

ChipNeMo: LLM adaptados al dominio para diseño de chips: Apéndice

ChipNeMo : LLM adaptés au domaine pour la conception de puces : annexe

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: परिशिष्ट

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 付録

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 부록

ChipNeMo: LLMs adaptados ao domínio para design de chips: Apêndice

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: Приложение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Ek

ChipNeMo: LLM thích ứng theo miền cho thiết kế chip: Phụ lục

ChipNeMo：适用于芯片设计的领域法学硕士：附录

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VIII. CONCLUSIONS The authors would like to thank: NVIDIA IT teams for their support on NVBugs integration; NVIDIA Hardware Security team for their support on security issues; NVIDIA NeMo teams for their support and guidance on training and inference of ChipNeMo models; NVIDIA Infrastructure teams for supporting the GPU training and inference resources for the project; NVIDIA Hardware design teams for their support and insight. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VIII. CONCLUSIONS The authors would like to thank: NVIDIA IT teams for their support on NVBugs integration; NVIDIA Hardware Security team for their support on security issues; NVIDIA NeMo teams for their support and guidance on training and inference of ChipNeMo models; NVIDIA Infrastructure teams for supporting the GPU training and inference resources for the project; NVIDIA Hardware design teams for their support and insight. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Conclusions

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: উপসংহার

ChipNeMo: Domänenangepasste LLMs für den Chipentwurf: Schlussfolgerungen

ChipNeMo: LLM adaptados al dominio para el diseño de chips: Conclusiones

ChipNeMo : LLM adaptés au domaine pour la conception de puces : conclusions

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: निष्कर्ष

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 結論

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 결론

ChipNeMo: LLMs adaptados ao domínio para design de chips: conclusões

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: выводы

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Sonuçlar

ChipNeMo: LLM thích ứng với tên miền cho thiết kế chip: Kết luận

ChipNeMo：适用于芯片设计的领域适应性法学硕士：结论

ChipNeMo: Domain-Adapted LLMs for Chip Design: Abstract and Intro

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: অ্যাবস্ট্রাক্ট এবং ইন্ট্রো

ChipNeMo: Fachangepasste LLMs für Chipdesign: Abstract und Einführung

ChipNeMo: LLM adaptados al dominio para diseño de chips: resumen e introducción

ChipNeMo : LLM adaptés au domaine pour la conception de puces : résumé et introduction

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: सार और परिचय

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 概要と概要

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 개요 및 소개

ChipNeMo: LLMs adaptados ao domínio para design de chips: resumo e introdução

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: Краткое содержание и введение

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Özet ve Giriş

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Tóm tắt và giới thiệu

ChipNeMo：芯片设计领域适应性法学硕士：摘要和简介

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix IV. LLM APPLICATIONS We conducted a survey of potential LLM applications within our design teams and categorized them into four buckets: code generation, question & answer, analysis and reporting, and triage. Code generation refers to LLM generating design code, testbenches, assertions, internal tools scripts, etc.; Q & A refers to an LLM answering questions about designs, tools, infrastructures, etc.; Analysis and reporting refers to an LLM analyzing data and providing reports; triage refers to an LLM helping debug design or tool problems given logs and reports. We selected one key application from each category to study in this work, except for the triage category which we leave for further research. The motivation and technical details of each application are given below. A. Engineering Assistant Chatbot This application aims to help design engineers with answers to their architecture, design, verification, and build questions, which could significantly improve their overall productivity without impacting the productivity of others. It is observed that design engineers often enjoy brainstorming, designing hardware, and writing code, but can be slowed down waiting for answers on design knowledge they lack. Design productivity can also be enhanced by avoiding having engineers write code based on mistaken assumptions or debugging code that they are unfamiliar with. Internal studies have shown that up to 60% of a typical chip designer’s time is spent in debug or checklist related tasks across a range of topics including design specifications, testbench construction, architecture definition, and tools or infrastructure. Experts on these issues are often spread around the globe in a multinational company, such that it is not always convenient to find immediate help. Therefore, an engineering assistant chatbot based on knowledge extracted from internal design documents, code, any recorded data about designs and technical communications such as emails and corporate instant communications, etc. could help significantly improve design productivity. We implemented this application with the domain-adapted RAG method mentioned in Section III-D. B. EDA Script Generation Another common task in an industrial chip design flow is writing EDA scripts to accomplish a variety of tasks such as design implementation, introspection and transformation. These scripts often leverage both tool-specific and custom internal script libraries. Learning these libraries, navigating tool documentation, and writing and debugging these scripts, can take up a significant amount of engineering time. LLMs have proven adept at small scale code generation on a wide array of tasks [32] and therefore customizing these models to accelerate engineer productivity in this domain specific task is a natural fit. In this work we focus on generating two different types of scripts from natural language task descriptions. The first are scripts which leverage Tool1, an internal python library for design editing and analysis. The second are Tcl scripts that use the command interface provided by Tool2, which is a leading industrial static timing analysis tool. In order to build our domain-specific fine-tuning dataset for this task, production scripts for both tools were collected from design experts. We observed that our DAPT models can generate reasonable inline comments for the code. This enabled us to use these models to improve the quality of collected scripts by generating additional inline comments. Human experts later verified and corrected these comments and created an associated prompt. These prompts and code pairs make up the data used for DSFT in the format discussed in Section III-C. To provide and collect feedback in the most meaningful way, we spent significant effort building the flow shown in Fig. 4 where engineers can both query the model and run generated code through the same interface. This allows us to be confident in the correctness of generated code as well as provide accurate feedback by allowing engineers to see how many corrections they might need to get a functioning script. We support Tool1 and Tool2 integration by establishing interactive connections to tool servers. Additionally, we provide a user feedback form, allowing us to compare different models and glean valuable insights from user feedback. This valuable information can aid us in further refining our models. C. Bug Summarization and Analysis Tracking the reporting, triage, debug and resolution of various features and bugs across stages of the production flow is a time-consuming process. Engineering managers spend a lot of time reviewing internal issue tracking databases to build understanding of the state of the project and help speed their execution. Therefore, a tool that is able to look at all supporting information and quickly summarize both technical and managerial data as well as suggest next steps would boost team productivity. We focus on using LLMs to generate three different outputs - one focused on technical details, one on managerial details and one recommending task assignment. To study these tasks we used NVIDIA’s internal bug database, NVBugs. This database is used for bug reporting, tracking and resolution as well as general task and feature tracking across the company. We expect ChipNeMo models to perform well on this task as a large amount of bug data was included in the DAPT dataset. Additionally, we built a domain-specific SFT dataset for this task that includes examples of the bug summarizing and task assignment tasks. Often, bug descriptions contain large snippets of log files or code dumps along with long comment histories. In such cases, the bug text is too large for our LLM context windows. To work around this, we implemented two solutions. First, we found and replaced long path names with shorter aliases to allow the model to associate paths that occur in multiple places in the bug without needing to process the entire string. Second, we split the summarization task into an incremental task where the model is tasked with accumulating data across multiple summary and bug data chunks. We use a hierarchical approach where the bug is first separated into chunks that fit into the context window. Those chunks are then summarized and the summaries are accumulated then separated into chunks. This process is repeated until the entire set of summaries fits into a single context window and a single summary is generated. We use this same approach independent of the LLM used for summarization. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix IV. LLM APPLICATIONS We conducted a survey of potential LLM applications within our design teams and categorized them into four buckets: code generation, question & answer, analysis and reporting , and triage . Code generation refers to LLM generating design code, testbenches, assertions, internal tools scripts, etc.; Q & A refers to an LLM answering questions about designs, tools, infrastructures, etc.; Analysis and reporting refers to an LLM analyzing data and providing reports; triage refers to an LLM helping debug design or tool problems given logs and reports. We selected one key application from each category to study in this work, except for the triage category which we leave for further research. The motivation and technical details of each application are given below. code generation, question & answer, analysis and reporting triage triage A. Engineering Assistant Chatbot A. Engineering Assistant Chatbot This application aims to help design engineers with answers to their architecture, design, verification, and build questions, which could significantly improve their overall productivity without impacting the productivity of others. It is observed that design engineers often enjoy brainstorming, designing hardware, and writing code, but can be slowed down waiting for answers on design knowledge they lack. Design productivity can also be enhanced by avoiding having engineers write code based on mistaken assumptions or debugging code that they are unfamiliar with. Internal studies have shown that up to 60% of a typical chip designer’s time is spent in debug or checklist related tasks across a range of topics including design specifications, testbench construction, architecture definition, and tools or infrastructure. Experts on these issues are often spread around the globe in a multinational company, such that it is not always convenient to find immediate help. Therefore, an engineering assistant chatbot based on knowledge extracted from internal design documents, code, any recorded data about designs and technical communications such as emails and corporate instant communications, etc. could help significantly improve design productivity. We implemented this application with the domain-adapted RAG method mentioned in Section III-D. B. EDA Script Generation B. EDA Script Generation Another common task in an industrial chip design flow is writing EDA scripts to accomplish a variety of tasks such as design implementation, introspection and transformation. These scripts often leverage both tool-specific and custom internal script libraries. Learning these libraries, navigating tool documentation, and writing and debugging these scripts, can take up a significant amount of engineering time. LLMs have proven adept at small scale code generation on a wide array of tasks [32] and therefore customizing these models to accelerate engineer productivity in this domain specific task is a natural fit. In this work we focus on generating two different types of scripts from natural language task descriptions. The first are scripts which leverage Tool1, an internal python library for design editing and analysis. The second are Tcl scripts that use the command interface provided by Tool2, which is a leading industrial static timing analysis tool. In order to build our domain-specific fine-tuning dataset for this task, production scripts for both tools were collected from design experts. We observed that our DAPT models can generate reasonable inline comments for the code. This enabled us to use these models to improve the quality of collected scripts by generating additional inline comments. Human experts later verified and corrected these comments and created an associated prompt. These prompts and code pairs make up the data used for DSFT in the format discussed in Section III-C. To provide and collect feedback in the most meaningful way, we spent significant effort building the flow shown in Fig. 4 where engineers can both query the model and run generated code through the same interface. This allows us to be confident in the correctness of generated code as well as provide accurate feedback by allowing engineers to see how many corrections they might need to get a functioning script. We support Tool1 and Tool2 integration by establishing interactive connections to tool servers. correctness Additionally, we provide a user feedback form, allowing us to compare different models and glean valuable insights from user feedback. This valuable information can aid us in further refining our models. C. Bug Summarization and Analysis C. Bug Summarization and Analysis Tracking the reporting, triage, debug and resolution of various features and bugs across stages of the production flow is a time-consuming process. Engineering managers spend a lot of time reviewing internal issue tracking databases to build understanding of the state of the project and help speed their execution. Therefore, a tool that is able to look at all supporting information and quickly summarize both technical and managerial data as well as suggest next steps would boost team productivity. We focus on using LLMs to generate three different outputs - one focused on technical details, one on managerial details and one recommending task assignment. To study these tasks we used NVIDIA’s internal bug database, NVBugs. This database is used for bug reporting, tracking and resolution as well as general task and feature tracking across the company. We expect ChipNeMo models to perform well on this task as a large amount of bug data was included in the DAPT dataset. Additionally, we built a domain-specific SFT dataset for this task that includes examples of the bug summarizing and task assignment tasks. Often, bug descriptions contain large snippets of log files or code dumps along with long comment histories. In such cases, the bug text is too large for our LLM context windows. To work around this, we implemented two solutions. First, we found and replaced long path names with shorter aliases to allow the model to associate paths that occur in multiple places in the bug without needing to process the entire string. Second, we split the summarization task into an incremental task where the model is tasked with accumulating data across multiple summary and bug data chunks. We use a hierarchical approach where the bug is first separated into chunks that fit into the context window. Those chunks are then summarized and the summaries are accumulated then separated into chunks. This process is repeated until the entire set of summaries fits into a single context window and a single summary is generated. We use this same approach independent of the LLM used for summarization. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: LLM Applications

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: এলএলএম অ্যাপ্লিকেশন

ChipNeMo: Fachangepasste LLMs für Chipdesign: LLM-Anwendungen

ChipNeMo: LLM adaptados al dominio para diseño de chips: aplicaciones LLM

ChipNeMo : LLM adaptés au domaine pour la conception de puces : applications LLM

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: एलएलएम अनुप्रयोग

ChipNeMo: チップ設計のためのドメイン適応型 LLM: LLM アプリケーション

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: LLM 애플리케이션

ChipNeMo: LLMs adaptados ao domínio para design de chips: aplicações LLM

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: приложения LLM

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Yüksek Lisans Uygulamaları

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Ứng dụng LLM

ChipNeMo：适用于芯片设计的领域法学硕士：法学硕士应用

Authors:
(1) Mingjie Liu, NVIDIA {Equal contribution};
(2) Teodor-Dumitru Ene, NVIDIA {Equal contribution};
(3) Robert Kirby, NVIDIA {Equal contribution};
(4) Chris Cheng, NVIDIA {Equal contribution};
(5) Nathaniel Pinckney, NVIDIA {Equal contribution};
(6) Rongjian Liang, NVIDIA {Equal contribution};
(7) Jonah Alben, NVIDIA;
(8) Himyanshu Anand, NVIDIA;
(9) Sanmitra Banerjee, NVIDIA;
(10) Ismet Bayraktaroglu, NVIDIA;
(11) Bonita Bhaskaran, NVIDIA;
(12) Bryan Catanzaro, NVIDIA;
(13) Arjun Chaudhuri, NVIDIA;
(14) Sharon Clay, NVIDIA;
(15) Bill Dally, NVIDIA;
(16) Laura Dang, NVIDIA;
(17) Parikshit Deshpande, NVIDIA;
(18) Siddhanth Dhodhi, NVIDIA;
(19) Sameer Halepete, NVIDIA;
(20) Eric Hill, NVIDIA;
(21) Jiashang Hu, NVIDIA;
(22) Sumit Jain, NVIDIA;
(23) Brucek Khailany, NVIDIA;
(24) George Kokai, NVIDIA;
(25) Kishor Kunal, NVIDIA;
(26) Xiaowei Li, NVIDIA;
(27) Charley Lind, NVIDIA;
(28) Hao Liu, NVIDIA;
(29) Stuart Oberman, NVIDIA;
(30) Sujeet Omar, NVIDIA;
(31) Sreedhar Pratty, NVIDIA;
(23) Jonathan Raiman, NVIDIA;
(33) Ambar Sarkar, NVIDIA;
(34) Zhengjiang Shao, NVIDIA;
(35) Hanfei Sun, NVIDIA;
(36) Pratik P Suthar, NVIDIA;
(37) Varun Tej, NVIDIA;
(38) Walker Turner, NVIDIA;
(39) Kaizhe Xu, NVIDIA;
(40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro
Dataset
ChipNemo Domain Adaptation Methods
LLM Applications
Evaluations
Discussion
Related Works
Conclusions
Acknowledgments, Contributions and References
Appendix VII. RELATED WORKS Many domains have a significant amount of proprietary data which can be used to train a domain-specific LLM. One approach is to train a domain specific foundation model from scratch, e.g., BloombergGPT [10] for finance, BioMedLLM [11] for biomed, and Galactica [38] for science. These models were usually trained on more than 100B tokens of raw domain data. The second approach is domain-adaptive pretraining (DAPT) [14] which continues to train a pretrained foundation model on additional raw domain data. It shows slight performance boost on domain-specific tasks in domains such as biomedical, computer science publications, news, and reviews. In one example, [39] continued-pretrained a foundation model on technical content datasets and achieved state-of-theart performance on many quantitative reasoning tasks. Retrieval Augmented Generation (RAG) helps ground the LLM to generate accurate information and to extract up-to-date information to improve knowledge-intensive NLP tasks [40]. It is observed that smaller models with RAG can outperform larger models without RAG [41]. Retrieval methods include sparse retrieval methods such as TF-IDF or BM25 [42], which analyze word statistic information and find matching documents with a high dimensional sparse vector. Dense retrieval methods such as [43] [44] find matching documents on an embedding space generated by a retrieval model pretrained on a large corpus with or without fine-tuning on a retrieval dataset. The retrieval model can be trained standalone [43] [44] [45] or jointly with language models [46] [41]. In addition, it has been shown that off-the-shelf general purpose retrievers can improve a baseline language model significantly without further finetuning [47]. RAG is also proposed to perform code generation tasks [48] by retrieving from coding documents. Foundation models are completion models, which have limited chat and instruction following capabilities. Therefore, a model alignment process is applied to the foundation models to train a corresponding chat model. Instruction fine-tuning [20] and reinforcement learning from human feedback (RLHF) [36] are two common model alignment techniques. Instruction fine-tuning further trains a foundation model using instructions datasets. RLHF leverages human feedback to label a dataset to train a reward model and applies reinforcement learning to further improve models given the trained reward model. RLHF is usually more complex and resource hungry than instruction fine-tuning. Therefore, recent studies also propose to reduce this overhead with simpler methods such as DPO [49] and SteerLM [50]. Researchers have started to apply LLM to chip design problems. Early works such as Dave [51] first explored the possibility of generating Verilog from English with a language model (GPT-2). Following that work, [6] showed that fine-tuned open-source LLMs (CodeGen) on Verilog datasets collected from GitHub and Verilog textbooks outperformed state-of-the-art OpenAI models such as code-davinci-002 on 17 Verilog questions. [12] proposed a benchmark with more than 150 problems and demonstrated that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs. Chip-Chat [7] experimented with conversational flows to design and verify a 8-bit accumulator-based microprocessor with GPT-4 and GPT-3.5. Their findings showed that although GPT-4 produced relatively high-quality codes, it still does not perform well enough at understanding and fixing the errors. ChipEDA [8] proposed to use LLMs to generate EDA tools scripts. It also demonstrated that fine-tuned LLaMA2 70B model outperforms GPT-4 model on this task. This paper is available on arxiv under CC 4.0 license. Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Authors: Authors: (1) Mingjie Liu, NVIDIA {Equal contribution}; (2) Teodor-Dumitru Ene, NVIDIA {Equal contribution}; (3) Robert Kirby, NVIDIA {Equal contribution}; (4) Chris Cheng, NVIDIA {Equal contribution}; (5) Nathaniel Pinckney, NVIDIA {Equal contribution}; (6) Rongjian Liang, NVIDIA {Equal contribution}; (7) Jonah Alben, NVIDIA; (8) Himyanshu Anand, NVIDIA; (9) Sanmitra Banerjee, NVIDIA; (10) Ismet Bayraktaroglu, NVIDIA; (11) Bonita Bhaskaran, NVIDIA; (12) Bryan Catanzaro, NVIDIA; (13) Arjun Chaudhuri, NVIDIA; (14) Sharon Clay, NVIDIA; (15) Bill Dally, NVIDIA; (16) Laura Dang, NVIDIA; (17) Parikshit Deshpande, NVIDIA; (18) Siddhanth Dhodhi, NVIDIA; (19) Sameer Halepete, NVIDIA; (20) Eric Hill, NVIDIA; (21) Jiashang Hu, NVIDIA; (22) Sumit Jain, NVIDIA; (23) Brucek Khailany, NVIDIA; (24) George Kokai, NVIDIA; (25) Kishor Kunal, NVIDIA; (26) Xiaowei Li, NVIDIA; (27) Charley Lind, NVIDIA; (28) Hao Liu, NVIDIA; (29) Stuart Oberman, NVIDIA; (30) Sujeet Omar, NVIDIA; (31) Sreedhar Pratty, NVIDIA; (23) Jonathan Raiman, NVIDIA; (33) Ambar Sarkar, NVIDIA; (34) Zhengjiang Shao, NVIDIA; (35) Hanfei Sun, NVIDIA; (36) Pratik P Suthar, NVIDIA; (37) Varun Tej, NVIDIA; (38) Walker Turner, NVIDIA; (39) Kaizhe Xu, NVIDIA; (40) Haoxing Ren, NVIDIA. Table of Links Abstract and Intro Dataset ChipNemo Domain Adaptation Methods LLM Applications Evaluations Discussion Related Works Conclusions Acknowledgments, Contributions and References Appendix Abstract and Intro Abstract and Intro Dataset Dataset ChipNemo Domain Adaptation Methods ChipNemo Domain Adaptation Methods LLM Applications LLM Applications Evaluations Evaluations Discussion Discussion Related Works Related Works Conclusions Conclusions Acknowledgments, Contributions and References Acknowledgments, Contributions and References Appendix Appendix VII. RELATED WORKS Many domains have a significant amount of proprietary data which can be used to train a domain-specific LLM. One approach is to train a domain specific foundation model from scratch, e.g., BloombergGPT [10] for finance, BioMedLLM [11] for biomed, and Galactica [38] for science. These models were usually trained on more than 100B tokens of raw domain data. The second approach is domain-adaptive pretraining (DAPT) [14] which continues to train a pretrained foundation model on additional raw domain data. It shows slight performance boost on domain-specific tasks in domains such as biomedical, computer science publications, news, and reviews. In one example, [39] continued-pretrained a foundation model on technical content datasets and achieved state-of-theart performance on many quantitative reasoning tasks. Retrieval Augmented Generation (RAG) helps ground the LLM to generate accurate information and to extract up-to-date information to improve knowledge-intensive NLP tasks [40]. It is observed that smaller models with RAG can outperform larger models without RAG [41]. Retrieval methods include sparse retrieval methods such as TF-IDF or BM25 [42], which analyze word statistic information and find matching documents with a high dimensional sparse vector. Dense retrieval methods such as [43] [44] find matching documents on an embedding space generated by a retrieval model pretrained on a large corpus with or without fine-tuning on a retrieval dataset. The retrieval model can be trained standalone [43] [44] [45] or jointly with language models [46] [41]. In addition, it has been shown that off-the-shelf general purpose retrievers can improve a baseline language model significantly without further finetuning [47]. RAG is also proposed to perform code generation tasks [48] by retrieving from coding documents. Foundation models are completion models, which have limited chat and instruction following capabilities. Therefore, a model alignment process is applied to the foundation models to train a corresponding chat model. Instruction fine-tuning [20] and reinforcement learning from human feedback (RLHF) [36] are two common model alignment techniques. Instruction fine-tuning further trains a foundation model using instructions datasets. RLHF leverages human feedback to label a dataset to train a reward model and applies reinforcement learning to further improve models given the trained reward model. RLHF is usually more complex and resource hungry than instruction fine-tuning. Therefore, recent studies also propose to reduce this overhead with simpler methods such as DPO [49] and SteerLM [50]. Researchers have started to apply LLM to chip design problems. Early works such as Dave [51] first explored the possibility of generating Verilog from English with a language model (GPT-2). Following that work, [6] showed that fine-tuned open-source LLMs (CodeGen) on Verilog datasets collected from GitHub and Verilog textbooks outperformed state-of-the-art OpenAI models such as code-davinci-002 on 17 Verilog questions. [12] proposed a benchmark with more than 150 problems and demonstrated that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs. Chip-Chat [7] experimented with conversational flows to design and verify a 8-bit accumulator-based microprocessor with GPT-4 and GPT-3.5. Their findings showed that although GPT-4 produced relatively high-quality codes, it still does not perform well enough at understanding and fixing the errors. ChipEDA [8] proposed to use LLMs to generate EDA tools scripts. It also demonstrated that fine-tuned LLaMA2 70B model outperforms GPT-4 model on this task. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ChipNeMo: Domain-Adapted LLMs for Chip Design: Related Works

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: সম্পর্কিত কাজ

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Verwandte Arbeiten

ChipNeMo: LLM adaptados al dominio para diseño de chips: trabajos relacionados

ChipNeMo : LLM adaptés au domaine pour la conception de puces : travaux connexes

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: संबंधित कार्य

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 関連研究

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 관련 연구

ChipNeMo: LLMs adaptados ao domínio para design de chips: trabalhos relacionados

ChipNeMo: Адаптированные к предметной области программы LLM для проектирования микросхем: сопутствующие работы

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: İlgili Çalışmalar

ChipNeMo: LLM được điều chỉnh theo tên miền cho thiết kế chip: Các tác phẩm liên quan

ChipNeMo：适用于芯片设计的领域自适应法学硕士 (LLM)：相关作品

ChipNeMo: Domain-Adapted LLMs for Chip Design: Acknowledgements, Contributions and References

ChipNeMo: চিপ ডিজাইনের জন্য ডোমেন-অ্যাডাপ্টেড এলএলএম: স্বীকৃতি, অবদান এবং রেফারেন্স

ChipNeMo: Domänenangepasste LLMs für Chipdesign: Danksagungen, Beiträge und Referenzen

ChipNeMo: LLM adaptados al dominio para diseño de chips: agradecimientos, contribuciones y referencias

ChipNeMo : LLM adaptés au domaine pour la conception de puces : remerciements, contributions et références

चिपनेमो: चिप डिजाइन के लिए डोमेन-अनुकूलित एलएलएम: आभार, योगदान और संदर्भ

ChipNeMo: チップ設計のためのドメイン適応型 LLM: 謝辞、貢献、参考文献

ChipNeMo: 칩 설계를 위한 도메인 적응 LLM: 감사의 글, 기여 및 참고 자료

ChipNeMo: LLMs adaptados ao domínio para design de chips: reconhecimentos, contribuições e referências

ChipNeMo: Адаптированные к предметной области программы магистратуры по проектированию микросхем: благодарности, вклад и ссылки

ChipNeMo: Çip Tasarımı için Etki Alanına Uyarlanmış Yüksek Lisanslar: Teşekkür, Katkı ve Referanslar

ChipNeMo: LLM được điều chỉnh theo miền cho thiết kế chip: Lời cảm ơn, đóng góp và tài liệu tham khảo

ChipNeMo：适用于芯片设计的领域法学硕士：致谢、贡献和参考文献

How a Herd of Models Challenges ChatGPT's Dominance: Conclusion, Discussion, and References

Authors:
(1) Prerak Gandhi, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, prerakgandhi@cse.iitb.ac.in, and these authors contributed equally to this work;
(2) Vishal Pramanik, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, vishalpramanik,pb@cse.iitb.ac.in, and these authors contributed equally to this work;
(3) Pushpak Bhattacharyya, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai. Table of Links Abstract and Intro
Motivation
Related Work
Dataset
Experiments and Evaluation
Results and Analysis
Conclusion and Future Work
Limitations and References
A. Appendix 8. Limitations • In the plot generation dataset, the Wikipedia plots are sometimes not written by professional content writers from the film industry. Therefore these plots may fail to include the main events of the movie. • In a few cases, the model fails to generate coherent events along with the abrupt introduction of characters in the plots and scenes. • Although it has been noticed only a few times, the plot or scene generated contains repeated clauses or phrases. • The model hallucinates and generates factually incorrect things, making it incapable of generating biographies or documentaries. • The plot or scene may not abide by the theme of the input or genre mentioned along with the prompt. References Selmer Bringsjord and David Ferrucci. 1999. Artificial intelligence and literary creativity: Inside the mind of brutus, a storytelling machine. Psychology Press. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109. S. Field. 1979. Screenplay: The Foundations of Screenwriting. A Delta book. Dell Publishing Company. Chenyang Huang, Osmar R Zaiane, Amine Trabelsi, and Nouha Dziri. 2018. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 49–54. Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233–1239. Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranarayanan. 2017. Story generation from sequence of independent short descriptions. arXiv preprint arXiv:1707.05501. Frederick Jelinek, Robert L. Mercer, Lalit R. Bahl, and J. Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62. Mihir Kale and Abhinav Rastogi. 2020. Template guided text generation for task-oriented dialogue. arXiv preprint arXiv:2004.15006. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541. Rensis Likert. 1932. A technique for the measurement of attitudes / by Rensis Likert. Archives of psychology ; no. 140. [s.n.], New York. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. Rafael Pérez Ý Pérez and Mike Sharples. 2001. Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental & Theoretical Artificial Intelligence, 13(2):119–139. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. Plotmachines: Outline conditioned generation with dynamic plot state tracking. arXiv preprint arXiv:2004.14967. Mark O Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. Journal of Artificial Intelligence Research, 39:217–268. Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Targetguided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624–5634, Florence, Italy. Association for Computational Linguistics. Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016. Chinese poetry generation with planning based neural network. arXiv preprint arXiv:1610.09889. Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794–3804, Florence, Italy. Association for Computational Linguistics. Rui Yan. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In IJCAI, volume 2238, page 2244. Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive summarization. arXiv preprint arXiv:2009.13312. Yutao Zhu, Ruihua Song, Zhicheng Dou, Jian-Yun Nie, and Jin Zhou. 2020. ScriptWriter: Narrative-guided script generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8647–8657, Online. Association for Computational Linguistics. This paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Prerak Gandhi, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, prerakgandhi@cse.iitb.ac.in, and these authors contributed equally to this work; (2) Vishal Pramanik, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, vishalpramanik,pb@cse.iitb.ac.in, and these authors contributed equally to this work; (3) Pushpak Bhattacharyya, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai. Authors: Authors: (1) Prerak Gandhi, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, prerakgandhi@cse.iitb.ac.in, and these authors contributed equally to this work; (2) Vishal Pramanik, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai, vishalpramanik,pb@cse.iitb.ac.in, and these authors contributed equally to this work; (3) Pushpak Bhattacharyya, Department of Computer Science and Engineering, Indian Institute of Technology Bombay, Mumbai. Table of Links Abstract and Intro Motivation Related Work Dataset Experiments and Evaluation Results and Analysis Conclusion and Future Work Limitations and References A. Appendix Abstract and Intro Abstract and Intro Motivation Motivation Related Work Related Work Dataset Dataset Experiments and Evaluation Experiments and Evaluation Results and Analysis Results and Analysis Conclusion and Future Work Conclusion and Future Work Limitations and References Limitations and References A. Appendix A. Appendix 8. Limitations • In the plot generation dataset, the Wikipedia plots are sometimes not written by professional content writers from the film industry. Therefore these plots may fail to include the main events of the movie. • In a few cases, the model fails to generate coherent events along with the abrupt introduction of characters in the plots and scenes. • Although it has been noticed only a few times, the plot or scene generated contains repeated clauses or phrases. • The model hallucinates and generates factually incorrect things, making it incapable of generating biographies or documentaries. • The plot or scene may not abide by the theme of the input or genre mentioned along with the prompt. References Selmer Bringsjord and David Ferrucci. 1999. Artificial intelligence and literary creativity: Inside the mind of brutus, a storytelling machine. Psychology Press. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109. S. Field. 1979. Screenplay: The Foundations of Screenwriting. A Delta book. Dell Publishing Company. Chenyang Huang, Osmar R Zaiane, Amine Trabelsi, and Nouha Dziri. 2018. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 49–54. Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233–1239. Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranarayanan. 2017. Story generation from sequence of independent short descriptions. arXiv preprint arXiv:1707.05501. Frederick Jelinek, Robert L. Mercer, Lalit R. Bahl, and J. Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62. Mihir Kale and Abhinav Rastogi. 2020. Template guided text generation for task-oriented dialogue. arXiv preprint arXiv:2004.15006. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541. Rensis Likert. 1932. A technique for the measurement of attitudes / by Rensis Likert. Archives of psychology ; no. 140. [s.n.], New York. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. Rafael Pérez Ý Pérez and Mike Sharples. 2001. Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental & Theoretical Artificial Intelligence, 13(2):119–139. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. Plotmachines: Outline conditioned generation with dynamic plot state tracking. arXiv preprint arXiv:2004.14967. Mark O Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. Journal of Artificial Intelligence Research, 39:217–268. Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Targetguided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624–5634, Florence, Italy. Association for Computational Linguistics. Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016. Chinese poetry generation with planning based neural network. arXiv preprint arXiv:1610.09889. Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794–3804, Florence, Italy. Association for Computational Linguistics. Rui Yan. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In IJCAI, volume 2238, page 2244. Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive summarization. arXiv preprint arXiv:2009.13312. Yutao Zhu, Ruihua Song, Zhicheng Dou, Jian-Yun Nie, and Jin Zhou. 2020. ScriptWriter: Narrative-guided script generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8647–8657, Online. Association for Computational Linguistics. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv

"Kurosawa": A Script Writer's Assistant: Limitations and References

"কুরোসাওয়া": একজন স্ক্রিপ্ট লেখকের সহকারী: সীমাবদ্ধতা এবং উল্লেখ

„Kurosawa“: Der Assistent eines Drehbuchautors: Einschränkungen und Referenzen

"Kurosawa": asistente de guionista: limitaciones y referencias

"Kurosawa" : un assistant de scénariste : limites et références

"कुरोसावा": एक पटकथा लेखक का सहायक: सीमाएँ और संदर्भ

「黒澤」脚本家アシスタント：限界と参考文献

"구로사와": 각본가의 조수: 제한 사항 및 참고 사항

"Kurosawa": um assistente de roteirista: limitações e referências

«Куросава»: помощник сценариста: ограничения и ссылки

"Kurosawa": Bir Senaryo Yazarının Asistanı: Sınırlamalar ve Referanslar

"Kurosawa": Trợ lý biên kịch: Hạn chế và tài liệu tham khảo

“黑泽明”：编剧助手的局限性与借鉴

Concluding Our Characterizing Biases in Cable News Study

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 6 Conclusion In this paper, we introduce Multi-EuP, a novel dataset for multilingual information retrieval across 24 languages, collected from European Parliament debates. The demographic information provided by the Multi-EuP dataset serves a dual purpose: not only does it contribute to multilingual retrieval tasks, but it also holds significant potential for advancing research in the realm of fairness and bias. This dataset can play a pivotal role in investigating issues of equitable representations and mitigation of biases within document ranking settings. Multi-EuP facilitates diverse information retrieval (IR) scenarios, encompassing one-vs-one, one-vs-many, and many-vs-many settings. We demonstrated the utility of Multi-EuP as a benchmark for evaluating both monolingual and multilingual IR. Our study reveals the presence of language bias in multilingual IR when employing BM25. We further validate the effectiveness of mitigating this bias through the strategic implementation of whitespace as a language tokenizer. We propose to conduct future work in three main areas. First, we intend to expand our investigation of language bias to encompass a broader range of ranking methods, including neural methods such as mDPR (Zhang et al., 2021), mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we will expand the dataset by developing an automated API to retrieve data published by the European Parliament (EP), thereby ensuring realtime synchronization of our dataset. Lastly, our current experiments have explored language bias only, but we plan to further investigate gender bias, age bias, and nationality bias. Limitations The limitations of the Multi-EuP dataset are notable but navigable. Primarily, the temporal coverage of the dataset is confined to the past three years. This temporal constraint arises due to the fact that, preceding 2020, documents released by the EU were predominantly available in mono-lingual versions only. However, a potential remedy lies in the amalgamation of the Europarl (Koehn, 2005) collection, enabling a more comprehensive and holistic MultiEuP dataset. Furthermore, it is worth noting the domain skew of the dataset, in that Multi-EuP inevitably centers on political matters. While this presents challenges, particularly in terms of the intricate nuances of political language, it inherently serves as an excellent foundational stepping stone for delving into the intricacies of multilingual retrieval. We believe, however, that this dataset can serve as a launching pad for broader explorations encompassing crossdomain and open-domain transfer learning scenarios, thus contributing to the broader landscape of language understanding and retrieval. Ethics Statement The dataset contains publicly-available EP data that does not include personal or sensitive information, with the exception of information relating to public officeholders, e.g., the names of the active members of the European Parliament, European Council, or other official administration bodies. The collected data is licensed under the Creative Commons Attribution 4.0 International licence. [8] Acknowledgements This research was funded by Melbourne Research Scholarship and undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200. We would like to thank George Buchanan for providing valuable feedback. References Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021. mMARCO: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.13897. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 8 https://eur-lex.europa.eu/cont Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023. Neural approaches to multilingual information retrieval. arXiv cs.IR 2209.01335. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. https://github.com/ castorini/pyserini. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain. Association for Computational Linguistics. Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015. Multilingual information retrieval in the language modeling framework. Information Retrieval Journal, 18:246–281. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, page 391, Alicante, Spain. Denny Vrandeciˇ c and Markus Krötzsch. 2014. ´ Wikidata: A free collaborative knowledge base. Communications of the ACM, 57:78–85. Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023. BEIRPL: Zero shot information retrieval benchmark for the Polish language. arXiv cs.IR 2305.19840. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. arXiv cs.CL 2108.08787. A. Appendix [8] https://eur-lex.europa.eu/content/ legal-notice/legal-notice.html

Multi-EuP: Analysis of Bias in Information Retrieval - Conclusion, Limitations, and Ethics Statement

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 2 Background and Related Work The European Parliament (EP) serves as an important forum for political debates and decisionmaking at the European Union level. Members of the European Parliament (MEP) are elected in direct elections across the EU. The European Parliament debate is presided over by the President, who guides MEPs in discussing specific subjects. EP debates have been the source of three key datasets. First, was crafted by Koehn (2005) by collecting EP debates documents from 1996 to 2011, and extracting translations as a parallel corpus for statistical machine translation, enriched with attributes including , and Europarl-2005 debate date chapter id, MEP id, language, MEP name, MEP party. Later, Rabinovich et al. (2017) built upon , by introducing additional demographic attributes: MEP gender and . These were sourced from sources such as Wikidata (Vrandeciˇ c and Krötzsch ´ , 2014) and automatic annotation tools such as *Genderize[*2] and . [3] However, is limited to only two language pairs: English– German and English–French. (Vanmassenhove and Hardmeier, 2018) expanded upon to add twenty additional language pairs, based on the manual translations in the EP archives. These corpora have been used primarily for machine translation research. Europarl2017 Europarl-2005 MEP age AlchemyVision Europarl-2017 Europarl-2018 Europarl-2017 Since 2020, the EU has publicly released raw debates in the form of transcribed source-language speeches with rich multilingual topic index data, along with the original video and audio recordings. This forms the basis of the Multi-EuP dataset, with additional attributes for each speaking MEP such as an image, birthplace, and nationality. Zhang et al. (2021) introduced Mr. TYDI, an evaluation benchmark dataset for dense retrieval assessment over 11 languages. This dataset is constructed from TYDI (Clark et al., 2020), a question answering dataset. For each language, annotators assign relevance scores as judgments for questions, derived from Wikipedia articles. Notably, the questions for different languages are crafted independently, and relevance judgements are provided in-language only. Based on the dataset, the authors evaluate on monolingual retrieval tasks for non-English languages using BM25 and mDPR as zero-shot baselines. However, Mr. TYDI’s scope is limited in that it is not truly multilingual, in that queries in a given language are only performed over documents in that language. This is part of the void our work aims to address. MS MARCO (Nguyen et al., 2016) is a widelyused dataset, sourced from Bing’s search query logs, but for English queries and documents only. To mitigate this, Bonifacio et al. (2021) introduced mMARCO, a multilingual variant of the MS MARCO passage ranking dataset, spanning 13 languages and created through machine translation, based on one open-source approach (Tiedemann and Thottingal, 2020) and one commercial system in the form of Google Translate.[4] Analysis of the authors’ results reveals a positive correlation between translation quality and retrieval performance, with higher translation BLEU scores yielding improved retrieval MRR outcomes. However, similar to Mr. TYDI, mMARCO focuses on in-language retrieval only for multiple languages, rather than multilingual retrieval. Throughout the past few decades, numerous datasets and tasks pertaining to multilingual retrieval have been developed for evaluation, through efforts such as CLEF, TREC, and FIRE, each contributing standardized document collections and evaluation procedures. These evaluation datasets facilitate genuine multilingual IR research such as Rahimi et al. (2015) and Lawrie et al. (2023). However, the scope of these datasets is generally limited to a small number of queries. For example, in the case of CLEF 2001-2003, each edition encompasses a mere few dozen queries. This limitation tends to confine research predominantly to evaluation and not offer a resource for training a multilingual ranking model. Our dataset is of a scale to accommodate both large-scale training and evaluation of multilingual retrieval methods. Compared with the related work above, our work augments the multilingual mixture of queries and documents compared to Mr.TYDI, preserves the authenticity of multilingual contexts compared to mMARCO’s translation-based approach, and surpasses the query count limitations of tasks like CLEF. [2] https://genderize.io/ [3] https://www.ibm.com/smarterplanet/us/en/ ibmwatson/developercloud/alchemy-vision.html [4] https://cloud.google.com/translate

Multi-EuP: Analysis of Bias in Information Retrieval - Background and Related Work

Stories, News & Mentions about BOHNI

Bohni’s Stories on HackerNoon