Boeing must develop a comprehensive plan to address &quot;systemic quality-control issues&quot; within 90 days, the head of the U.S. Federal Aviation Administration said Wednesday, after a mid-air emergency last month sparked renewed safety concerns.

FAA gives Boeing 90 days to develop plan to address quality issues | Reuters

As the dust settles on the last few eventful days, it&#x27;s clear that Russell Wilson will soon be moving on from the Denver Broncos. For proof, look no further than how Broncos head coach Sean Payton openly broke down the fundamental deficiencies with which Wilson saddled him in 2023 · What happens ...

Sean Payton Shares Requisite Quality of Next Broncos QB - Sports Illustrated Mile High Huddle: Denver Broncos News, Analysis and More

The source for business news and analysis. Covering economic policy, business policy, financial news, economic issues, stock market data, local business, technology and more.

FAA ups pressure on Boeing to improve quality control

According to a survey, New Jersey grocery stores ranked towards the bottom on prices, quality of products and customer experience.

These N.J. grocery stores have the worst lines, quality, and prices, according to survey - nj.com

Our Water Quality companies help our customers manage, treat, purify, and protect the world’s waters, from wastewater facilities to lakes, rivers, watersheds, and oceans.

Danaher Water Quality Platform

Companies track all profits and costs related to producing a product. Good quality products and services should be able to lower production costs and increase profit margins. As repeat sales and customer loyalty increases because of quality, a business’ profit should rise.

What Is Quality in a Business? | Indeed.com

ASQ is a global leader in quality and consists of a community of passionate people who use their tools, ideas and expertise to make our world better.

Excellence Through Quality | ASQ

Canadian-based bespoke furniture manufacturing for luxury hotels and restaurants around the world.

Home - Quality & Company

For more than 40 years, Quality Sausage Company in Dallas, TX has been supplying products to food businesses. Contact us to learn more.

Fully Cooked Meats in Dallas, TX | Quality Sausage Company

QUALITY ... Is In All We Do Quality Companies is a consortium of premier oil and gas product and service companies strategically located throughout the Gulf Coast region and beyond. Four industry-leading entities make up the Quality Companies family. The Quality Companies umbrella holds true ...

Home - Quality Companies

Quoality

quoality

Want to make royalty-free music with minimal effort? Whether you're looking for a cool backing track for your next social post, a funky theme song for your creative project, or even just some fresh tunes to listen to while working out, Tad AI can make it all happen. It’s an AI music generator, using advanced artificial intelligence technology to generate unique musical creations to suit a user’s prompts. You can ask it, for example, for a smooth synthpop song or a piece of electro-jazz, and it'll make exactly what you want, in an instant. Read on through our Tad review to learn all about this amazing AI music maker. What Is Tad AI How Does It Work? Tad is a web-based, text-to-music AI song generator. In simple terms, that means it can take a textual prompt, like “a classical song about floating on a lake” and make an actual original piece of music based on that prompt. Its groundbreaking technology, with almost limitless potential for future applications, and a lot of users, from artists to students to business owners, are using this tool daily to create new music. In terms of how it works, it's similar to how language models like ChatGPT are able to create content and produce an output based on a user prompt. Those models are trained on huge amounts of data to deliver accurate, relevant results to a user's query. With Tad, it's a similar story, except instead of producing a text-based output, it creates music, and the results are out-of-this-world, with countless fresh, inspiring, entertaining, and realistic tunes created daily. Key Features of Tad.ai So, what can you actually do with Tad? Here’s a quick rundown of all its main features: Create AI Music: As an AI music maker, Tad.ai allows users to generate musical tracks, songs, and clips, all with the power of AI. Users just have to type in the song title or lyrics they can think of, and then get some music made for them. It’s that simple, and it works incredibly well to produce fun, interesting, and entertaining output. Any Genre Supported: With Tad.ai, there are no limitations to the kind of content you can create. It understands the unique differences between various musical genres. So, you have the freedom to request any genre of music - classical, jazz, synth, pop, rock, metal, and others - and receive a precise, pertinent track consistently. Save and Download On-Demand: Whenever you make an AI song with Tad.ai, you’re free to save and download it to your devices, like your phone or computer. From there, you can even use it in a range of ways – like as a backing track on a YouTube video or social media post, for instance. Set the Mood: Tad.ai offers the unique feature of mood customization for your AI-generated music. Choose from a range of emotions, such as upbeat, energetic, romantic, or melancholic, to create tracks that precisely match the desired atmosphere for your creative projects. Generate AI Lyrics: In addition to composing AI music, Tad.ai also supports AI-generated lyrics, providing users with a comprehensive music creation experience. Experiment with the platform's advanced capabilities to produce engaging lyrics that perfectly complement your AI-generated tracks. Unique Advantages of Tad AI Next, let’s see some of the main selling points that set Tad.ai apart from other AI music generators: Free to Try: A great thing about Tad compared to other AI music makers is that it’s free to try for new users. So you can test the waters and listen to AI tracks without having to spend any money right away. Royalty-Free Output: All music made by Tad.ai is royalty-free. That means you can go ahead and use these tracks for commercial or personal purposes, like your social media account or business ads, without having to pay royalties. Easy to Work With: Tad.ai is a veritable joy to work with. It’s super simple, with an easy three-step process. You just enter a song title and lyrics, let Tad.ai generate a tune for you, and then listen and save it. Complete Customization: Last but not least, Tad AI gives you absolute freedom to make the sort of music you want. You can set the tempo, pick the genre, choose the mood, and more. You’re in charge. The Final Word To sum up, there really is no better AI music maker out there at the moment than Tad. With state-of-the-art technology and incredible output potential, it’s the perfect tool to use for making royalty-free tracks of any style, easily and conveniently. Test out Tad today and start creating your own classic tunes. This story was distributed by Margrowth under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author Want to make royalty-free music with minimal effort? Whether you're looking for a cool backing track for your next social post, a funky theme song for your creative project, or even just some fresh tunes to listen to while working out, Tad AI can make it all happen. Tad AI Tad AI It’s an AI music generator, using advanced artificial intelligence technology to generate unique musical creations to suit a user’s prompts. You can ask it, for example, for a smooth synthpop song or a piece of electro-jazz, and it'll make exactly what you want, in an instant. Read on through our Tad review to learn all about this amazing AI music maker. What Is Tad AI How Does It Work? Tad is a web-based, text-to-music AI song generator . AI song generator AI song generator In simple terms, that means it can take a textual prompt, like “a classical song about floating on a lake” and make an actual original piece of music based on that prompt. Its groundbreaking technology, with almost limitless potential for future applications, and a lot of users, from artists to students to business owners, are using this tool daily to create new music. In terms of how it works, it's similar to how language models like ChatGPT are able to create content and produce an output based on a user prompt. Those models are trained on huge amounts of data to deliver accurate, relevant results to a user's query. With Tad, it's a similar story, except instead of producing a text-based output, it creates music, and the results are out-of-this-world, with countless fresh, inspiring, entertaining, and realistic tunes created daily. Key Features of Tad.ai So, what can you actually do with Tad? Here’s a quick rundown of all its main features: Create AI Music: As an AI music maker, Tad.ai allows users to generate musical tracks, songs, and clips, all with the power of AI. Users just have to type in the song title or lyrics they can think of, and then get some music made for them. It’s that simple, and it works incredibly well to produce fun, interesting, and entertaining output. Create AI Music: As an AI music maker, Tad.ai allows users to generate musical tracks, songs, and clips, all with the power of AI. Users just have to type in the song title or lyrics they can think of, and then get some music made for them. It’s that simple, and it works incredibly well to produce fun, interesting, and entertaining output. Any Genre Supported: With Tad.ai, there are no limitations to the kind of content you can create. It understands the unique differences between various musical genres. So, you have the freedom to request any genre of music - classical, jazz, synth, pop, rock, metal, and others - and receive a precise, pertinent track consistently. Any Genre Supported: With Tad.ai, there are no limitations to the kind of content you can create. It understands the unique differences between various musical genres. So, you have the freedom to request any genre of music - classical, jazz, synth, pop, rock, metal, and others - and receive a precise, pertinent track consistently. Save and Download On-Demand: Whenever you make an AI song with Tad.ai, you’re free to save and download it to your devices, like your phone or computer. From there, you can even use it in a range of ways – like as a backing track on a YouTube video or social media post, for instance. Save and Download On-Demand: Whenever you make an AI song with Tad.ai, you’re free to save and download it to your devices, like your phone or computer. From there, you can even use it in a range of ways – like as a backing track on a YouTube video or social media post, for instance. Set the Mood: Tad.ai offers the unique feature of mood customization for your AI-generated music. Choose from a range of emotions, such as upbeat, energetic, romantic, or melancholic, to create tracks that precisely match the desired atmosphere for your creative projects. Set the Mood: Tad.ai offers the unique feature of mood customization for your AI-generated music. Choose from a range of emotions, such as upbeat, energetic, romantic, or melancholic, to create tracks that precisely match the desired atmosphere for your creative projects. Generate AI Lyrics: In addition to composing AI music, Tad.ai also supports AI-generated lyrics, providing users with a comprehensive music creation experience. Experiment with the platform's advanced capabilities to produce engaging lyrics that perfectly complement your AI-generated tracks. Generate AI Lyrics: In addition to composing AI music, Tad.ai also supports AI-generated lyrics, providing users with a comprehensive music creation experience. Experiment with the platform's advanced capabilities to produce engaging lyrics that perfectly complement your AI-generated tracks. Unique Advantages of Tad AI Next, let’s see some of the main selling points that set Tad.ai apart from other AI music generators: Free to Try: A great thing about Tad compared to other AI music makers is that it’s free to try for new users. So you can test the waters and listen to AI tracks without having to spend any money right away. Free to Try: A great thing about Tad compared to other AI music makers is that it’s free to try for new users. So you can test the waters and listen to AI tracks without having to spend any money right away. Royalty-Free Output: All music made by Tad.ai is royalty-free. That means you can go ahead and use these tracks for commercial or personal purposes, like your social media account or business ads, without having to pay royalties. Royalty-Free Output: All music made by Tad.ai is royalty-free. That means you can go ahead and use these tracks for commercial or personal purposes, like your social media account or business ads, without having to pay royalties. Easy to Work With: Tad.ai is a veritable joy to work with. It’s super simple, with an easy three-step process. You just enter a song title and lyrics, let Tad.ai generate a tune for you, and then listen and save it. Easy to Work With: Tad.ai is a veritable joy to work with. It’s super simple, with an easy three-step process. You just enter a song title and lyrics, let Tad.ai generate a tune for you, and then listen and save it. Complete Customization: Last but not least, Tad AI gives you absolute freedom to make the sort of music you want. You can set the tempo, pick the genre, choose the mood, and more. You’re in charge. Complete Customization: Last but not least, Tad AI gives you absolute freedom to make the sort of music you want. You can set the tempo, pick the genre, choose the mood, and more. You’re in charge. The Final Word To sum up, there really is no better AI music maker out there at the moment than Tad. With state-of-the-art technology and incredible output potential, it’s the perfect tool to use for making royalty-free tracks of any style, easily and conveniently. Test out Tad today and start creating your own classic tunes. This story was distributed by Margrowth under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author This story was distributed by Margrowth under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author This story was distributed by Margrowth under HackerNoon’s Brand As An Author Program. Learn more about the program here: https://business.hackernoon.com/brand-as-author https://business.hackernoon.com/brand-as-author

Tad AI Review: Make High-Quality, Royalty-Free Music with AI

ታድ AI ክለሳ፡ ከፍተኛ ጥራት ያለው፣ ከሮያልቲ-ነጻ ሙዚቃ ከ AI ጋር ይስሩ

مراجعة Tad AI: اصنع موسيقى عالية الجودة وخالية من حقوق الملكية باستخدام الذكاء الاصطناعي

Tad AI পর্যালোচনা: AI দিয়ে উচ্চ-মানের, রয়্যালটি-মুক্ত সঙ্গীত তৈরি করুন

Tad AI Review: Lav royaltyfri musik af høj kvalitet med AI

Reseña de Tad AI: crea música de alta calidad y sin derechos de autor con IA

Test de Tad AI : créez de la musique de haute qualité et libre de droits avec l'IA

टैड एआई समीक्षा: एआई के साथ उच्च-गुणवत्ता, रॉयल्टी-मुक्त संगीत बनाएं

Tad AI レビュー: AI で高品質、ロイヤリティフリーの音楽を作成

Tad AI Review: Maak hoogwaardige, royaltyvrije muziek met AI

Tad AI සමාලෝචනය: AI සමඟ උසස් තත්ත්වයේ, රාජකීයත්වය-නිදහස් සංගීතය සාදන්න

Recenzia Tad AI: Vytvárajte vysokokvalitnú hudbu bez licenčných poplatkov pomocou AI

Tad AI syn: AI bilen ýokary hilli, roýaltsyz aýdym-saz düzüň

Sports analytics has long relied on basic statistics such as points per game, field goal percentage, and rebounds to assess player performance. While these metrics provide some insight, they fall short of capturing the nuances of a player's skill and decision-making abilities. Coaches and analysts often struggle with evaluating the quality of a player's shots, which is crucial for devising effective game strategies. In that realm, Ricky Zhang has made strides. A data scientist at Amazon with a background in economics, mathematics, and computer science from Emory University, Zhang has developed a project that is set to redefine how we understand player performance on the basketball court. His 'Shot Quality' project, using advanced machine learning techniques, specifically beta-binomial regression, addresses the limitations of traditional basketball metrics. The Problem: Stale Metrics and Missed Opportunities Basketball analytics, for ages, leaned on basic stats to gauge player performance. These metrics, though handy, miss the finer points of a player’s skills and decisions. Coaches and analysts often hit walls when judging shot quality—a key factor for crafting game strategies. Ricky Zhang saw these gaps and aimed higher. With his machine learning and AI chops, he was inspired to craft a model that nails down the true quality of basketball shots. A New Era in Basketball Analytics Inspired by a baseball batting averages analytics report, Zhang started creating a model that could accurately assess the quality of shots taken by basketball players. The 'Shot Quality' project was born out of this vision. Zhang’s project utilizes beta-binomial regression, a statistical method that accounts for the variability in players' shooting performance and the trust placed in them by their coaches. The 'Shot Quality' project zeroes in on the context and conditions of player shots. By crunching heaps of data—player positioning, defensive pressure, game situations—Zhang's model predicts shot success likelihood. This fresh method paints a clearer picture of a player's shooting prowess than old-school metrics. One of the biggest challenges in sports analytics is the mountain of data and its tricky interpretation. Zhang's machine learning chops let him efficiently sift through this data, pulling out valuable insights. His model tackles data noise and variability, giving coaches and analysts straightforward, actionable information. The 'Shot Quality' project's impact? Massive. Coaches can now make smarter calls on player rotations, shot choices, and defensive matchups. Knowing which players excel in specific scenarios lets teams play to their strengths and exploit opponents' weaknesses. Ricky Zhang: A Thought Leader in Sports Analytics Zhang's work redefines industry standards. His innovative machine learning approach places him as a thought leader. Beyond technical skills, he's committed to mentoring upcoming data scientists and sharing his insights. The heart of his leadership is all about resilience and adaptability, evident in how he tackled personal hurdles. When Twitch let him go due to company-wide cutbacks, he swiftly pivoted, using his network and skills to land a new gig at Amazon. This journey bolstered his resilience and dedication to nonstop learning—key ingredients in his recipe for success. Zhang's influence in sports analytics is poised to grow. He envisions further refining the 'Shot Quality' model and expanding its applications to other sports. His goal is to integrate real-time data and advanced AI techniques to provide even more accurate and actionable insights. Recent leaps in sports analytics are shining a light on how critical data-driven decisions have become in athletics. Player-tracking tech and high-res video analysis are shaking up the game, capturing every little player's move and interaction on the court, creating a treasure trove of data for the geeks to dive into. The brains at MIT Sloan Sports Analytics Conference are making noise about how machine learning and AI are flipping sports strategies on their heads. Turns out, models that mix player tracking with game context data can boost the accuracy of performance predictions and strategic calls. Take neural networks, for instance—these bad boys are getting good at predicting player performance and spotting injury risks, giving teams new ways to keep their stars healthy and sharp. Then there’s wearable tech, adding another layer to the data game. Real-time physiological and biomechanical stats are now at coaches’ fingertips, helping keep tabs on player fatigue, recovery, and overall health. This means smarter coaching decisions and players who are primed for peak performance. Zhang's 'Shot Quality' project aligns with these trends, showcasing how advanced statistical models can be applied to sports analytics. By leveraging beta-binomial regression, Zhang's model accounts for the inherent variability in player performance, providing a more nuanced understanding of shooting efficiency. This approach not only enhances the accuracy of performance metrics but also offers deeper insights into player decision-making and game dynamics. The Road Ahead for Ricky Zhang and Shot Quality Project Ricky Zhang’s 'Shot Quality' project represents a significant leap forward in sports analytics. By applying advanced machine learning techniques to basketball performance, he is transforming how we understand and evaluate players. As Zhang continues to innovate and lead in this field, the future of sports analytics looks incredibly promising. For those inspired by Zhang’s journey, there are countless opportunities to explore the intersections of data science and sports. Whether you're an aspiring data scientist or a seasoned professional, Zhang's work serves as a powerful reminder of the impact that innovative thinking and perseverance can have on an industry. If you’re interested in learning more about Ricky Zhang’s work or exploring the future of sports analytics, connect with him on LinkedIn. Stay tuned for more groundbreaking projects that will continue to reshape the world of sports. Sports analytics has long relied on basic statistics such as points per game, field goal percentage, and rebounds to assess player performance. While these metrics provide some insight, they fall short of capturing the nuances of a player's skill and decision-making abilities. Coaches and analysts often struggle with evaluating the quality of a player's shots, which is crucial for devising effective game strategies. In that realm, Ricky Zhang has made strides. Ricky Zhang Ricky Zhang A data scientist at Amazon with a background in economics, mathematics, and computer science from Emory University, Zhang has developed a project that is set to redefine how we understand player performance on the basketball court. His 'Shot Quality' project, using advanced machine learning techniques, specifically beta-binomial regression, addresses the limitations of traditional basketball metrics. The Problem: Stale Metrics and Missed Opportunities Basketball analytics, for ages, leaned on basic stats to gauge player performance. These metrics, though handy, miss the finer points of a player’s skills and decisions. Coaches and analysts often hit walls when judging shot quality—a key factor for crafting game strategies. Ricky Zhang saw these gaps and aimed higher. With his machine learning and AI chops, he was inspired to craft a model that nails down the true quality of basketball shots. A New Era in Basketball Analytics Inspired by a baseball batting averages analytics report , Zhang started creating a model that could accurately assess the quality of shots taken by basketball players. The 'Shot Quality' project was born out of this vision. Zhang’s project utilizes beta-binomial regression, a statistical method that accounts for the variability in players' shooting performance and the trust placed in them by their coaches. baseball batting averages analytics report baseball batting averages analytics report The 'Shot Quality' project zeroes in on the context and conditions of player shots. By crunching heaps of data—player positioning, defensive pressure, game situations—Zhang's model predicts shot success likelihood. This fresh method paints a clearer picture of a player's shooting prowess than old-school metrics. One of the biggest challenges in sports analytics is the mountain of data and its tricky interpretation. Zhang's machine learning chops let him efficiently sift through this data, pulling out valuable insights. His model tackles data noise and variability, giving coaches and analysts straightforward, actionable information. The 'Shot Quality' project's impact? Massive. Coaches can now make smarter calls on player rotations, shot choices, and defensive matchups. Knowing which players excel in specific scenarios lets teams play to their strengths and exploit opponents' weaknesses. Ricky Zhang: A Thought Leader in Sports Analytics Zhang's work redefines industry standards. His innovative machine learning approach places him as a thought leader. Beyond technical skills, he's committed to mentoring upcoming data scientists and sharing his insights. The heart of his leadership is all about resilience and adaptability, evident in how he tackled personal hurdles. When Twitch let him go due to company-wide cutbacks, he swiftly pivoted, using his network and skills to land a new gig at Amazon. This journey bolstered his resilience and dedication to nonstop learning—key ingredients in his recipe for success. Zhang's influence in sports analytics is poised to grow. He envisions further refining the 'Shot Quality' model and expanding its applications to other sports. His goal is to integrate real-time data and advanced AI techniques to provide even more accurate and actionable insights. Recent leaps in sports analytics are shining a light on how critical data-driven decisions have become in athletics. Player-tracking tech and high-res video analysis are shaking up the game, capturing every little player's move and interaction on the court, creating a treasure trove of data for the geeks to dive into. The brains at MIT Sloan Sports Analytics Conference are making noise about how machine learning and AI are flipping sports strategies on their heads. Turns out, models that mix player tracking with game context data can boost the accuracy of performance predictions and strategic calls. Take neural networks, for instance—these bad boys are getting good at predicting player performance and spotting injury risks, giving teams new ways to keep their stars healthy and sharp. Then there’s wearable tech, adding another layer to the data game. Real-time physiological and biomechanical stats are now at coaches’ fingertips, helping keep tabs on player fatigue, recovery, and overall health. This means smarter coaching decisions and players who are primed for peak performance. Zhang's 'Shot Quality' project aligns with these trends, showcasing how advanced statistical models can be applied to sports analytics. By leveraging beta-binomial regression, Zhang's model accounts for the inherent variability in player performance, providing a more nuanced understanding of shooting efficiency. This approach not only enhances the accuracy of performance metrics but also offers deeper insights into player decision-making and game dynamics. The Road Ahead for Ricky Zhang and Shot Quality Project Ricky Zhang’s 'Shot Quality' project represents a significant leap forward in sports analytics. By applying advanced machine learning techniques to basketball performance, he is transforming how we understand and evaluate players. As Zhang continues to innovate and lead in this field, the future of sports analytics looks incredibly promising. For those inspired by Zhang’s journey, there are countless opportunities to explore the intersections of data science and sports. Whether you're an aspiring data scientist or a seasoned professional, Zhang's work serves as a powerful reminder of the impact that innovative thinking and perseverance can have on an industry. If you’re interested in learning more about Ricky Zhang’s work or exploring the future of sports analytics, connect with him on LinkedIn . Stay tuned for more groundbreaking projects that will continue to reshape the world of sports. LinkedIn LinkedIn

The Future of Sports Analytics: Ricky Zhang's Groundbreaking 'Shot Quality' Project

İdman Analitikasının Gələcəyi: Ricky Zhang'ın təməlqoyma "Çəkiliş keyfiyyəti" layihəsi

Бъдещето на спортните анализи: новаторският проект на Рики Джан „Качество на удара“.

El futuro del análisis deportivo: el innovador proyecto 'Shot Quality' de Ricky Zhang

O futuro da analítica deportiva: o innovador proxecto "Shot Quality" de Ricky Zhang

Budućnost sportske analitike: revolucionarni projekt 'Kvaliteta šuta' Rickyja Zhanga

Masa Depan Analisis Olahraga: Proyek 'Kualitas Tembakan' Ricky Zhang yang Terobosan

スポーツ分析の未来: リッキー・チャンの画期的な「ショット品質」プロジェクト

Avenir ya Analytique ya Sport: Projet ya 'Shot Quality' ya sika ya Ricky Zhang

O futuro da análise esportiva: o inovador projeto "Qualidade de chute" de Ricky Zhang

Будущее спортивной аналитики: новаторский проект Рики Чжана «Качество броска»

መጻኢ ትንታነ ስፖርት፡ ሪኪ ዣንግ ዝሰርሖ መሰረት ዝሰበረ ፕሮጀክት 'ጽሬት ተኹሲ'

Spor Analitiğinin Geleceği: Ricky Zhang'ın Çığır Açan 'Şut Kalitesi' Projesi

Authors:
(1) Dustin Podell, Stability AI, Applied Research;
(2) Zion English, Stability AI, Applied Research;
(3) Kyle Lacey, Stability AI, Applied Research;
(4) Andreas Blattmann, Stability AI, Applied Research;
(5) Tim Dockhorn, Stability AI, Applied Research;
(6) Jonas Müller, Stability AI, Applied Research;
(7) Joe Penna, Stability AI, Applied Research;
(8) Robin Rombach, Stability AI, Applied Research. Table of Links Abstract and 1 Introduction 2 Improving Stable Diffusion 2.1 Architecture & Scale 2.2 Micro-Conditioning 2.3 Multi-Aspect Training 2.4 Improved Autoencoder and 2.5 Putting Everything Together 3 Future Work Appendix A Acknowledgements B Limitations C Diffusion Models D Comparison to the State of the Art E Comparison to Midjourney v5.1 F On FID Assessment of Generative Text-Image Foundation Models G Additional Comparison between Single- and Two-Stage SDXL pipeline References 2.4 Improved Autoencoder Stable Diffusion is a LDM, operating in a pretrained, learned (and fixed) latent space of an autoencoder. While the bulk of the semantic composition is done by the LDM [38], we can improve local, high-frequency details in generated images by improving the autoencoder. To this end, we train the same autoencoder architecture used for the original Stable Diffusion at a larger batch-size (256 vs 9) and additionally track the weights with an exponential moving average. The resulting autoencoder outperforms the original model in all evaluated reconstruction metrics, see Tab. 3. We use this autoencoder for all of our experiments. 2.5 Putting Everything Together We train the final model, SDXL, in a multi-stage procedure. SDXL uses the autoencoder from Sec. 2.4 and a discrete-time diffusion schedule [14, 45] with 1000 steps. First, we pretrain a base model (see Tab. 1) on an internal dataset whose height- and width-distribution is visualized in Fig. 2 for 600 000 optimization steps at a resolution of 256 × 256 pixels and a batch-size of 2048, using sizeand crop-conditioning as described in Sec. 2.2. We continue training on 512 × 512 pixel images for another 200 000 optimization steps, and finally utilize multi-aspect training (Sec. 2.3) in combination with an offset-noise [11, 25] level of 0.05 to train the model on different aspect ratios (Sec. 2.3, App. I) of ∼ 1024 × 1024 pixel area. Refinement Stage Empirically, we find that the resulting model sometimes yields samples of low local quality, see Fig. 6. To improve sample quality, we train a separate LDM in the same latent space, which is specialized on high-quality, high resolution data and employ a noising-denoising process as introduced by SDEdit [28] on the samples from the base model. We follow [1] and specialize this refinement model on the first 200 (discrete) noise scales. During inference, we render latents from the base SDXL, and directly diffuse and denoise them in latent space with the refinement model (see Fig. 1), using the same text input. We note that this step is optional, but improves sample quality for detailed backgrounds and human faces, as demonstrated in Fig. 6 and Fig. 13. To assess the performance of our model (with and without refinement stage), we conduct a user study, and let users pick their favorite generation from the following four models: SDXL, SDXL (with refiner), Stable Diffusion 1.5 and Stable Diffusion 2.1. The results demonstrate the SDXL with the refinement stage is the highest rated choice, and outperforms Stable Diffusion 1.5 & 2.1 by a significant margin (win rates: SDXL w/ refinement: 48.44%, SDXL base: 36.93%, Stable Diffusion 1.5: 7.91%, Stable Diffusion 2.1: 6.71%). See Fig. 1, which also provides an overview of the full pipeline. However, when using classical performance metrics such as FID and CLIP scores the improvements of SDXL over previous methods are not reflected as shown in Fig. 12 and discussed in App. F. This aligns with and further backs the findings of Kirstain et al. [23]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Dustin Podell, Stability AI, Applied Research; (2) Zion English, Stability AI, Applied Research; (3) Kyle Lacey, Stability AI, Applied Research; (4) Andreas Blattmann, Stability AI, Applied Research; (5) Tim Dockhorn, Stability AI, Applied Research; (6) Jonas Müller, Stability AI, Applied Research; (7) Joe Penna, Stability AI, Applied Research; (8) Robin Rombach, Stability AI, Applied Research. Authors: Authors: (1) Dustin Podell, Stability AI, Applied Research; (2) Zion English, Stability AI, Applied Research; (3) Kyle Lacey, Stability AI, Applied Research; (4) Andreas Blattmann, Stability AI, Applied Research; (5) Tim Dockhorn, Stability AI, Applied Research; (6) Jonas Müller, Stability AI, Applied Research; (7) Joe Penna, Stability AI, Applied Research; (8) Robin Rombach, Stability AI, Applied Research. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Improving Stable Diffusion 2 Improving Stable Diffusion 2.1 Architecture & Scale 2.1 Architecture & Scale 2.2 Micro-Conditioning 2.2 Micro-Conditioning 2.3 Multi-Aspect Training 2.3 Multi-Aspect Training 2.4 Improved Autoencoder and 2.5 Putting Everything Together 2.4 Improved Autoencoder and 2.5 Putting Everything Together 3 Future Work 3 Future Work Appendix Appendix A Acknowledgements A Acknowledgements B Limitations B Limitations C Diffusion Models C Diffusion Models D Comparison to the State of the Art D Comparison to the State of the Art E Comparison to Midjourney v5.1 E Comparison to Midjourney v5.1 F On FID Assessment of Generative Text-Image Foundation Models F On FID Assessment of Generative Text-Image Foundation Models G Additional Comparison between Single- and Two-Stage SDXL pipeline G Additional Comparison between Single- and Two-Stage SDXL pipeline References References 2.4 Improved Autoencoder Stable Diffusion is a LDM, operating in a pretrained, learned (and fixed) latent space of an autoencoder. While the bulk of the semantic composition is done by the LDM [38], we can improve local, high-frequency details in generated images by improving the autoencoder. To this end, we train the same autoencoder architecture used for the original Stable Diffusion at a larger batch-size (256 vs 9) and additionally track the weights with an exponential moving average. The resulting autoencoder outperforms the original model in all evaluated reconstruction metrics, see Tab. 3. We use this autoencoder for all of our experiments. 2.5 Putting Everything Together We train the final model, SDXL, in a multi-stage procedure. SDXL uses the autoencoder from Sec. 2.4 and a discrete-time diffusion schedule [14, 45] with 1000 steps. First, we pretrain a base model (see Tab. 1) on an internal dataset whose height- and width-distribution is visualized in Fig. 2 for 600 000 optimization steps at a resolution of 256 × 256 pixels and a batch-size of 2048, using sizeand crop-conditioning as described in Sec. 2.2. We continue training on 512 × 512 pixel images for another 200 000 optimization steps, and finally utilize multi-aspect training (Sec. 2.3) in combination with an offset-noise [11, 25] level of 0.05 to train the model on different aspect ratios (Sec. 2.3, App. I) of ∼ 1024 × 1024 pixel area. Refinement Stage Empirically, we find that the resulting model sometimes yields samples of low local quality, see Fig. 6. To improve sample quality, we train a separate LDM in the same latent space, which is specialized on high-quality, high resolution data and employ a noising-denoising process as introduced by SDEdit [28] on the samples from the base model. We follow [1] and specialize this refinement model on the first 200 (discrete) noise scales. During inference, we render latents from the base SDXL, and directly diffuse and denoise them in latent space with the refinement model (see Fig. 1), using the same text input. We note that this step is optional, but improves sample quality for detailed backgrounds and human faces, as demonstrated in Fig. 6 and Fig. 13. Refinement Stage To assess the performance of our model (with and without refinement stage), we conduct a user study, and let users pick their favorite generation from the following four models: SDXL, SDXL (with refiner), Stable Diffusion 1.5 and Stable Diffusion 2.1. The results demonstrate the SDXL with the refinement stage is the highest rated choice, and outperforms Stable Diffusion 1.5 & 2.1 by a significant margin (win rates: SDXL w/ refinement: 48.44%, SDXL base: 36.93%, Stable Diffusion 1.5: 7.91%, Stable Diffusion 2.1: 6.71%). See Fig. 1, which also provides an overview of the full pipeline. However, when using classical performance metrics such as FID and CLIP scores the improvements of SDXL over previous methods are not reflected as shown in Fig. 12 and discussed in App. F. This aligns with and further backs the findings of Kirstain et al. [23]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Improving Image Quality with Better Autoencoders

Authors:
(1) Suriya Gunasekar, Microsoft Research;
(2) Yi Zhang, Microsoft Research;
(3) Jyoti Aneja, Microsoft Research;
(4) Caio C´esar Teodoro Mendes, Microsoft Research;
(5) Allie Del Giorno, Microsoft Research;
(6) Sivakanth Gopi, Microsoft Research;
(7) Mojan Javaheripi, Microsoft Research;
(8) Piero Kauffmann, Microsoft Research;
(9) Gustavo de Rosa, Microsoft Research;
(10) Olli Saarikivi, Microsoft Research;
(11) Adil Salim, Microsoft Research;
(12) Shital Shah, Microsoft Research;
(13) Harkirat Singh Behl, Microsoft Research;
(14) Xin Wang, Microsoft Research;
(15) S´ebastien Bubeck, Microsoft Research;
(16) Ronen Eldan, Microsoft Research;
(17) Adam Tauman Kalai, Microsoft Research;
(18) Yin Tat Lee, Microsoft Research;
(19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction
2 Training details and the importance of high-quality data
2.1 Filtering of existing code datasets using a transformer-based classifier
2.2 Creation of synthetic textbook-quality datasets
2.3 Model architecture and training
3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries
4 Evaluation on unconventional problems with LLM grading
5 Data pruning for unbiased performance evaluation
5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis
6 Conclusion and References
A Additional examples for Section 3
B Limitation of phi-1
C Examples for Section 5 2.2 Creation of synthetic textbook-quality datasets One of the main challenges in creating a high-quality dataset for code generation is ensuring that the examples are diverse and non-repetitive. By diversity, we mean that the examples should cover a wide range of coding concepts, skills, and scenarios, and that they should vary in their level of difficulty, complexity, and style. Diversity is important for several reasons: it exposes the language model to different ways of expressing and solving problems in code, it reduces the risk of overfitting or memorizing specific patterns or solutions, and it increases the generalization and robustness of the model to unseen or novel tasks. However, achieving diversity is not trivial, especially when using synthetic data generated by another language model. Simply prompting the model to produce a coding textbook or a set of exercises, even with some variation in the instructions or the parameters, will likely result in a very homogeneous and redundant dataset, where the same concepts and solutions are repeated over and over with minor changes. This is because language models tend to follow the most probable or common paths given their training data and their priors, and they lack the creativity or the incentive to explore alternative or novel ways of generating code. Therefore, one needs to find the right “trick” that will induce the language model to be more creative and diverse in its output, while still maintaining the quality and the coherence of the examples. Inspired by [EL23], where a diverse set of short stories were created by including a random subset of words chosen from some fixed vocabulary in the prompt and requiring that they would be somehow combined in the generated text, we look for ways to inject randomness into the prompt in a way that gives rise to the generation of a diverse dataset. The synthetic textbook dataset This dataset consists of less that 1B tokens of GPT-3.5 generated Python textbooks, synthesized to provide a high-quality source of natural language heavy text interleaved with relevant code snippets. We further targeted the content of these textbooks to cover topics that promote reasoning and basic algorithmic skills. Here, diversity is obtained by providing constraints on topics and target audience of the generated textbook. The following is an example text from the synthetic textbook: The CodeExercises dataset This is a small synthetic exercises dataset consisting of less than 180M tokens of Python exercises and solutions. Each exercise is a docstring of a function that needs to be completed. The goal of this dataset is to align the model to perform function completion tasks based on natural language instructions. This dataset was also generated by GPT-3.5, where the main means of eliciting diversity is by constraining the function names. For this dataset in particular, we conduct explicit decontamination and alternative evaluations in the following sections to ensure that problems similar to those from HumanEval benchmark are not seen during finetuning. The following snippet illustrates a synthetically generated exercise. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Authors: Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References A Additional examples for Section 3 B Limitation of phi-1 C Examples for Section 5 Abstract and 1. Introduction Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References 6 Conclusion and References A Additional examples for Section 3 A Additional examples for Section 3 B Limitation of phi-1 B Limitation of phi-1 C Examples for Section 5 C Examples for Section 5 2.2 Creation of synthetic textbook-quality datasets One of the main challenges in creating a high-quality dataset for code generation is ensuring that the examples are diverse and non-repetitive. By diversity, we mean that the examples should cover a wide range of coding concepts, skills, and scenarios, and that they should vary in their level of difficulty, complexity, and style. Diversity is important for several reasons: it exposes the language model to different ways of expressing and solving problems in code, it reduces the risk of overfitting or memorizing specific patterns or solutions, and it increases the generalization and robustness of the model to unseen or novel tasks. However, achieving diversity is not trivial, especially when using synthetic data generated by another language model. Simply prompting the model to produce a coding textbook or a set of exercises, even with some variation in the instructions or the parameters, will likely result in a very homogeneous and redundant dataset, where the same concepts and solutions are repeated over and over with minor changes. This is because language models tend to follow the most probable or common paths given their training data and their priors, and they lack the creativity or the incentive to explore alternative or novel ways of generating code. Therefore, one needs to find the right “trick” that will induce the language model to be more creative and diverse in its output, while still maintaining the quality and the coherence of the examples. Inspired by [EL23], where a diverse set of short stories were created by including a random subset of words chosen from some fixed vocabulary in the prompt and requiring that they would be somehow combined in the generated text, we look for ways to inject randomness into the prompt in a way that gives rise to the generation of a diverse dataset. The synthetic textbook dataset The synthetic textbook dataset This dataset consists of less that 1B tokens of GPT-3.5 generated Python textbooks, synthesized to provide a high-quality source of natural language heavy text interleaved with relevant code snippets. We further targeted the content of these textbooks to cover topics that promote reasoning and basic algorithmic skills. Here, diversity is obtained by providing constraints on topics and target audience of the generated textbook. The following is an example text from the synthetic textbook: The CodeExercises dataset The CodeExercises dataset This is a small synthetic exercises dataset consisting of less than 180M tokens of Python exercises and solutions. Each exercise is a docstring of a function that needs to be completed. The goal of this dataset is to align the model to perform function completion tasks based on natural language instructions. This dataset was also generated by GPT-3.5, where the main means of eliciting diversity is by constraining the function names. For this dataset in particular, we conduct explicit decontamination and alternative evaluations in the following sections to ensure that problems similar to those from HumanEval benchmark are not seen during finetuning. The following snippet illustrates a synthetically generated exercise. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Textbooks are All You Need: Creation of Synthetic Textbook-quality Datasets

Authors:
(1) Suriya Gunasekar, Microsoft Research;
(2) Yi Zhang, Microsoft Research;
(3) Jyoti Aneja, Microsoft Research;
(4) Caio C´esar Teodoro Mendes, Microsoft Research;
(5) Allie Del Giorno, Microsoft Research;
(6) Sivakanth Gopi, Microsoft Research;
(7) Mojan Javaheripi, Microsoft Research;
(8) Piero Kauffmann, Microsoft Research;
(9) Gustavo de Rosa, Microsoft Research;
(10) Olli Saarikivi, Microsoft Research;
(11) Adil Salim, Microsoft Research;
(12) Shital Shah, Microsoft Research;
(13) Harkirat Singh Behl, Microsoft Research;
(14) Xin Wang, Microsoft Research;
(15) S´ebastien Bubeck, Microsoft Research;
(16) Ronen Eldan, Microsoft Research;
(17) Adam Tauman Kalai, Microsoft Research;
(18) Yin Tat Lee, Microsoft Research;
(19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction
2 Training details and the importance of high-quality data
2.1 Filtering of existing code datasets using a transformer-based classifier
2.2 Creation of synthetic textbook-quality datasets
2.3 Model architecture and training
3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries
4 Evaluation on unconventional problems with LLM grading
5 Data pruning for unbiased performance evaluation
5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis
6 Conclusion and References
A Additional examples for Section 3
B Limitation of phi-1
C Examples for Section 5 2 Training details and the importance of high-quality data As alluded to in the title of the paper, the central ingredient our model relies on textbook-quality training data. Unlike previous work that used standard sources of text data for code generation, such as The Stack [KLA+ 22] (which contains sourcecode from repositories with permissive licenses) and other web-based datasets (e.g., StackOverflow and CodeContest [LCC+ 22]), we argue that these sources are not optimal for teaching the model how to reason and plan algorithmically. On the other hand, our model architecture and training methods are fairly conventional (Section 2.3), so we devote this section primarily to explaining how we curated our data. The standard code datasets [KLA+ 22, LCC+ 22] form a large and diverse corpus covering broad range of topics and use cases. However, based on manual inspection of random samples we observe that many of these snippets are not very instructive for learning the basics of coding, and suffer from several drawbacks: • Many samples are not self-contained, meaning that they depend on other modules or files that are external to the snippet, making them hard to understand without additional context. • Typical examples do not involve any meaningful computation, but rather consist of trivial or boilerplate code, such as defining constants, setting parameters, or configuring GUI elements. • Samples that do contain algorithmic logic are often buried inside complex or poorly documented functions, making them difficult to follow or learn from. • The examples are skewed towards certain topics or use cases, resulting in an unbalanced distribution of coding concepts and skills across the dataset. One can only imagine how frustrating and inefficient it would be for a human learner to try to acquire coding skills from these datasets, as they would have to deal with a lot of noise, ambiguity, and incompleteness in the data. We hypothesize that these issues also affect the performance of language models, as they reduce the quality and quantity of the signal that maps natural language to code. We conjecture that language models would benefit from a training set that has the same qualities as a good “textbook”: it should be clear, self-contained, instructive, and balanced. In this work, we address this challenge directly and show that by intentionally selecting and generating high-quality data, we can achieve state-of-the-art results on code-generation tasks with a much smaller model and less compute than existing approaches. Our training relies on three main datasets: • A filtered code-language dataset, which is a subset of The Stack and StackOverflow, obtained by using a language model-based classifier (consisting of about 6B tokens). • A synthetic textbook dataset consisting of <1B tokens of GPT-3.5 generated Python textbooks. • A small synthetic exercises dataset consisting of ∼180M tokens of Python exercises and solutions. We describe those datasets in more detail in the next subsections. Taken together, the above datasets contain less than 7B tokens. We refer to the combination of filtered code-language and synthetic textbook datasets as “CodeTextbook” and use it in the pretraining phase to obtain our base model phi-1-base —this model already achieves a competitive HumanEval performance of 29%. Then we use the 180M token synthetic exercises dataset, referred to as “CodeExercises”, to finetune our phi-1-base model to obtain phi-1. Despite the small size of the “CodeExercises” dataset, finetuning with this dataset is crucial not only for large improvements in generating simple Python function as shown in Figure 2.1, but more broadly to unlock many interesting emergent capabilities in our phi-1 model that are not observed in phi-1-base (see Section 3). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Authors: Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References A Additional examples for Section 3 B Limitation of phi-1 C Examples for Section 5 Abstract and 1. Introduction Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References 6 Conclusion and References A Additional examples for Section 3 A Additional examples for Section 3 B Limitation of phi-1 B Limitation of phi-1 C Examples for Section 5 C Examples for Section 5 2 Training details and the importance of high-quality data As alluded to in the title of the paper, the central ingredient our model relies on textbook-quality training data. Unlike previous work that used standard sources of text data for code generation, such as The Stack [KLA+ 22] (which contains sourcecode from repositories with permissive licenses) and other web-based datasets (e.g., StackOverflow and CodeContest [LCC+ 22]), we argue that these sources are not optimal for teaching the model how to reason and plan algorithmically. On the other hand, our model architecture and training methods are fairly conventional (Section 2.3), so we devote this section primarily to explaining how we curated our data. The standard code datasets [KLA+ 22, LCC+ 22] form a large and diverse corpus covering broad range of topics and use cases. However, based on manual inspection of random samples we observe that many of these snippets are not very instructive for learning the basics of coding, and suffer from several drawbacks: • Many samples are not self-contained, meaning that they depend on other modules or files that are external to the snippet, making them hard to understand without additional context. • Typical examples do not involve any meaningful computation, but rather consist of trivial or boilerplate code, such as defining constants, setting parameters, or configuring GUI elements. • Samples that do contain algorithmic logic are often buried inside complex or poorly documented functions, making them difficult to follow or learn from. • The examples are skewed towards certain topics or use cases, resulting in an unbalanced distribution of coding concepts and skills across the dataset. One can only imagine how frustrating and inefficient it would be for a human learner to try to acquire coding skills from these datasets, as they would have to deal with a lot of noise, ambiguity, and incompleteness in the data. We hypothesize that these issues also affect the performance of language models, as they reduce the quality and quantity of the signal that maps natural language to code. We conjecture that language models would benefit from a training set that has the same qualities as a good “textbook”: it should be clear, self-contained, instructive, and balanced. In this work, we address this challenge directly and show that by intentionally selecting and generating high-quality data, we can achieve state-of-the-art results on code-generation tasks with a much smaller model and less compute than existing approaches. Our training relies on three main datasets: • A filtered code-language dataset, which is a subset of The Stack and StackOverflow, obtained by using a language model-based classifier (consisting of about 6B tokens). filtered code-language • A synthetic textbook dataset consisting of <1B tokens of GPT-3.5 generated Python textbooks. synthetic textbook • A small synthetic exercises dataset consisting of ∼180M tokens of Python exercises and solutions. synthetic exercises We describe those datasets in more detail in the next subsections. Taken together, the above datasets contain less than 7B tokens. We refer to the combination of filtered code-language and synthetic textbook datasets as “CodeTextbook” and use it in the pretraining phase to obtain our base model phi-1-base —this model already achieves a competitive HumanEval performance of 29%. Then we use the 180M token synthetic exercises dataset, referred to as “CodeExercises”, to finetune our phi-1-base model to obtain phi-1. Despite the small size of the “CodeExercises” dataset, finetuning with this dataset is crucial not only for large improvements in generating simple Python function as shown in Figure 2.1, but more broadly to unlock many interesting emergent capabilities in our phi-1 model that are not observed in phi-1-base (see Section 3). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Textbooks are All You Need: Training Details and the Importance of High-quality Data

You may have heard the term ‘flight to quality’ many times in the financial press. The term refers to the situation when investors shift their funds to the highest-quality instruments within an asset class or choose a ‘safer’ asset class. Flight to quality occurs when investors believe in the solid long-term prospects of specific assets. When investors are outright scared, they move to cash. Right now, we can witness the flight to quality in crypto markets. BTC dominance, which measures Bitcoin’s market share as a percentage of total crypto market capitalization, has recently moved above the 56% level. At the start of this year, BTC dominance hovered near the 50% level, so the world’s main cryptocurrency has significantly increased its market share in the first eight months of 2024. BTC’s market share gains have triggered market share losses for other asset classes in the crypto space. For example, the BitMEX Meme Basket Index (BMEMET) has settled well below the levels seen at the start of this year. The BMEMET pullback from March 2024 highs looks especially brutal, as the index has lost more than 80% amid declining interest for meme coins. The dynamics of the memecoin index show that the popularity of this asset class has fallen sharply. Meanwhile, the general correction in crypto markets has been rather modest by crypto standards. These developments show that crypto investors have been selling the most speculative assets (meme coins) while keeping their exposure to the highest-quality assets like BTC and ETH. So, what should investors do when the flight to quality begins? The rule of thumb is that you should not fight the tide. When the market starts selling speculative assets, the selling pressure could be extremely strong, and such assets would suffer from serious pullbacks. The flight-to-quality scenario is a perfect time to build positions in high-quality assets. As flight to quality occurs at times when the market is not bullish, the prices of the highest-quality assets would be rather moderate, providing investors with an opportunity to increase their positions at attractive levels. At the same time, investors should stay away from coins with weak fundamentals, as such coins are usually owned by speculative traders who would quickly sell them if they do not enjoy immediate upside. In this light, it is not surprising to see that BTC's dominance continues to grow month after month. The market votes for the world’s key decentralized cryptocurrency, whose long-term prospects are beyond any doubt. Is it possible for a new bull market to begin without a corresponding rally in speculative assets? Sure! Typically, the highest-quality assets are the first to start the bull run. When speculative traders see that prices of the main coins are rising, they start searching for opportunities in lower-quality coins. That said, individual coins that generate buzz can rally even if general market sentiment shows a high preference for high-quality assets. However, finding such coins could be a complex task, so regular investors who do not have the time to do thorough market research would be better off sticking to top names in the crypto space. You may have heard the term ‘flight to quality’ many times in the financial press. The term refers to the situation when investors shift their funds to the highest-quality instruments within an asset class or choose a ‘safer’ asset class. You may have heard the term ‘flight to quality’ many times in the financial press. The term refers to the situation when investors shift their funds to the highest-quality instruments within an asset class or choose a ‘safer’ asset class. Flight to quality occurs when investors believe in the solid long-term prospects of specific assets. When investors are outright scared, they move to cash. Right now, we can witness the flight to quality in crypto markets. BTC dominance, which measures Bitcoin’s market share as a percentage of total crypto market capitalization, has recently moved above the 56% level. At the start of this year, BTC dominance hovered near the 50% level, so the world’s main cryptocurrency has significantly increased its market share in the first eight months of 2024. BTC’s market share gains have triggered market share losses for other asset classes in the crypto space. For example, the BitMEX Meme Basket Index (BMEMET) has settled well below the levels seen at the start of this year. The BMEMET pullback from March 2024 highs looks especially brutal, as the index has lost more than 80% amid declining interest for meme coins. The dynamics of the memecoin index show that the popularity of this asset class has fallen sharply. Meanwhile, the general correction in crypto markets has been rather modest by crypto standards. These developments show that crypto investors have been selling the most speculative assets (meme coins) while keeping their exposure to the highest-quality assets like BTC and ETH. So, what should investors do when the flight to quality begins? The rule of thumb is that you should not fight the tide. When the market starts selling speculative assets, the selling pressure could be extremely strong, and such assets would suffer from serious pullbacks. The flight-to-quality scenario is a perfect time to build positions in high-quality assets. As flight to quality occurs at times when the market is not bullish, the prices of the highest-quality assets would be rather moderate, providing investors with an opportunity to increase their positions at attractive levels. At the same time, investors should stay away from coins with weak fundamentals, as such coins are usually owned by speculative traders who would quickly sell them if they do not enjoy immediate upside. In this light, it is not surprising to see that BTC's dominance continues to grow month after month. The market votes for the world’s key decentralized cryptocurrency, whose long-term prospects are beyond any doubt. Is it possible for a new bull market to begin without a corresponding rally in speculative assets? Sure! Typically, the highest-quality assets are the first to start the bull run. When speculative traders see that prices of the main coins are rising, they start searching for opportunities in lower-quality coins. That said, individual coins that generate buzz can rally even if general market sentiment shows a high preference for high-quality assets. However, finding such coins could be a complex task, so regular investors who do not have the time to do thorough market research would be better off sticking to top names in the crypto space.

Flight to Quality in Crypto Explained

Authors:
(1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution;
(2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution;
(3) Nicholas D. Lane, University of Cambridge and Flower Labs;
(4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University;
(5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction
Motivation and Setup: How low-quality data affects the performance of Collaborative Training
Proposed Workflow for Data Quality Control
Experiments
Conclusion and Future Work, and References
A. Related Work
B. Heterogeneity Settings
C. Experimental Details
D. Ablation study of Unified Scoring with Anchor Data
E. Examples for low-and high- quality Data A RELATED WORK Large Language Models The remarkable achievements of large language models (LLM) have recently impacted the field of natural language processing. OpenAI’s GPT-4 (OpenAI et al., 2023), for instance, has demonstrated exceptional capabilities in various generative tasks including question answering. LLaMA (Touvron et al., 2023a;b), an open-source large language modelwith 7 to 65 billion parameters, offers an alternative platform for research. These developments have sparked interest in adapting LLMs for medical applications. Yet, most medical models are fine-tuned based on LLaMA on a small medical corpus, resulting in a deficiency of comprehensive medical knowledge integration. There has been recent efforts on training LLMs for medical domains, for example, BioBert (Lee et al., 2020), BioMedGPT (Luo et al., 2023), PMC-LLama (Wu et al., 2023). These domain-specific LLMs have been exclusively trained on medical corpora. However, there has been a lack of collaborative or federated training work in medical LLMs. Data Valuation and Attribution The seminal work on data valuation/attribution of Koh & Liang (2017) proposes attribution via approximate influence functions. It identifies training samples most responsible for a given prediction by estimating the effect of removing or slightly modifying a single training sample. In a related approach, TracIn (Pruthi et al., 2020) estimates the influence of each sample in training set on the test example by measuring the change in loss from gradient updates of mini-batches. Another related line of work has utilized Shapley values (Lundberg & Lee, 2017) to ascribe value to data, but Shapley values often require exponential time to compute. Federated Learning Federated Learning has garnered significant attention as a distributed machine learning paradigm. It shifts the traditional model training process by sharing model parameters instead of raw data. With Federated Averaging (FedAvg) (McMahan et al., 2017), participating clients train models using their own private datasets locally, and the updated model parameters are aggregated on the server. This preserves the privacy of the underlying data while collectively benefiting from the knowledge gained during the training process (Konecnˇ y et al., 2016). Recent ` work (Zhao et al., 2024) proposes a federated parameter-efficient fine-tuning paradigm for large language models, demonstrating not only its advantages in data- and parameter-efficiency but also in better generalization and stability. Despite abundant research made on federated fine-tuning LLMs, data quality control in federated manner remains under-explored. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Authors: Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Experiments Conclusion and Future Work, and References A. Related Work B. Heterogeneity Settings C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data Abstract and Introduction Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Proposed Workflow for Data Quality Control Experiments Experiments Conclusion and Future Work, and References Conclusion and Future Work, and References A. Related Work A. Related Work B. Heterogeneity Settings B. Heterogeneity Settings C. Experimental Details C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data E. Examples for low-and high- quality Data A RELATED WORK Large Language Models The remarkable achievements of large language models (LLM) have recently impacted the field of natural language processing. OpenAI’s GPT-4 (OpenAI et al., 2023), for instance, has demonstrated exceptional capabilities in various generative tasks including question answering. LLaMA (Touvron et al., 2023a;b), an open-source large language modelwith 7 to 65 billion parameters, offers an alternative platform for research. These developments have sparked interest in adapting LLMs for medical applications. Yet, most medical models are fine-tuned based on LLaMA on a small medical corpus, resulting in a deficiency of comprehensive medical knowledge integration. There has been recent efforts on training LLMs for medical domains, for example, BioBert (Lee et al., 2020), BioMedGPT (Luo et al., 2023), PMC-LLama (Wu et al., 2023). These domain-specific LLMs have been exclusively trained on medical corpora. However, there has been a lack of collaborative or federated training work in medical LLMs. Large Language Models Data Valuation and Attribution The seminal work on data valuation/attribution of Koh & Liang (2017) proposes attribution via approximate influence functions. It identifies training samples most responsible for a given prediction by estimating the effect of removing or slightly modifying a single training sample. In a related approach, TracIn (Pruthi et al., 2020) estimates the influence of each sample in training set on the test example by measuring the change in loss from gradient updates of mini-batches. Another related line of work has utilized Shapley values (Lundberg & Lee, 2017) to ascribe value to data, but Shapley values often require exponential time to compute. Data Valuation and Attribution Federated Learning Federated Learning has garnered significant attention as a distributed machine learning paradigm. It shifts the traditional model training process by sharing model parameters instead of raw data. With Federated Averaging (FedAvg) (McMahan et al., 2017), participating clients train models using their own private datasets locally, and the updated model parameters are aggregated on the server. This preserves the privacy of the underlying data while collectively benefiting from the knowledge gained during the training process (Konecnˇ y et al., 2016). Recent ` work (Zhao et al., 2024) proposes a federated parameter-efficient fine-tuning paradigm for large language models, demonstrating not only its advantages in data- and parameter-efficiency but also in better generalization and stability. Despite abundant research made on federated fine-tuning LLMs, data quality control in federated manner remains under-explored. Federated Learning This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Enhancing Data Quality in Federated Fine-Tuning of Foundation Models: Related Work

Enhancing Data Quality: Conclusion and Future Work, and References

Authors:
(1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution;
(2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution;
(3) Nicholas D. Lane, University of Cambridge and Flower Labs;
(4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University;
(5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction
Motivation and Setup: How low-quality data affects the performance of Collaborative Training
Proposed Workflow for Data Quality Control
Experiments
Conclusion and Future Work, and References
A. Related Work
B. Heterogeneity Settings
C. Experimental Details
D. Ablation study of Unified Scoring with Anchor Data
E. Examples for low-and high- quality Data 2 MOTIVATION AND SETUP: HOW LOW-QUALITY DATA AFFECTS THE PERFORMANCE OF COLLABORATIVE TRAINING In our paper, we identify two unique challenges for federated fine-tuning of LLMs in terms of data quality. 1) Real low-quality data Firstly, we aim to highlight three prevalent patterns of low-quality data observed in real-world corpora: cut, deletion and exchange. The cut category encompasses scenarios where content is truncated due to word limit constraints, deletion pertains to instances where critical terminologies are absent from the corpus, and exchange refers to examples containing entirely incorrect information. We provide specific examples of these categories in Appendix E. 2) Quality heterogeneity Quality heterogeneity refers to the variability in the quality of data collection across different clients in federated learning. Given that federated learning often encompasses a vast number of clients, each with varying capabilities in data synthesis, it is impractical to assume uniformity in data quality among all participants. Consequently, some clients may possess a higher proportion of low-quality data compared to others, highlighting the absence of a uniform standard for data sample quality across all participants. We provide two Non-IID settings in Appendix B. In our preliminary experiments, we consider the two factors above, adjust the proportion of low-quality data of composition of PMCLLama (Wu et al., 2023) and Medalpacaflashcards (Han et al., 2023) datasets in federated training, shown in Figure 1. Higher scores indicate better performance (for more details about the metrics, see Appendix C.2). The key observation is, the quality of the training data has a significant effect on the performance of collaborative training: low-quality data consistently lead to worse influence on all the metrics. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Authors: Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Experiments Conclusion and Future Work, and References A. Related Work B. Heterogeneity Settings C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data Abstract and Introduction Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Proposed Workflow for Data Quality Control Experiments Experiments Conclusion and Future Work, and References Conclusion and Future Work, and References A. Related Work A. Related Work B. Heterogeneity Settings B. Heterogeneity Settings C. Experimental Details C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data E. Examples for low-and high- quality Data 2 MOTIVATION AND SETUP: HOW LOW-QUALITY DATA AFFECTS THE PERFORMANCE OF COLLABORATIVE TRAINING In our paper, we identify two unique challenges for federated fine-tuning of LLMs in terms of data quality. 1) Real low-quality data Firstly, we aim to highlight three prevalent patterns of low-quality data observed in real-world corpora: cut, deletion and exchange. The cut category encompasses scenarios where content is truncated due to word limit constraints, deletion pertains to instances where critical terminologies are absent from the corpus, and exchange refers to examples containing entirely incorrect information. We provide specific examples of these categories in Appendix E. Real low-quality data 2) Quality heterogeneity Quality heterogeneity refers to the variability in the quality of data collection across different clients in federated learning. Given that federated learning often encompasses a vast number of clients, each with varying capabilities in data synthesis, it is impractical to assume uniformity in data quality among all participants. Consequently, some clients may possess a higher proportion of low-quality data compared to others, highlighting the absence of a uniform standard for data sample quality across all participants. We provide two Non-IID settings in Appendix B. Quality heterogeneity Quality In our preliminary experiments, we consider the two factors above, adjust the proportion of low-quality data of composition of PMCLLama (Wu et al., 2023) and Medalpacaflashcards (Han et al., 2023) datasets in federated training, shown in Figure 1. Higher scores indicate better performance (for more details about the metrics, see Appendix C.2). The key observation is, the quality of the training data has a significant effect on the performance of collaborative training: low-quality data consistently lead to worse influence on all the metrics. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Enhancing Data Quality: How low-quality data affects the performance of Collaborative Training

Authors:
(1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution;
(2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution;
(3) Nicholas D. Lane, University of Cambridge and Flower Labs;
(4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University;
(5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction
Motivation and Setup: How low-quality data affects the performance of Collaborative Training
Proposed Workflow for Data Quality Control
Experiments
Conclusion and Future Work, and References
A. Related Work
B. Heterogeneity Settings
C. Experimental Details
D. Ablation study of Unified Scoring with Anchor Data
E. Examples for low-and high- quality Data C EXPERIMENTAL DETAILS C.1 IMPLEMENTATION DETAILS Training setups We use Hugging Face’s transformers library (Wolf et al., 2020) and PEFT library (Mangrulkar et al., 2022) for loading pre-trained models and LoRA configurations. For our training and evaluation, we use PyTorch as the underlying auto-differentiation framework (Paszke et al., 2019). We use the AdamW optimizer (Loshchilov & Hutter, 2019; Kingma & Ba, 2015) for all experiments. All the experiments are conducted on the machines with the same hardware configuration using NVIDIA A40. In all experiments, we use 8 bit quantization, set batch_size = 16, and LoRA rank to 16. For the federated setting, we consider 300 communication rounds, n = 20 clients, 10 local steps of model training (equal to 3 epoch for local training). Training data We use 16k sample in total, with 8k samples randomly sampled from PMCLLama (Wu et al., 2023) and Medalpaca-flashcards (Han et al., 2023) each. In the low-quality data synthetic processes, 3.2k samples (40% total data) are either polluted with cut (10% total data), delete (15% total data) and exchange (15% of total data) tricks. These 40% low-quality data together with the rest of high-quality data composites the low-quality data set. While the oracle data set only includes the rest of high-quality data set. C.2 EVALUTION METRICS GPT-4 scoring 200 samples are randomly selected from the Medalpaca-flashcards dataset to serve as the test set. We evaluate the models that need to be compared on the test set to generate responses respectively. Then we use OpenAI GPT-4 model API to assign scores for their responses. Each response of is rated by the judge on a scale from 0 to 1, reflecting how well the answer aligns with the ground truth. OpenAssistant OpenAssistant (Köpf et al., 2023) is a reward model built on the DeBERTa (He et al., 2020) architecture, trained with human feedback data from studies by Nakano et al. (2021); Stiennon et al. (2020); Bai et al. (2022). In this metric, 200 samples were randomly selected from the Medalpaca-flashcards dataset as the test set, with each assigned a score from -5 to 5 to reflect data quality. Knowledge-based benchmarks For evaluation on multiple-choice questions, we use the USMLE (Jin et al., 2021), MedMCQA (Pal et al., 2022), PubMedQA (Jin et al., 2019) and MMLU clinical topics (Hendrycks et al., 2021b) datasets. More details refer to Appendix C.4. C.3 IN CONTEXT LEARNING C.4 KNOWLEDGE-BASED BENCHMARK MMLU MMLU (Hendrycks et al., 2021b) provides a comprehensive suite of tests for assessing text models in multi-task contexts. We utilized the clinical topics from the MMLU test set as our testing ground, encompassing a diverse range of subjects including 265 questions on Clinical Knowledge, 100 on Medical Genetics, 135 on Anatomy, 272 on Professional Medicine, 144 on College Biology, and 173 on College Medicine, all formatted as multiple-choice questions. MedMCQA MedMCQA (Pal et al., 2022) is a dataset comprised of multiple-choice questions derived from mock exams and past papers of two major Indian medical school entrance examinations, namely AIIMS and NEET-PG. The dataset is divided into two parts: the training split, which includes 182,822 questions, and the test split, consisting of 4,183 questions. Each question in the dataset is accompanied by four possible answers. PubMedQA PubMedQA (Jin et al., 2019) is a biomedical QA benchmark collected from PubMed abstracts. The PubMedQA task is designed to answer research questions with responses categorized as yes/no/maybe, effectively framing it as a multiple-choice question format. The dataset is divided into three subsets: 1,000 manually labeled question-answer pairs (denoted as PQA-L), 61,200 unlabeled pairs (PQA-U), and 211,300 pairs that have been artificially generated (PQA-A). Consistent with previous studies (Diao et al., 2023; Singhal et al., 2023), we employ the PQA-L subset as the test set for evaluating the model’s performance. USMLE USMLE (Jin et al., 2021) consists of multiple-choice questions (with 4 choices per question) that are based on the United States Medical Licensing Exams. This dataset has been compiled from questions used in professional medical board examinations and is unique in its multilingual composition, including English, Simplified Chinese, and Traditional Chinese versions. It contains 12,724 questions in English, 34,251 in Simplified Chinese, and 14,123 in Traditional Chinese. For our purposes, we focus on the English component of the dataset, which is further divided into 10,178 questions for the training set, 1,273 for the validation set, and 1,273 for the test set, adhering to the official distribution of the dataset. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Authors: Authors: (1) Wanru Zhao, University of Cambridge, Shanghai AI Laboratory with Equal contribution; (2) Yaxin Du, Shanghai Jiao Tong University with Equal contribution; (3) Nicholas D. Lane, University of Cambridge and Flower Labs; (4) Siheng Chen, Shanghai AI Laboratory and Shanghai Jiao Tong University; (5) Yanfeng Wang, Shanghai AI Laboratory and Shanghai Jiao Tong University. Table of Links Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Experiments Conclusion and Future Work, and References A. Related Work B. Heterogeneity Settings C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data Abstract and Introduction Abstract and Introduction Motivation and Setup: How low-quality data affects the performance of Collaborative Training Motivation and Setup: How low-quality data affects the performance of Collaborative Training Proposed Workflow for Data Quality Control Proposed Workflow for Data Quality Control Experiments Experiments Conclusion and Future Work, and References Conclusion and Future Work, and References A. Related Work A. Related Work B. Heterogeneity Settings B. Heterogeneity Settings C. Experimental Details C. Experimental Details D. Ablation study of Unified Scoring with Anchor Data D. Ablation study of Unified Scoring with Anchor Data E. Examples for low-and high- quality Data E. Examples for low-and high- quality Data C EXPERIMENTAL DETAILS C.1 IMPLEMENTATION DETAILS Training setups We use Hugging Face’s transformers library (Wolf et al., 2020) and PEFT library (Mangrulkar et al., 2022) for loading pre-trained models and LoRA configurations. For our training and evaluation, we use PyTorch as the underlying auto-differentiation framework (Paszke et al., 2019). We use the AdamW optimizer (Loshchilov & Hutter, 2019; Kingma & Ba, 2015) for all experiments. All the experiments are conducted on the machines with the same hardware configuration using NVIDIA A40. In all experiments, we use 8 bit quantization, set batch_size = 16, and LoRA rank to 16. For the federated setting, we consider 300 communication rounds, n = 20 clients, 10 local steps of model training (equal to 3 epoch for local training). Training setups Training data We use 16k sample in total, with 8k samples randomly sampled from PMCLLama (Wu et al., 2023) and Medalpaca-flashcards (Han et al., 2023) each. In the low-quality data synthetic processes, 3.2k samples (40% total data) are either polluted with cut (10% total data), delete (15% total data) and exchange (15% of total data) tricks. These 40% low-quality data together with the rest of high-quality data composites the low-quality data set. While the oracle data set only includes the rest of high-quality data set. Training data C.2 EVALUTION METRICS GPT-4 scoring 200 samples are randomly selected from the Medalpaca-flashcards dataset to serve as the test set. We evaluate the models that need to be compared on the test set to generate responses respectively. Then we use OpenAI GPT-4 model API to assign scores for their responses. Each response of is rated by the judge on a scale from 0 to 1, reflecting how well the answer aligns with the ground truth. GPT-4 scoring OpenAssistant OpenAssistant (Köpf et al., 2023) is a reward model built on the DeBERTa (He et al., 2020) architecture, trained with human feedback data from studies by Nakano et al. (2021); Stiennon et al. (2020); Bai et al. (2022). In this metric, 200 samples were randomly selected from the Medalpaca-flashcards dataset as the test set, with each assigned a score from -5 to 5 to reflect data quality. OpenAssistant Knowledge-based benchmarks For evaluation on multiple-choice questions, we use the USMLE (Jin et al., 2021), MedMCQA (Pal et al., 2022), PubMedQA (Jin et al., 2019) and MMLU clinical topics (Hendrycks et al., 2021b) datasets. More details refer to Appendix C.4. Knowledge-based benchmarks C.3 IN CONTEXT LEARNING C.4 KNOWLEDGE-BASED BENCHMARK MMLU MMLU (Hendrycks et al., 2021b) provides a comprehensive suite of tests for assessing text models in multi-task contexts. We utilized the clinical topics from the MMLU test set as our testing ground, encompassing a diverse range of subjects including 265 questions on Clinical Knowledge, 100 on Medical Genetics, 135 on Anatomy, 272 on Professional Medicine, 144 on College Biology, and 173 on College Medicine, all formatted as multiple-choice questions. MMLU MedMCQA MedMCQA (Pal et al., 2022) is a dataset comprised of multiple-choice questions derived from mock exams and past papers of two major Indian medical school entrance examinations, namely AIIMS and NEET-PG. The dataset is divided into two parts: the training split, which includes 182,822 questions, and the test split, consisting of 4,183 questions. Each question in the dataset is accompanied by four possible answers. MedMCQA PubMedQA PubMedQA (Jin et al., 2019) is a biomedical QA benchmark collected from PubMed abstracts. The PubMedQA task is designed to answer research questions with responses categorized as yes/no/maybe, effectively framing it as a multiple-choice question format. The dataset is divided into three subsets: 1,000 manually labeled question-answer pairs (denoted as PQA-L), 61,200 unlabeled pairs (PQA-U), and 211,300 pairs that have been artificially generated (PQA-A). Consistent with previous studies (Diao et al., 2023; Singhal et al., 2023), we employ the PQA-L subset as the test set for evaluating the model’s performance. PubMedQA USMLE USMLE (Jin et al., 2021) consists of multiple-choice questions (with 4 choices per question) that are based on the United States Medical Licensing Exams. This dataset has been compiled from questions used in professional medical board examinations and is unique in its multilingual composition, including English, Simplified Chinese, and Traditional Chinese versions. It contains 12,724 questions in English, 34,251 in Simplified Chinese, and 14,123 in Traditional Chinese. For our purposes, we focus on the English component of the dataset, which is further divided into 10,178 questions for the training set, 1,273 for the validation set, and 1,273 for the test set, adhering to the official distribution of the dataset. USMLE This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about QUOALITY

Quoality’s Stories on HackerNoon

Quoality's News & Mentions