Table of Links Abstract and 1 Introduction 2 Pre-Training 2.1 Tokenization 2.2 Pre-Training Data 2.3 Stability 2.4 Inference 3 Alignment and 3.1 Data 3.2 Fine-Tuning Strategy 4 Human Evaluations and Safety Testing, and 4.1 Prompts for Evaluation 4.2 Baselines and Evaluations 4.3 Inter-annotator Agreement 4.4 Safety Testing 4.5 Discussion 5 Benchmark Evaluations and 5.1 Text 5.2 Image-To-Text 6 Related Work 7 Conclusion, Acknowledgements, Contributors, and References Appendix A. Samples B. Additional Information of Human Evaluations 5 Benchmark Evaluations Given the general capabilities of Chameleon, there is not a single model that we can directly evaluate against; therefore, we evaluate against the best models in every category within our capabilities. 5.1 Text We evaluate the general text-only capabilities of our pre-trained (not SFT’d) model against other state-of-theart text-only large language models. We follow the evaluation protocol outlined by Touvron et al. (2023). Specifically we evaluate all models, using an in-house evaluation platform on the areas of commonsense reasoning, reading comprehension, math problems, and world knowledge. We report our results in Table 6. • Commonsense Reasoning and Reading Comprehension: We report 0-shot performance on the following benchmarks that measure commonsense reasoning and reading comprehension capabilities: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and BoolQ (Clark et al., 2019). We score the prompt with each candidate answer and compute accuracy using the candidate with the highest score. All baseline model performances except a few are taken directly from the reported sources. We observe that Chameleon-7B and Chameleon-34B are competitive with the corresponding Llama-2 models, with Chameleon-34B even outperforming Llama-2 70B on 5/8 tasks and performing on par with Mixtral 8x7B. • MATH and World Knowledge We report 8-shot performance on GSM8K (Cobbe et al., 2021) i.e., grade school math word problems and 4-shot performance on the MATH (Hendrycks et al., 2021) benchmark. We report maj@N exact match accuracy for both benchmarks by sampling N generations from the model (greedy sampling for N=1) and choosing the answer via majority voting. Despite training for additional modalities, both Chameleon models demonstrate strong math capabilities. On GSM8k, Chameleon-7B outperforms the corresponding Llama-2 models, with performance comparable to Mistral 7B (50.9 vs 52.1 maj@8). Furthermore, Chameleon-34B can outperform Llama2-70B on maj@1 (61.4 vs 56.8) and Mixtral 8x7B on maj@32 (77.0 vs 75.1). Similarly, on MATH, Chameleon-7B outperforms Llama-2 and matches Mistral 7B on maj@4, while Chameleon-34B outperforms Llama2-70B, approaching the performance of Mixtral 8x7B on maj@4 (24.7 vs 28.4). We also report performance on MMLU (Hendrycks et al., 2020), which measures world/in-domain knowledge and problem-solving abilities using 57 subjects, including elementary mathematics, US history, computer science, and law. Both Chameleon models outperform their Llama-2 counterparts with Chameleon-34B approaching the performance of Mixtral 8x7B/Gemini-Pro (65.8 vs 70.6/71.8). Overall, Chameleon outperforms LLaMa-2 across the board, with performance approaching Mistral 7B/Mixtral 8x7B (Jiang et al., 2023, 2024) on some tasks. These gains are likely due to multiple factors. First, we do two epochs over the LLaMa-2 pre-training data, and in general use more compute for pretraining. Second, including code data significantly improves performance on text-only reasoning tasks. Lastly, having higher quality data in the last 20% of pre-training significantly improves performance. Author:
(1) Chameleon Team, FAIR at Meta. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Pre-Training 2 Pre-Training 2.1 Tokenization 2.1 Tokenization 2.2 Pre-Training Data 2.2 Pre-Training Data 2.3 Stability 2.3 Stability 2.4 Inference 2.4 Inference 3 Alignment and 3.1 Data 3 Alignment and 3.1 Data 3.2 Fine-Tuning Strategy 3.2 Fine-Tuning Strategy 4 Human Evaluations and Safety Testing, and 4.1 Prompts for Evaluation 4 Human Evaluations and Safety Testing, and 4.1 Prompts for Evaluation 4.2 Baselines and Evaluations 4.2 Baselines and Evaluations 4.3 Inter-annotator Agreement 4.3 Inter-annotator Agreement 4.4 Safety Testing 4.4 Safety Testing 4.5 Discussion 4.5 Discussion 5 Benchmark Evaluations and 5.1 Text 5 Benchmark Evaluations and 5.1 Text 5.2 Image-To-Text 5.2 Image-To-Text 6 Related Work 6 Related Work 7 Conclusion, Acknowledgements, Contributors, and References 7 Conclusion, Acknowledgements, Contributors, and References Appendix Appendix A. Samples A. Samples B. Additional Information of Human Evaluations B. Additional Information of Human Evaluations 5 Benchmark Evaluations Given the general capabilities of Chameleon, there is not a single model that we can directly evaluate against; therefore, we evaluate against the best models in every category within our capabilities. 5.1 Text We evaluate the general text-only capabilities of our pre-trained (not SFT’d) model against other state-of-theart text-only large language models. We follow the evaluation protocol outlined by Touvron et al. (2023). Specifically we evaluate all models, using an in-house evaluation platform on the areas of commonsense reasoning, reading comprehension, math problems, and world knowledge. We report our results in Table 6. • Commonsense Reasoning and Reading Comprehension: We report 0-shot performance on the following benchmarks that measure commonsense reasoning and reading comprehension capabilities: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and BoolQ (Clark et al., 2019). We score the prompt with each candidate answer and compute accuracy using the candidate with the highest score. All baseline model performances except a few are taken directly from the reported sources. We observe that Chameleon-7B and Chameleon-34B are competitive with the corresponding Llama-2 models, with Chameleon-34B even outperforming Llama-2 70B on 5/8 tasks and performing on par with Mixtral 8x7B. • Commonsense Reasoning and Reading Comprehension: • MATH and World Knowledge We report 8-shot performance on GSM8K (Cobbe et al., 2021) i.e., grade school math word problems and 4-shot performance on the MATH (Hendrycks et al., 2021) benchmark. We report maj@N exact match accuracy for both benchmarks by sampling N generations from the model (greedy sampling for N=1) and choosing the answer via majority voting. Despite training for additional modalities, both Chameleon models demonstrate strong math capabilities. On GSM8k, Chameleon-7B outperforms the corresponding Llama-2 models, with performance comparable to Mistral 7B (50.9 vs 52.1 maj@8). Furthermore, Chameleon-34B can outperform Llama2-70B on maj@1 (61.4 vs 56.8) and Mixtral 8x7B on maj@32 (77.0 vs 75.1). Similarly, on MATH, Chameleon-7B outperforms Llama-2 and matches Mistral 7B on maj@4, while Chameleon-34B outperforms Llama2-70B, approaching the performance of Mixtral 8x7B on maj@4 (24.7 vs 28.4). • MATH and World Knowledge We also report performance on MMLU (Hendrycks et al., 2020), which measures world/in-domain knowledge and problem-solving abilities using 57 subjects, including elementary mathematics, US history, computer science, and law. Both Chameleon models outperform their Llama-2 counterparts with Chameleon-34B approaching the performance of Mixtral 8x7B/Gemini-Pro (65.8 vs 70.6/71.8). Overall, Chameleon outperforms LLaMa-2 across the board, with performance approaching Mistral 7B/Mixtral 8x7B (Jiang et al., 2023, 2024) on some tasks. These gains are likely due to multiple factors. First, we do two epochs over the LLaMa-2 pre-training data, and in general use more compute for pretraining. Second, including code data significantly improves performance on text-only reasoning tasks. Lastly, having higher quality data in the last 20% of pre-training significantly improves performance. Author: (1) Chameleon Team, FAIR at Meta. Author: Author: (1) Chameleon Team, FAIR at Meta. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Chameleon AI Shows Competitive Edge Over LLaMa-2 and Other Models

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Chameleon Sets New Benchmarks in AI Image-Text Tasks

This AI Model Doesn’t See the Line Between Text and Images

How Chameleon AI Can Understand Images and Text Together

Overcoming Training Hurdles in Multimodal AI Models

Inside the Multi-Modal Training That Powers Safer AI

Comparing Chameleon with GPT-4V and Gemini

Chameleon Sets New Benchmarks in AI Image-Text Tasks

This AI Model Doesn’t See the Line Between Text and Images

How Chameleon AI Can Understand Images and Text Together

Overcoming Training Hurdles in Multimodal AI Models

Inside the Multi-Modal Training That Powers Safer AI

Comparing Chameleon with GPT-4V and Gemini

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps