Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Concepts in Pretraining Data and Quantifying Frequency 2 Concepts in Pretraining Data and Quantifying Frequency 3 Comparing Pretraining Frequency & “Zero-Shot” Performance and 3.1 Experimental Setup 3 Comparing Pretraining Frequency & “Zero-Shot” Performance and 3.1 Experimental Setup 3.2 Result: Pretraining Frequency is Predictive of “Zero-Shot” Performance 3.2 Result: Pretraining Frequency is Predictive of “Zero-Shot” Performance 4 Stress-Testing the Concept Frequency-Performance Scaling Trend and 4.1 Controlling for Similar Samples in Pretraining and Downstream Data 4 Stress-Testing the Concept Frequency-Performance Scaling Trend and 4.1 Controlling for Similar Samples in Pretraining and Downstream Data 4.2 Testing Generalization to Purely Synthetic Concept and Data Distributions 4.2 Testing Generalization to Purely Synthetic Concept and Data Distributions 5 Additional Insights from Pretraining Concept Frequencies 5 Additional Insights from Pretraining Concept Frequencies 6 Testing the Tail: Let It Wag! 6 Testing the Tail: Let It Wag! 7 Related Work 7 Related Work 8 Conclusions and Open Problems, Acknowledgements, and References 8 Conclusions and Open Problems, Acknowledgements, and References Part I Part I Appendix Appendix A. Concept Frequency is Predictive of Performance Across Prompting Strategies A. Concept Frequency is Predictive of Performance Across Prompting Strategies B. Concept Frequency is Predictive of Performance Across Retrieval Metrics B. Concept Frequency is Predictive of Performance Across Retrieval Metrics C. Concept Frequency is Predictive of Performance for T2I Models C. Concept Frequency is Predictive of Performance for T2I Models D. Concept Frequency is Predictive of Performance across Concepts only from Image and Text Domains D. Concept Frequency is Predictive of Performance across Concepts only from Image and Text Domains E. Experimental Details E. Experimental Details F. Why and How Do We Use RAM++? F. Why and How Do We Use RAM++? G. Details about Misalignment Degree Results G. Details about Misalignment Degree Results H. T2I Models: Evaluation H. T2I Models: Evaluation I. Classification Results: Let It Wag! I. Classification Results: Let It Wag! D Concept Frequency is Predictive of Performance across Concepts only from Image and Text Domains In all the main performance-frequency plots we have presented until now, the concept frequencies were estimated using the intersection of the image-frequencies and the text-frequencies. Here, we showcase results with using them independently in Figs. 17 and 18 respectively. We note that both independent searching methods showcase log-linear trends as before confirming our main result. We observe that the strong log-linear trend between concept frequency and zero-shot performance robustly holds across concepts derived from image and text domains independently as well the strong log-linear trend between concept frequency and zero-shot performance robustly holds across concepts derived from image and text domains independently as well the strong log-linear trend between concept frequency and zero-shot performance robustly holds across concepts derived from image and text domains independently as well E Experimental Details E.1 Setup of Mayilvahanan et al. [79] LAION-200M is a dataset obtained by deduplicating LAION-400M by pruning exact duplicates, near duplicates, and semantically similar samples within LAION-400M [10]. The control pretraining set is created by pruning 50 million highly similar samples from LAION in the order of decreasing perceptual similarity to datapoints in ImageNet-val set. We use the 150M pretraining set for obtaining the concept distribution. We evaluate the performance of a ViT-B/32 CLIP model trained on this dataset on our downstream tasks, and present our analysis on those tasks. E.2 Let It Wag!: Test Set Curation Let It Wag! To ensure our datasets are thoroughly cleaned and diverse, we follow a meticulous process: Diverse Sourcing: We gather images from three different online sources—Flickr, DuckDuckGo, and Bing Search—to maximize the variety of our dataset, while retaining very easy-to-classify images[2].


Temporal Filtering: We applied a filter to only retrieve images after January 2023 to minimize overlap with images used in the pre-training of Vision-Language Models (VLMs). Note this helps mitigate but does not ensure the overlap problem is resolved.


Outlier Removal: We employ a pre-trained InceptionNet [111] to remove outliers from the entire image pool. We do this by taking all pairwise cosine-similarities between all images in the pool, and removing the images that are in the bottom 5% of the similarity values[3].


Initial De-duplication with an InceptionNet: We employ a pre-trained InceptionNet [111] model to identify and remove duplicates. This step involves setting high thresholds for soft de-duplication (0.9 for common classes and 0.95 for fine-grained classes) to ensure only minor, precise exclusions. A threshold of 0.9/0.95 means that we consider images to be duplicates if the cosine similarity of that image’s embedding (from InceptionNet) with any other image’s embedding in the image pool is larger than 0.9/0.95.


Manual Verification: Following the automated cleaning, we manually inspect and verify the accuracy of the remaining images for each class to ensure they meet quality standards.


Second-level De-duplication with Perceptual Hashing: Post-verification, we use perceptual hashing [37] with a threshold of 10 bits to identify and remove duplicate images within each class, ensuring uniqueness across our dataset[4].


Class Balancing: Finally, we balance the dataset to ensure an equal representation of classes. This process was followed for increased quality and reliability of our dataset for image recognition tasks. Diverse Sourcing: We gather images from three different online sources—Flickr, DuckDuckGo, and Bing Search—to maximize the variety of our dataset, while retaining very easy-to-classify images[2]. Diverse Sourcing: We gather images from three different online sources—Flickr, DuckDuckGo, and Bing Search—to maximize the variety of our dataset, while retaining very easy-to-classify images[2]. Diverse Sourcing: Temporal Filtering: We applied a filter to only retrieve images after January 2023 to minimize overlap with images used in the pre-training of Vision-Language Models (VLMs). Note this helps mitigate but does not ensure the overlap problem is resolved. Temporal Filtering: We applied a filter to only retrieve images after January 2023 to minimize overlap with images used in the pre-training of Vision-Language Models (VLMs). Note this helps mitigate but does not ensure the overlap problem is resolved. Temporal Filtering: Outlier Removal: We employ a pre-trained InceptionNet [111] to remove outliers from the entire image pool. We do this by taking all pairwise cosine-similarities between all images in the pool, and removing the images that are in the bottom 5% of the similarity values[3]. Outlier Removal: We employ a pre-trained InceptionNet [111] to remove outliers from the entire image pool. We do this by taking all pairwise cosine-similarities between all images in the pool, and removing the images that are in the bottom 5% of the similarity values[3]. Outlier Removal: Initial De-duplication with an InceptionNet: We employ a pre-trained InceptionNet [111] model to identify and remove duplicates. This step involves setting high thresholds for soft de-duplication (0.9 for common classes and 0.95 for fine-grained classes) to ensure only minor, precise exclusions. A threshold of 0.9/0.95 means that we consider images to be duplicates if the cosine similarity of that image’s embedding (from InceptionNet) with any other image’s embedding in the image pool is larger than 0.9/0.95. Initial De-duplication with an InceptionNet: We employ a pre-trained InceptionNet [111] model to identify and remove duplicates. This step involves setting high thresholds for soft de-duplication (0.9 for common classes and 0.95 for fine-grained classes) to ensure only minor, precise exclusions. A threshold of 0.9/0.95 means that we consider images to be duplicates if the cosine similarity of that image’s embedding (from InceptionNet) with any other image’s embedding in the image pool is larger than 0.9/0.95. Initial De-duplication with an InceptionNet: Manual Verification: Following the automated cleaning, we manually inspect and verify the accuracy of the remaining images for each class to ensure they meet quality standards. Manual Verification: Following the automated cleaning, we manually inspect and verify the accuracy of the remaining images for each class to ensure they meet quality standards. Manual Verification: Second-level De-duplication with Perceptual Hashing: Post-verification, we use perceptual hashing [37] with a threshold of 10 bits to identify and remove duplicate images within each class, ensuring uniqueness across our dataset[4]. Second-level De-duplication with Perceptual Hashing: Post-verification, we use perceptual hashing [37] with a threshold of 10 bits to identify and remove duplicate images within each class, ensuring uniqueness across our dataset[4]. Second-level De-duplication with Perceptual Hashing: Class Balancing: Finally, we balance the dataset to ensure an equal representation of classes. This process was followed for increased quality and reliability of our dataset for image recognition tasks. Class Balancing: Finally, we balance the dataset to ensure an equal representation of classes. This process was followed for increased quality and reliability of our dataset for image recognition tasks. Class Balancing: Authors:
(1) Vishaal Udandarao, Tubingen AI Center, University of Tubingen, University of Cambridge, and equal contribution;
(2) Ameya Prabhu, Tubingen AI Center, University of Tubingen, University of Oxford, and equal contribution;
(3) Adhiraj Ghosh, Tubingen AI Center, University of Tubingen;
(4) Yash Sharma, Tubingen AI Center, University of Tubingen;
(5) Philip H.S. Torr, University of Oxford;
(6) Adel Bibi, University of Oxford;
(7) Samuel Albanie, University of Cambridge and equal advising, order decided by a coin flip;
(8) Matthias Bethge, Tubingen AI Center, University of Tubingen and equal advising, order decided by a coin flip. Authors: Authors: (1) Vishaal Udandarao, Tubingen AI Center, University of Tubingen, University of Cambridge, and equal contribution; (2) Ameya Prabhu, Tubingen AI Center, University of Tubingen, University of Oxford, and equal contribution; (3) Adhiraj Ghosh, Tubingen AI Center, University of Tubingen; (4) Yash Sharma, Tubingen AI Center, University of Tubingen; (5) Philip H.S. Torr, University of Oxford; (6) Adel Bibi, University of Oxford; (7) Samuel Albanie, University of Cambridge and equal advising, order decided by a coin flip; (8) Matthias Bethge, Tubingen AI Center, University of Tubingen and equal advising, order decided by a coin flip. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Quant

DuckDuckGo

The Link Between Concept Frequency and AI Performance, Seen Through Images and Words

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Close Look at Misalignment in Pretraining Datasets

What 300GB of AI Research Reveals About the True Limits of “Zero-Shot” Intelligence

How AI Models Count and Match Concepts in Images and Text

Analyzing the Impact of Pretraining Frequency on Zero-Shot Performance in Multimodal Models

AI Models Trained on Synthetic Data Still Follow Concept Frequency Trends

A Close Look at Misalignment in Pretraining Datasets

What 300GB of AI Research Reveals About the True Limits of “Zero-Shot” Intelligence

How AI Models Count and Match Concepts in Images and Text

Analyzing the Impact of Pretraining Frequency on Zero-Shot Performance in Multimodal Models

AI Models Trained on Synthetic Data Still Follow Concept Frequency Trends

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps