Author:
(1) Vojtech ˇ Cerm ˇ ak, Czech Technical University in Prague (Email: cermavo3@fel.cvut.cz);
(2) Lukas Picek, University of West Bohemia & INRIA (Email: picekl@kky.zcu.cz/lpicek@inria.cz);
(3) Luka´s Adam, Czech Technical University in Prague (Email: lukas.adam.cr@gmail.com);
(4) Kostas Papafitsoros, Queen Mary University of London (Email: k.papafitsoros@qmul.ac.uk). Table of Links Abstract and Introduction
Related work
The WildlifeDatasets toolkit
MegaDescriptor – Methodology
Ablation studies
Performance evaluation
Conclusion and References 5. Ablation studies This section presents a set of ablation studies to empirically validate the design choices related to model distillation (i.e., selecting methods, architectures, and appropriate hyperparameters) while constructing the MegaDescriptor feature extractor, i.e., first-ever foundation model for animal re-identification. Furthermore, we provide both qualitative and quantitative performance evaluation comparing the newly proposed MegaDescriptor in a zero-shot setting with other methods, including SIFT, Superpoint, ImageNet, CLIP, and DINOv2. 5.1. Loss and backbone components To determine the optimal metric learning loss function and backbone architecture configuration, we conducted an ablation study, comparing the performance (median accuracy) of ArcFace and Triplet loss with either a transformer- (Swin-B) or CNN-based backbone (EfficientNet-B3) on all available re-identification datasets. In most cases, the Swin-B with ArcFace combination maintains competitive or better performance than other variants. Overall, ArcFace and transformer-based backbone (Swin-B) performed better than Triplet and CNN backbone (EfficientNet-B3). First quantiles and top whiskers indicate that Triplet loss underperforms compared to ArcFace even with correctly set hyperparameters. The full comparison in the form of a box plot is provided in Figure 5. 5.2. Hyperparameter tunning In order to overcome the performance sensitivity of metric learning approaches regarding hyperparameter selection and to select the generally optimal parameters, we have performed a comprehensive grid search strategy. Following the results from the previous ablation, we evaluate how various hyperparameter settings affect the performance of a Swin-B backbone optimized with Arcface and Triplet losses. In the case of ArcFace, the best setting (i.e., lr = 0.001, m = 0.5, and s = 64) achieved a median performance of 87.3% with 25% and 75% quantiles of 49.2% and 96.4%, respectively. Interestingly, three settings underperformed by a significant margin, most likely due to unexpected divergence in the training[5]. The worst settings achieved a mean accuracy of 6.4%, 6.1%, and 4.0%. Compared to ArcFace, Triplet loss configurations showed higher performance on both 25% and 75% quantiles, indicating significant performance variability. The outcomes of the study are visualized in Figure 6 as a boxplot, where each box consists of 29 values. 5.3. Metric learning vs. Local features The results conducted over 29 datasets suggested that both metric learning approaches (Triplet and ArcFace) outperformed the local-feature-based methods on most datasets by a significant margin. The comparison of local-featurebased methods (SIFT and Superpoint) revealed that Superpoints are a better fit for animal re-identification, even though they are rarely used over SIFT descriptors in the literature. A detailed comparison is provided in Table 3. Note that the Giraffes dataset was labeled using local descriptors; hence, the performance is inflated and better than for metric learning. The same experiment revealed that several datasets, e.g., AerialCattle2017, SMALST, MacaqueFaces, Giraffes, and AAUZebraFish, are solved or close to that point and should be omitted from development and benchmarking. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. [5] These three settings were excluded from further evaluation and visualization for a more fair comparison. Author: (1) Vojtech ˇ Cerm ˇ ak, Czech Technical University in Prague (Email: cermavo3@fel.cvut.cz); (2) Lukas Picek, University of West Bohemia & INRIA (Email: picekl@kky.zcu.cz/lpicek@inria.cz); (3) Luka´s Adam, Czech Technical University in Prague (Email: lukas.adam.cr@gmail.com); (4) Kostas Papafitsoros, Queen Mary University of London (Email: k.papafitsoros@qmul.ac.uk). Author: Author: (1) Vojtech ˇ Cerm ˇ ak, Czech Technical University in Prague (Email: cermavo3@fel.cvut.cz); (2) Lukas Picek, University of West Bohemia & INRIA (Email: picekl@kky.zcu.cz/lpicek@inria.cz); (3) Luka´s Adam, Czech Technical University in Prague (Email: lukas.adam.cr@gmail.com); (4) Kostas Papafitsoros, Queen Mary University of London (Email: k.papafitsoros@qmul.ac.uk). Table of Links Abstract and Introduction Related work The WildlifeDatasets toolkit MegaDescriptor – Methodology Ablation studies Performance evaluation Conclusion and References Abstract and Introduction Abstract and Introduction Related work Related work The WildlifeDatasets toolkit The WildlifeDatasets toolkit MegaDescriptor – Methodology MegaDescriptor – Methodology Ablation studies Ablation studies Performance evaluation Performance evaluation Conclusion and References Conclusion and References 5. Ablation studies This section presents a set of ablation studies to empirically validate the design choices related to model distillation (i.e., selecting methods, architectures, and appropriate hyperparameters) while constructing the MegaDescriptor feature extractor, i.e., first-ever foundation model for animal re-identification. Furthermore, we provide both qualitative and quantitative performance evaluation comparing the newly proposed MegaDescriptor in a zero-shot setting with other methods, including SIFT, Superpoint, ImageNet, CLIP, and DINOv2. 5.1. Loss and backbone components To determine the optimal metric learning loss function and backbone architecture configuration, we conducted an ablation study, comparing the performance (median accuracy) of ArcFace and Triplet loss with either a transformer- (Swin-B) or CNN-based backbone (EfficientNet-B3) on all available re-identification datasets. In most cases, the Swin-B with ArcFace combination maintains competitive or better performance than other variants. Overall, ArcFace and transformer-based backbone (Swin-B) performed better than Triplet and CNN backbone (EfficientNet-B3). First quantiles and top whiskers indicate that Triplet loss underperforms compared to ArcFace even with correctly set hyperparameters. The full comparison in the form of a box plot is provided in Figure 5. 5.2. Hyperparameter tunning In order to overcome the performance sensitivity of metric learning approaches regarding hyperparameter selection and to select the generally optimal parameters, we have performed a comprehensive grid search strategy. Following the results from the previous ablation, we evaluate how various hyperparameter settings affect the performance of a Swin-B backbone optimized with Arcface and Triplet losses. In the case of ArcFace, the best setting (i.e., lr = 0.001, m = 0.5, and s = 64) achieved a median performance of 87.3% with 25% and 75% quantiles of 49.2% and 96.4%, respectively. Interestingly, three settings underperformed by a significant margin, most likely due to unexpected divergence in the training[5]. The worst settings achieved a mean accuracy of 6.4%, 6.1%, and 4.0%. Compared to ArcFace, Triplet loss configurations showed higher performance on both 25% and 75% quantiles, indicating significant performance variability. The outcomes of the study are visualized in Figure 6 as a boxplot, where each box consists of 29 values. 5.3. Metric learning vs. Local features The results conducted over 29 datasets suggested that both metric learning approaches (Triplet and ArcFace) outperformed the local-feature-based methods on most datasets by a significant margin. The comparison of local-featurebased methods (SIFT and Superpoint) revealed that Superpoints are a better fit for animal re-identification, even though they are rarely used over SIFT descriptors in the literature. A detailed comparison is provided in Table 3. Note that the Giraffes dataset was labeled using local descriptors; hence, the performance is inflated and better than for metric learning. The same experiment revealed that several datasets, e.g., AerialCattle2017, SMALST, MacaqueFaces, Giraffes, and AAUZebraFish, are solved or close to that point and should be omitted from development and benchmarking. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [5] These three settings were excluded from further evaluation and visualization for a more fair comparison.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Concrete Steps for Implementing Diversity and Inclusion in AI

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Conclusion and References

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Ablation Studies

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Geek’s Guide to ML Experimentation

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: the WildlifeDatasets Toolkit

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: MegaDescriptor – Methodology

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Abstract and Introduction

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Related Work

The Geek’s Guide to ML Experimentation

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: the WildlifeDatasets Toolkit

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: MegaDescriptor – Methodology

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Abstract and Introduction

WildlifeDatasets: an Open-source Toolkit for Animal Re-identification: Related Work

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps