MLLM Adapters: Review of VPGs and Multimodal Fusion

Table of Links

Abstract and 1 Introduction

2. Related Work

   2.1. Multimodal Learning

   2.2. Multiple Instance Learning

3. Methodology

   3.1. Preliminaries and Notations

   3.2. Relations between Attention-based VPG and MIL

   3.3. MIVPG for Multiple Visual Inputs

   3.4. Unveiling Instance Correlation in MIVPG for Enhanced Multi-instance Scenarios

4. Experiments and 4.1. General Setup

   4.2. Scenario 1: Samples with Single Image

   4.3. Scenario 2: Samples with Multiple Images, with Each Image as a General Embedding

   4.4. Scenario 3: Samples with Multiple Images, with Each Image Having Multiple Patches to be Considered and 4.5. Case Study

5. Conclusion and References

Supplementary Material

A. Detailed Architecture of QFormer

B. Proof of Proposition

C. More Experiments

2. Related Work

2.1. Multimodal Learning

Recently, various vision-language models (VLMs) have been proposed to enhance the fusion of text and images. For example, TCL [42] employed triplet contrastive learning to simultaneously learn from text and images. Many state-ofthe-art MLLMs have also emerged, with one major distinction lying in the design of VPGs. For instance, FROMAGe [18] and LLaVA [24] employ a straightforward linear projection as their VPGs. On the other hand, Flamingo [2] introduces the novel use of the Perceiver Resampler, incorporating cross attention and learnable query embeddings. BLIP2 [22] innovatively employs the QFormer to improve image-text alignment. Meanwhile, MiniGPT-4 [48] integrates a frozen QFormer with additional learnable layers for enhanced performance.

While successful in diverse tasks, current multimodal models are primarily designed under the assumption of a one-to-one relationship between texts and image inputs. In reality, the relationship between text and images can be one-to-many or many-to-many. Effectively applying multimodal models in such scenarios poses an open challenge.