Table of Links Abstract and 1 Introduction Abstract and 1 Introduction Related Work
2.1. Multimodal Learning
2.2. Multiple Instance Learning


Methodology
3.1. Preliminaries and Notations
3.2. Relations between Attention-based VPG and MIL
3.3. MIVPG for Multiple Visual Inputs
3.4. Unveiling Instance Correlation in MIVPG for Enhanced Multi-instance Scenarios


Experiments and 4.1. General Setup
4.2. Scenario 1: Samples with Single Image
4.3. Scenario 2: Samples with Multiple Images, with Each Image as a General Embedding
4.4. Scenario 3: Samples with Multiple Images, with Each Image Having Multiple Patches to be Considered and 4.5. Case Study


Conclusion and References Related Work
2.1. Multimodal Learning
2.2. Multiple Instance Learning Related Work 2.1. Multimodal Learning 2.1. Multimodal Learning 2.2. Multiple Instance Learning 2.2. Multiple Instance Learning Methodology
3.1. Preliminaries and Notations
3.2. Relations between Attention-based VPG and MIL
3.3. MIVPG for Multiple Visual Inputs
3.4. Unveiling Instance Correlation in MIVPG for Enhanced Multi-instance Scenarios Methodology 3.1. Preliminaries and Notations 3.1. Preliminaries and Notations 3.2. Relations between Attention-based VPG and MIL 3.2. Relations between Attention-based VPG and MIL 3.3. MIVPG for Multiple Visual Inputs 3.3. MIVPG for Multiple Visual Inputs 3.4. Unveiling Instance Correlation in MIVPG for Enhanced Multi-instance Scenarios 3.4. Unveiling Instance Correlation in MIVPG for Enhanced Multi-instance Scenarios Experiments and 4.1. General Setup
4.2. Scenario 1: Samples with Single Image
4.3. Scenario 2: Samples with Multiple Images, with Each Image as a General Embedding
4.4. Scenario 3: Samples with Multiple Images, with Each Image Having Multiple Patches to be Considered and 4.5. Case Study Experiments and 4.1. General Setup Experiments and 4.1. General Setup 4.2. Scenario 1: Samples with Single Image 4.2. Scenario 1: Samples with Single Image 4.3. Scenario 2: Samples with Multiple Images, with Each Image as a General Embedding 4.3. Scenario 2: Samples with Multiple Images, with Each Image as a General Embedding 4.4. Scenario 3: Samples with Multiple Images, with Each Image Having Multiple Patches to be Considered and 4.5. Case Study 4.4. Scenario 3: Samples with Multiple Images, with Each Image Having Multiple Patches to be Considered and 4.5. Case Study Conclusion and References Conclusion and References Conclusion and References Supplementary Material Supplementary Material A. Detailed Architecture of QFormer A. Detailed Architecture of QFormer B. Proof of Proposition B. Proof of Proposition C. More Experiments C. More Experiments A. Detailed Architecture of QFormer The architecture overview is depicted in Figure 7. Specifically, QFormer is initialized as a BERT-based model[8] comprising a total of L = 12 layers. In contrast to typical BERT models that process textual inputs, QFormer takes R = 32 learnable query embeddings as inputs. These embeddings are utilized to extract visual information from the input visual data during Stage-1 pretraining in BLIP2[22]. Subsequently, they serve as visual prompt embeddings for the LLM inputs after projection. Inside the QFormer, each layer includes a self-attention module composed of a Multi-Head Attention component and a Forward module (consisting of Linear, LayerNorm, and Residual Connection). The cross-attention module, initialized with random values, is inserted every G layers, where learnable query embeddings interact with visual embeddings. In the main paper, for the sake of conciseness, we condensed the representation of the multi-head attention and forward modules into self(cross) attention modules. Furthermore, we exclusively illustrated the modifications made to the cross-attention module in MIVPG, as the self-attention modules remain unchanged. The final QFormer output is represented by the last layer’s query embeddings. In the main paper, for the sake of conciseness, we condensed the representation of the multi-head attention and forward modules into self(cross) attention modules. Furthermore, we exclusively illustrated the modifications made to the cross-attention module in MIVPG, as the self-attention modules remain unchanged. The final QFormer output is represented by the last layer’s query embeddings. For a more comprehensive understanding, readers are encouraged to refer to [22]. Authors:
(1) Wenliang Zhong, The University of Texas at Arlington (wxz9204@mavs.uta.edu);
(2) Wenyi Wu, Amazon (wenyiwu@amazon.com);
(3) Qi Li, Amazon (qlimz@amazon.com);
(4) Rob Barton, Amazon (rab@amazon.com);
(5) Boxin Du, Amazon (boxin@amazon.com);
(6) Shioulin Sam, Amazon (shioulin@amazon.com);
(7) Karim Bouyarmane, Amazon (bouykari@amazon.com);
(8) Ismail Tutar, Amazon (ismailt@amazon.com);
(9) Junzhou Huang, The University of Texas at Arlington (jzhuang@uta.edu). Authors: Authors: (1) Wenliang Zhong, The University of Texas at Arlington (wxz9204@mavs.uta.edu); (2) Wenyi Wu, Amazon (wenyiwu@amazon.com); (3) Qi Li, Amazon (qlimz@amazon.com); (4) Rob Barton, Amazon (rab@amazon.com); (5) Boxin Du, Amazon (boxin@amazon.com); (6) Shioulin Sam, Amazon (shioulin@amazon.com); (7) Karim Bouyarmane, Amazon (bouykari@amazon.com); (8) Ismail Tutar, Amazon (ismailt@amazon.com); (9) Junzhou Huang, The University of Texas at Arlington (jzhuang@uta.edu). This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Amazon

Visual Prompt Generation: Cross-Attention in Q-Former

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Coarse-to-Fine Approach to Text-Guided 3D Localization Without Ground Truth

Evaluating Visual Adapters: MIVPG Performance on Single and Multi-Image Inputs

MIVPG on E-commerce: Multi-Image/Multi-Patch Aggregation for Captioning

Visual Prompt Generators (VPGs): Encoding Images to LLM Tokens

MIL Perspective: Analyzing Q-Former as a Multi-Head Mechanism

Multimodal Fusion: MIVPG's Hierarchical MIL Approach for Multi-Image Samples

A Coarse-to-Fine Approach to Text-Guided 3D Localization Without Ground Truth

Evaluating Visual Adapters: MIVPG Performance on Single and Multi-Image Inputs

MIVPG on E-commerce: Multi-Image/Multi-Patch Aggregation for Captioning

Visual Prompt Generators (VPGs): Encoding Images to LLM Tokens

MIL Perspective: Analyzing Q-Former as a Multi-Head Mechanism

Multimodal Fusion: MIVPG's Hierarchical MIL Approach for Multi-Image Samples

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps