UAE Researchers Say Audio is the Secret Sauce in Helping AI Understand Videos

Table of Links

• Abstract and 1 Introduction
• 2. Related Works
• 3. PG-Video-LLaVA
• 3.1. Overview
• 3.2. Architecture
• 4. Experiments
• 4.1. Implementation Details
• 4.2. Stronger Baseline
• 4.3. Spatial Grounding in Videos
• 4.4. Zero-Shot Visual Question Answering
• 5. Conclusion and References

Supplementary Material

• A. Audio Modality Integration
• B. Visual Grounding: Quantitative Evaluation
• C. Qualitative Results for Visual Grounding
• D. Quantitative Evaluations of Video-based Conversation Performance

4.4. Zero-Shot Visual Question Answering

For PG-Video-LLaVA, zero-shot question-answering (QA) capabilities were evaluated quantitatively using several established open-ended QA datasets: MSRVTT-QA[40], MSVD-QA [39], TGIF-QA [16], and ActivityNet-QA [44]. These datasets are benchmarks for assessing a model’s ability to generate accurate answers without any datasetspecific fine-tuning. We adopted a zero-shot evaluation methodology, utilizing Vicuna-13b-v1.5 to evaluate the model’s understanding and predictive accuracy, with scores

assigned on a scale from 1 to 5. Results are presented in Table 3.

In comparison to Video-ChatGPT, PG-Video-LLaVA demonstrates superior performance, surpassing not only the predecessor but also other notable models in the field, such as FrozenBiLM[41] and Video Chat[15]. The results from our evaluations indicate that PG-Video-LLaVA has significantly enhanced its ability to comprehend video content and generate contextually relevant answers, thus establishing a new state-of-the-art in zero-shot VideoQA.

As shown in Figure 4, our method is able to visually ground the key objects in the given video. Improvement in the model’s capability to describe the content in the video is demonstrated in Figure 3. Further, it can be observed that adding the audio modality helps make correct outputs, whereas the model without audio modality fails to capture those details from visual content alone (Figure 5).