Authors:
(1) Shehan Munasinghe, Mohamed bin Zayed University of AI and Equal Contribution;
(2) Rusiru Thushara, Mohamed bin Zayed University of AI and Equal Contribution;
(3) Muhammad Maaz, Mohamed bin Zayed University of AI;
(4) Hanoona Abdul Rasheed, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Mubarak Shah, University of Central Florida;
(7) Fahad Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 8 of 10 of a study detailing the development of a smarter AI model for videos. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Works


PG-Video-LLaVA


3.1. Overview
3.2. Architecture


Experiments


4.1. Implementation Details
4.2. Stronger Baseline
4.3. Spatial Grounding in Videos
4.4. Zero-Shot Visual Question Answering
5. Conclusion and References Supplementary Material A. Audio Modality Integration
B. Visual Grounding: Quantitative Evaluation
C. Qualitative Results for Visual Grounding
D. Quantitative Evaluations of Video-based Conversation Performance 4.4. Zero-Shot Visual Question Answering For PG-Video-LLaVA, zero-shot question-answering (QA) capabilities were evaluated quantitatively using several established open-ended QA datasets: MSRVTT-QA[40], MSVD-QA [39], TGIF-QA [16], and ActivityNet-QA [44]. These datasets are benchmarks for assessing a model’s ability to generate accurate answers without any datasetspecific fine-tuning. We adopted a zero-shot evaluation methodology, utilizing Vicuna-13b-v1.5 to evaluate the model’s understanding and predictive accuracy, with scores assigned on a scale from 1 to 5. Results are presented in Table 3. In comparison to Video-ChatGPT, PG-Video-LLaVA demonstrates superior performance, surpassing not only the predecessor but also other notable models in the field, such as FrozenBiLM[41] and Video Chat[15]. The results from our evaluations indicate that PG-Video-LLaVA has significantly enhanced its ability to comprehend video content and generate contextually relevant answers, thus establishing a new state-of-the-art in zero-shot VideoQA. As shown in Figure 4, our method is able to visually ground the key objects in the given video. Improvement in the model’s capability to describe the content in the video is demonstrated in Figure 3. Further, it can be observed that adding the audio modality helps make correct outputs, whereas the model without audio modality fails to capture those details from visual content alone (Figure 5). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Shehan Munasinghe, Mohamed bin Zayed University of AI and Equal Contribution; (2) Rusiru Thushara, Mohamed bin Zayed University of AI and Equal Contribution; (3) Muhammad Maaz, Mohamed bin Zayed University of AI; (4) Hanoona Abdul Rasheed, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Mubarak Shah, University of Central Florida; (7) Fahad Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Shehan Munasinghe, Mohamed bin Zayed University of AI and Equal Contribution; (2) Rusiru Thushara, Mohamed bin Zayed University of AI and Equal Contribution; (3) Muhammad Maaz, Mohamed bin Zayed University of AI; (4) Hanoona Abdul Rasheed, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Mubarak Shah, University of Central Florida; (7) Fahad Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 8 of 10 of a study detailing the development of a smarter AI model for videos. Read the rest below. Editor's Note: This is Part 8 of 10 of a study detailing the development of a smarter AI model for videos. Read the rest below. Editor's Note: This is Part 8 of 10 of a study detailing the development of a smarter AI model for videos. Read the rest below. Editor's Note: This is Part 8 of 10 of a study detailing the development of a smarter AI model for videos. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Works PG-Video-LLaVA 3.1. Overview 3.2. Architecture Experiments 4.1. Implementation Details 4.2. Stronger Baseline 4.3. Spatial Grounding in Videos 4.4. Zero-Shot Visual Question Answering 5. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Works 2. Related Works PG-Video-LLaVA PG-Video-LLaVA PG-Video-LLaVA 3.1. Overview 3.1. Overview 3.2. Architecture 3.2. Architecture Experiments Experiments Experiments 4.1. Implementation Details 4.1. Implementation Details 4.2. Stronger Baseline 4.2. Stronger Baseline 4.3. Spatial Grounding in Videos 4.3. Spatial Grounding in Videos 4.4. Zero-Shot Visual Question Answering 4.4. Zero-Shot Visual Question Answering 5. Conclusion and References 5. Conclusion and References Supplementary Material Supplementary Material A. Audio Modality Integration B. Visual Grounding: Quantitative Evaluation C. Qualitative Results for Visual Grounding D. Quantitative Evaluations of Video-based Conversation Performance A. Audio Modality Integration A. Audio Modality Integration B. Visual Grounding: Quantitative Evaluation B. Visual Grounding: Quantitative Evaluation C. Qualitative Results for Visual Grounding C. Qualitative Results for Visual Grounding D. Quantitative Evaluations of Video-based Conversation Performance D. Quantitative Evaluations of Video-based Conversation Performance 4.4. Zero-Shot Visual Question Answering For PG-Video-LLaVA, zero-shot question-answering (QA) capabilities were evaluated quantitatively using several established open-ended QA datasets: MSRVTT-QA[40], MSVD-QA [39], TGIF-QA [16], and ActivityNet-QA [44]. These datasets are benchmarks for assessing a model’s ability to generate accurate answers without any datasetspecific fine-tuning. We adopted a zero-shot evaluation methodology, utilizing Vicuna-13b-v1.5 to evaluate the model’s understanding and predictive accuracy, with scores assigned on a scale from 1 to 5. Results are presented in Table 3. In comparison to Video-ChatGPT, PG-Video-LLaVA demonstrates superior performance, surpassing not only the predecessor but also other notable models in the field, such as FrozenBiLM[41] and Video Chat[15]. The results from our evaluations indicate that PG-Video-LLaVA has significantly enhanced its ability to comprehend video content and generate contextually relevant answers, thus establishing a new state-of-the-art in zero-shot VideoQA. As shown in Figure 4, our method is able to visually ground the key objects in the given video. Improvement in the model’s capability to describe the content in the video is demonstrated in Figure 3. Further, it can be observed that adding the audio modality helps make correct outputs, whereas the model without audio modality fails to capture those details from visual content alone (Figure 5). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

What Does It Take to Make AI Watch, Listen, and Understand Videos Like Us?

Researchers in UAE Discover New System That Helps AI Understand Videos

AutoEncoder.tech

Too Long; Didn't Read

Make resilience your competitive advantage

UAE Researchers Say Audio is the Secret Sauce in Helping AI Understand Videos

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Improving Text Embeddings with Large Language Models: Training

This New AI Doesn’t Just Watch Videos—It Listens, Learns, and Talks Back Too

UAE Researchers Create First AI That Pinpoints Objects in Videos, Down to the Pixel

AI Just Got Better at Watching Videos

UAE Researchers Teach AI to Watch, Listen, and Understand Videos Like Humans

Improving Text Embeddings with Large Language Models: Training

This New AI Doesn’t Just Watch Videos—It Listens, Learns, and Talks Back Too

UAE Researchers Create First AI That Pinpoints Objects in Videos, Down to the Pixel

AI Just Got Better at Watching Videos

UAE Researchers Teach AI to Watch, Listen, and Understand Videos Like Humans

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps