Table of Links Abstract and 1 Introduction Abstract and 1 Introduction Related Works
2.1. Vision-and-Language Navigation
2.2. Semantic Scene Understanding and Instance Segmentation
2.3. 3D Scene Reconstruction


Methodology
3.1. Data Collection
3.2. Open-set Semantic Information from Images
3.3. Creating the Open-set 3D Representation
3.4. Language-Guided Navigation


Experiments
4.1. Quantitative Evaluation
4.2. Qualitative Results


Conclusion and Future Work, Disclosure statement, and References Related Works
2.1. Vision-and-Language Navigation
2.2. Semantic Scene Understanding and Instance Segmentation
2.3. 3D Scene Reconstruction Related Works 2.1. Vision-and-Language Navigation 2.1. Vision-and-Language Navigation 2.2. Semantic Scene Understanding and Instance Segmentation 2.2. Semantic Scene Understanding and Instance Segmentation 2.3. 3D Scene Reconstruction 2.3. 3D Scene Reconstruction Methodology
3.1. Data Collection
3.2. Open-set Semantic Information from Images
3.3. Creating the Open-set 3D Representation
3.4. Language-Guided Navigation Methodology Methodology 3.1. Data Collection 3.1. Data Collection 3.2. Open-set Semantic Information from Images 3.2. Open-set Semantic Information from Images 3.3. Creating the Open-set 3D Representation 3.3. Creating the Open-set 3D Representation 3.4. Language-Guided Navigation 3.4. Language-Guided Navigation Experiments
4.1. Quantitative Evaluation
4.2. Qualitative Results Experiments Experiments 4.1. Quantitative Evaluation 4.1. Quantitative Evaluation 4.2. Qualitative Results 4.2. Qualitative Results Conclusion and Future Work, Disclosure statement, and References Conclusion and Future Work, Disclosure statement, and References Conclusion and Future Work, Disclosure statement, and References 2. Related Works 2.1. Vision-and-Language Navigation: Vision-and-Language Navigation (VLN) has recently gained much traction because of its potential to improve autonomous navigation by combining human natural language understanding and visual perception [2, 6, 11, 12]. The development of foundation models like CLIP [9], which combines image and text data to learn the rich representations of the environment, has spurred considerable progress in vision and language understanding. The multi-modal nature of these foundation models allows them to comprehend concepts in both text and image and even connect concepts between the two modalities. There have been increasing advancements to address the gaps faced by previous methods for VLN, like navigation efficiency to spatial goals specified by language commands and zero-shot spatial goal navigation given unseen language instructions [2]. A major challenge in VLN is interpreting language instructions in unfamiliar environments. A significant limitation of previous studies in this domain is their handling of action errors. If a robot agent makes an incorrect action, it risks failing to reach its destination or exploring unnecessary areas, leading to increased computational demands and possibly entering a state from which recovery is unfeasible. State-of-the-art VLN methodologies employ diverse strategies to excel in such scenarios. Some methods adopt a specialized pre-training and fine-tuning approach designed explicitly for VLN tasks, utilizing transformer-based architectures [13]. These strategies often involve using image-text-action triplets in a self-supervised learning context [14]. Other approaches refine the pre-training process to enhance VLN task performance, for instance, by emphasizing the learning of spatiotemporal visual-textual relationships to better utilize past observations for future action prediction [15, 16, 17]. Furthermore, contemporary VLN systems predominantly rely on simulations due to their dependency on panoramic views and extracting region features, which can be computationally prohibitive. In contrast, our work demonstrates our pipeline’s efficiency and computational viability with real-world data, underscoring its practical applicability. Given the recent advances in the semantic understanding of images, there has been an increasing interest in using semantic reasoning to improve exploration efficiency in novel environments and handling semantic goals specified via categories, images, and language instructions. Most of these methods are specialized to a single task, i.e. they are uni-modal. Recent works have also been on executing tasks with lifelong learning, which means taking advantage of experience in the same environment for multi-modal navigation [11]. One such task is that a robot must be able to reach any object specified in any way and remember object locations to return to them. The work done on these tasks also utilizes CLIP to align both image and language embeddings, where they match language goal descriptions with all instances in the environment using the cosine similarity score between their CLIP features. Authors:
(1) Laksh Nanwani, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work;
(2) Kumaraditya Gupta, International Institute of Information Technology, Hyderabad, India;
(3) Aditya Mathur, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work.
(4) Swayam Agrawal, International Institute of Information Technology, Hyderabad, India;
(5) A.H. Abdul Hafez, Hasan Kalyoncu University, Sahinbey, Gaziantep, Turkey;
(6) K. Madhava Krishna, International Institute of Information Technology, Hyderabad, India. Authors: Authors: (1) Laksh Nanwani, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work; (2) Kumaraditya Gupta, International Institute of Information Technology, Hyderabad, India; (3) Aditya Mathur, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work. (4) Swayam Agrawal, International Institute of Information Technology, Hyderabad, India; (5) A.H. Abdul Hafez, Hasan Kalyoncu University, Sahinbey, Gaziantep, Turkey; (6) K. Madhava Krishna, International Institute of Information Technology, Hyderabad, India. This paper is available on arxiv under CC by-SA 4.0 Deed (Attribution-Sharealike 4.0 International) license. This paper is available on arxiv under CC by-SA 4.0 Deed (Attribution-Sharealike 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Quant

VLN Research: Foundation Models, Challenges, and Semantic Reasoning

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3D Semantic Instance Maps: Leveraging Foundation Models for Language-Guided Navigation

Balancing AI Innovation & Regulation: Perspectives on Foundation Models and Responsible Development

Intro to Foundation AI Models: Types, Use Cases, and How to Get Started

Quality-Diversity through AI Feedback: Appendix

Revolutionizing Creative Text Generation with Quality-Diversity through AI Feedback

AI's Next Stop: A Copyright Showdown

3D Semantic Instance Maps: Leveraging Foundation Models for Language-Guided Navigation

Balancing AI Innovation & Regulation: Perspectives on Foundation Models and Responsible Development

Intro to Foundation AI Models: Types, Use Cases, and How to Get Started

Quality-Diversity through AI Feedback: Appendix

Revolutionizing Creative Text Generation with Quality-Diversity through AI Feedback

AI's Next Stop: A Copyright Showdown

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps