Table of Links Abstract and 1 Introduction Abstract and 1 Introduction Related Works
2.1. Vision-and-Language Navigation
2.2. Semantic Scene Understanding and Instance Segmentation
2.3. 3D Scene Reconstruction


Methodology
3.1. Data Collection
3.2. Open-set Semantic Information from Images
3.3. Creating the Open-set 3D Representation
3.4. Language-Guided Navigation


Experiments
4.1. Quantitative Evaluation
4.2. Qualitative Results


Conclusion and Future Work, Disclosure statement, and References Related Works
2.1. Vision-and-Language Navigation
2.2. Semantic Scene Understanding and Instance Segmentation
2.3. 3D Scene Reconstruction Related Works 2.1. Vision-and-Language Navigation 2.1. Vision-and-Language Navigation 2.2. Semantic Scene Understanding and Instance Segmentation 2.2. Semantic Scene Understanding and Instance Segmentation 2.3. 3D Scene Reconstruction 2.3. 3D Scene Reconstruction Methodology
3.1. Data Collection
3.2. Open-set Semantic Information from Images
3.3. Creating the Open-set 3D Representation
3.4. Language-Guided Navigation Methodology Methodology 3.1. Data Collection 3.1. Data Collection 3.2. Open-set Semantic Information from Images 3.2. Open-set Semantic Information from Images 3.3. Creating the Open-set 3D Representation 3.3. Creating the Open-set 3D Representation 3.4. Language-Guided Navigation 3.4. Language-Guided Navigation Experiments
4.1. Quantitative Evaluation
4.2. Qualitative Results Experiments Experiments 4.1. Quantitative Evaluation 4.1. Quantitative Evaluation 4.2. Qualitative Results 4.2. Qualitative Results Conclusion and Future Work, Disclosure statement, and References Conclusion and Future Work, Disclosure statement, and References Conclusion and Future Work, Disclosure statement, and References 3.4. Language-Guided Navigation In this section, we leverage the LLM-based approach from [1], which uses ChatGPT [35] to understand and map language commands to pre-defined function primitives that the robot can understand and execute. However, there are a few differences between our current approach and the approach in [1] regarding the use case of the LLM and the implementation of our function primitives. The previous approach used the LLM’s ability to bring in an open-set understanding by mapping general queries to the already-known closed-set class labels obtained via Mask2Former [7]. However, given the open-set nature of our new representation, O3D-SIM, the LLM does not need to do that. Figure 4 shows both approaches’ code output differences. The function primitives work similarly to the older approach, requiring the desired object type and its instance as an input. But now, the desired object is not from a pre-defined set of classes but a small query defining the object, so the implementation to find the desired location changes. We use the text and image-aligned nature of CLIP embeddings to find the desired object, where the input description is passed to the model, and its corresponding embedding is used to find the object in O3D-SIM. A cosine similarity is calculated between the embedding of the description and all the embeddings of our representation. These are ranked in a decreasing order, and the desired instance is selected. Once the instance is finalized, a goal corresponding to this instance is generated and passed to the navigation stack for autonomous navigation of the robot, hence achieving Language-Guided Navigation. Authors:
(1) Laksh Nanwani, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work;
(2) Kumaraditya Gupta, International Institute of Information Technology, Hyderabad, India;
(3) Aditya Mathur, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work;
(4) Swayam Agrawal, International Institute of Information Technology, Hyderabad, India;
(5) A.H. Abdul Hafez, Hasan Kalyoncu University, Sahinbey, Gaziantep, Turkey;
(6) K. Madhava Krishna, International Institute of Information Technology, Hyderabad, India. Authors: Authors: (1) Laksh Nanwani, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work; (2) Kumaraditya Gupta, International Institute of Information Technology, Hyderabad, India; (3) Aditya Mathur, International Institute of Information Technology, Hyderabad, India; this author contributed equally to this work; (4) Swayam Agrawal, International Institute of Information Technology, Hyderabad, India; (5) A.H. Abdul Hafez, Hasan Kalyoncu University, Sahinbey, Gaziantep, Turkey; (6) K. Madhava Krishna, International Institute of Information Technology, Hyderabad, India. This paper is available on arxiv under CC by-SA 4.0 Deed (Attribution-Sharealike 4.0 International) license. This paper is available on arxiv under CC by-SA 4.0 Deed (Attribution-Sharealike 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Quant

VLN: LLM and CLIP for Instance-Specific Navigation on 3D Maps

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3D Mapping Initialization: Using RGB-D Images and Camera Parameters

The Next Frontier of Autonomous Navigation for UAVs: A Hybrid Approach to GPS-Denied Environments

Evaluating Novel 3D Semantic Instance Map for Vision-Language Navigation

Visualizing Embedding Alignment: UMAP Dimensionality Reduction in MindEye2

Semantic Instance Extraction: CLIP and DINO Features for 3D Mapping

Simplifying Vector Search: Part 1

3D Mapping Initialization: Using RGB-D Images and Camera Parameters

The Next Frontier of Autonomous Navigation for UAVs: A Hybrid Approach to GPS-Denied Environments

Evaluating Novel 3D Semantic Instance Map for Vision-Language Navigation

Visualizing Embedding Alignment: UMAP Dimensionality Reduction in MindEye2

Semantic Instance Extraction: CLIP and DINO Features for 3D Mapping

Simplifying Vector Search: Part 1

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps