Authors:
(1) Samson Yu, Dept. of Computer Science, National University of Singapore (samson.yu@u.nus.edu);
(2) Kelvin Lin. Dept. of Computer Science, National University of Singapore;
(3) Anxing Xiao, Dept. of Computer Science, National University of Singapore;
(4) Jiafei Duan, University of Washington;
(5) Harold Soh, Dept. of Computer Science, National University of Singapore and NUS Smart Systems Institute (harold@comp.nus.edu.sg). Authors: Authors: (1) Samson Yu, Dept. of Computer Science, National University of Singapore (samson.yu@u.nus.edu); (2) Kelvin Lin. Dept. of Computer Science, National University of Singapore; (3) Anxing Xiao, Dept. of Computer Science, National University of Singapore; (4) Jiafei Duan, University of Washington; (5) Harold Soh, Dept. of Computer Science, National University of Singapore and NUS Smart Systems Institute (harold@comp.nus.edu.sg). Table of Links Abstract and I. Introduction
II. Related Work
III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite
IV. Octopi - Vision-Language Property-Guided Physical Reasoning
V. Experimental Setup
VI. Experimental Results
VII. Ablations
VIII. Conclusion and Discussion, Acknowledgements, and References

Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Abstract and I. Introduction Abstract and I. Introduction II. Related Work II. Related Work III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite IV. Octopi - Vision-Language Property-Guided Physical Reasoning IV. Octopi - Vision-Language Property-Guided Physical Reasoning V. Experimental Setup V. Experimental Setup VI. Experimental Results VI. Experimental Results VII. Ablations VII. Ablations VIII. Conclusion and Discussion, Acknowledgements, and References

Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION VIII. Conclusion and Discussion, Acknowledgements, and References Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION APPENDIX A: ANNOTATION DETAILS APPENDIX B: OBJECT DETAILS APPENDIX C: PROPERTY STATISTICS APPENDIX D: SAMPLE VIDEO STATISTICS APPENDIX E: ENCODER ANALYSIS APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION V. EXPERIMENTAL SETUP In this section, we evaluate the physical property prediction and reasoning capabilities of our proposed method. We design several experiments to answer the following questions: Are our physical property predictions useful for OCTOPI to reason about everyday scenarios?


Can OCTOPI be used in real robots to help them accomplish tasks using physical reasoning?


Can OCTOPI’s understanding of the physical properties generalize to the unseen daily life objects? Are our physical property predictions useful for OCTOPI to reason about everyday scenarios? Are our physical property predictions useful for OCTOPI to reason about everyday scenarios? Can OCTOPI be used in real robots to help them accomplish tasks using physical reasoning? Can OCTOPI be used in real robots to help them accomplish tasks using physical reasoning? Can OCTOPI’s understanding of the physical properties generalize to the unseen daily life objects? Can OCTOPI’s understanding of the physical properties generalize to the unseen daily life objects? A. Data Processing A. Data Processing The tactile videos were processed into frames. To focus on a few salient frames for better efficiency, we selected frames that have the top 30% total pixel intensity difference with their preceding frames. We randomly sampled 5 frames from these salient frames during training and selected 5 frames at uniform intervals from the first salient frame during evaluation. Data augmentation was performed during training in the form of random horizontal and vertical flips with 50% chance for each flip. B. Training Hyperparameters B. Training Hyperparameters C. Training Requirements C. Training Requirements Encoder fine-tuning took 6 hours and required less than 5GB of GPU VRAM. Tactile feature alignment together with end-to-end fine-tuning took 5 hours for OCTOPI-7b and 6.5 hours for OCTOPI-13b. We used 1 NVIDIA RTX A6000 for OCTOPI-7b and 2 NVIDIA RTX A6000s for OCTOPI-13b. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

This AI Learns to Handle the Unknown—By Touch Alone

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Formalization of the SylloBio-NLI Resource Generation Process

Unlocking the Future of AI: Active Inference vs. LLMs

Is 2024 The Year of Advanced Robotics?

Building Embodied Conversational AI: How We Taught a Robot to Understand, Navigate, and Interact

Researchers Combine Touch and Language to Boost Robotic Reasoning

Researchers Bridge Vision-Language Models and Touch with New Framework OCTOPI

A Formalization of the SylloBio-NLI Resource Generation Process

Unlocking the Future of AI: Active Inference vs. LLMs

Is 2024 The Year of Advanced Robotics?

Building Embodied Conversational AI: How We Taught a Robot to Understand, Navigate, and Interact

Researchers Combine Touch and Language to Boost Robotic Reasoning

Researchers Bridge Vision-Language Models and Touch with New Framework OCTOPI

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps