Authors:
(1) Samson Yu, Dept. of Computer Science, National University of Singapore (samson.yu@u.nus.edu);
(2) Kelvin Lin. Dept. of Computer Science, National University of Singapore;
(3) Anxing Xiao, Dept. of Computer Science, National University of Singapore;
(4) Jiafei Duan, University of Washington;
(5) Harold Soh, Dept. of Computer Science, National University of Singapore and NUS Smart Systems Institute (harold@comp.nus.edu.sg). Authors: Authors: (1) Samson Yu, Dept. of Computer Science, National University of Singapore (samson.yu@u.nus.edu); (2) Kelvin Lin. Dept. of Computer Science, National University of Singapore; (3) Anxing Xiao, Dept. of Computer Science, National University of Singapore; (4) Jiafei Duan, University of Washington; (5) Harold Soh, Dept. of Computer Science, National University of Singapore and NUS Smart Systems Institute (harold@comp.nus.edu.sg). Table of Links Abstract and I. Introduction
II. Related Work
III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite
IV. Octopi - Vision-Language Property-Guided Physical Reasoning
V. Experimental Setup
VI. Experimental Results
VII. Ablations
VIII. Conclusion and Discussion, Acknowledgements, and References

Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Abstract and I. Introduction Abstract and I. Introduction II. Related Work II. Related Work III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite III. PhysiClear - Tactile and Physical Understanding Training & Evaluation Suite IV. Octopi - Vision-Language Property-Guided Physical Reasoning IV. Octopi - Vision-Language Property-Guided Physical Reasoning V. Experimental Setup V. Experimental Setup VI. Experimental Results VI. Experimental Results VII. Ablations VII. Ablations VIII. Conclusion and Discussion, Acknowledgements, and References

Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION VIII. Conclusion and Discussion, Acknowledgements, and References Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models

APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION Appendix for Octopi: Object Property Reasoning with Large Tactile-Language Models APPENDIX A: ANNOTATION DETAILS
APPENDIX B: OBJECT DETAILS
APPENDIX C: PROPERTY STATISTICS
APPENDIX D: SAMPLE VIDEO STATISTICS
APPENDIX E: ENCODER ANALYSIS
APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION APPENDIX A: ANNOTATION DETAILS APPENDIX B: OBJECT DETAILS APPENDIX C: PROPERTY STATISTICS APPENDIX D: SAMPLE VIDEO STATISTICS APPENDIX E: ENCODER ANALYSIS APPENDIX F: PG-INSTRUCTBLIP AVOCADO PROPERTY PREDICTION VI. EXPERIMENTAL RESULTS To address the above questions, we evaluated OCTOPI using (i) accuracy on the physical understanding tasks in PHYSICLEAR’s test set, (ii) accuracy on scenario reasoning tasks, (iii) task success rate on a real robot, and (iv) property prediction accuracy on unseen objects. We tested two versions of OCTOPI, OCTOPI-7b and OCTOPI-13b, which use Vicuna7b v1.5 and Vicuna-13b v1.5 as their LLMs respectively. A. Tactile-grounded Physical Understanding with Object Property Descriptions A. Tactile-grounded Physical Understanding with Object Property Descriptions During tactile feature alignment and end-to-end fine-tuning, we trained OCTOPI with comparison tasks (i.e. PC, PSS and POM) to align its physical understanding of our physical properties and objects with our labels. We evaluated OCTOPI’s physical understanding with the same single-step prompts used during training and on 500 question-answer pairs in total across the three tasks. The results for physical understanding of unseen test objects are shown in Table VI. Our results show that both OCTOPI-7b and OCTOPI-13b perform well on all three physical understanding tasks when they are trained to predict property descriptions. Using physical property descriptions, OCTOPI-7b achieves accuracies of 48.10% on PC, 74.67% on PSS and 44.39% on POM. OCTOPI13b outperforms OCTOPI-7b by 6.96% on PC, 9.33% on PSS and 16.04% on POM. This suggests that OCTOPI’s physical understanding improves significantly with LLM size. Further, we explored the effect of using physical property descriptions by fine-tuning both OCTOPI-7b and OCTOPI13b on the physical understanding tasks without intermediate physical property predictions. We found that predictions based on object properties notably improve physical understanding in both OCTOPI-7b and OCTOPI-13b. B. Scenario Reasoning B. Scenario Reasoning We assessed the usefulness of our physical property categories by testing how OCTOPI can reason about everyday scenarios using the physical properties. For reference, the different scenario questions are provided in Table V with the prompts shown in Table IV. Our results are summarized in Table VII. For both OCTOPI7b and OCTOPI-13b, including the object property significantly improves performance, which supports our overall hypothesis that leveraging these properties is helpful for these tasks. Interestingly, we observed that the 7b model marginally outperformed the 13b model. We provide two qualitative examples to show OCTOPI-13b performing commonsense physical reasoning effectively. In the first task, we provide a tactile video of a scoop of uncooked rice and first instruct it to describe the tactile video. We then follow up with an instruction to OCTOPI-13b to determine if the rice is uncooked or cooked. OCTOPI-13b is able to reason that the scoop of rice is uncooked due to its rough surface, as shown in Fig. 4. Next, we gave OCTOPI-13b two tactile videos corresponding to two different parts of the same toothbrush - the handle and the bristles. It is instructed to describe both objects using the physical properties. We then instruct it to determine which tactile video belongs to each object part using the physical properties. Fig. 5 shows that OCTOPI-13b is able to reason about the property-object match correctly. C. Avocado Ripeness Classification C. Avocado Ripeness Classification To evaluate OCTOPI’s usefulness as a tactile-grounded physical reasoning system for real world tasks, we integrated two GelSight sensors on a 7-DoF Franka Emika Panda robot and used it for avocado ripeness classification. While ripe avocados generally appear in a shade of brown, their ripeness is difficult to determine using vision alone. At the same time, ripe avacados are softer then unripe ones and thus, tactile sensations can improve classification. We performed property prediction and ripeness classification evaluations using a set of 10 avocados with 20 tactile samples collected from each avocado (i.e. 200 total samples). During ripeness classification, 100 pairs of avocado samples were selected and OCTOPI was tasked to identify which avocado is riper. Each pair contains avocados in different stages of ripeness. At test time, the Franka robot grasped each avocado once to collect the tactile readings, before passing it to the model. The model gave instructions on which avocado to place in the ripe bin, and this was relayed through ROS for execution. Table VIII shows OCTOPI-13b’s performance on both avocado property prediction and avocado ripeness classification. We split the evaluation into two parts and evaluate the avocado property prediction and ripeness classification separately. For avocado property prediction, OCTOPI-13b achieves an accuracy of 35.50%, which is significantly higher than that of the random baseline (3.70%). Unlike the PHYSICLEAR dataset, these tactile videos are collected with only pressing and without any rotation. This suggests that OCTOPI-13b’s physical property prediction capability is robust to differences in tactile exploratory procedures. OCTOPI-13b obtains a ripeness prediction accuracy of 63.00%. To set up this task, we first elicit OCTOPI’s commonsense reasoning for avocado ripeness using our properties by querying “You will be given tactile descriptions that consist of three physical properties: hardness, roughness, bumpiness. Hardness categories in increasing hardness: soft, moderately hard, hard. Roughness categories in increasing roughness: smooth, slightly smooth, rough. Bumpiness categories in increasing bumpiness: no bumps, small bumps, big bumps. Which of these properties help to determine avocado ripeness? Rank them.” OCTOPI-13b responds with “The properties that help determine avocado ripeness are hardness and bumpiness. Roughness is not a reliable indicator of avocado ripeness. In terms of hardness, a ripe avocado will be moderately hard, while an unripe avocado will be hard. In terms of bumpiness, a ripe avocado will have small bumps, while an unripe avocado will have no bumps or small bumps.” This exchange is provided as context to OCTOPI-13b during ripeness classification. For comparison, we evaluated a physically-grounded visionlanguage model, PG-InstructBLIP [17], on property predictions of our avocados. PG-InstructBLIP was trained to infer a predetermined set of physical properties from visual images of real objects in the EgoObjects dataset [65]. Table VIII shows PG-InstructBLIP’s performance on property prediction for our avocados was poor. Possible reasons for this are that (i) the definitions of the physical properties may not be wellaligned with PHYSICLEAR, and/or (ii) the physical properties of avocados are not clearly apparent using only the visual modality. We could not coax the PG-InstructBLIP model to directly classify avocado ripeness despite trying various prompts; it would always pick the first object. D. Object Property Description Prediction D. Object Property Description Prediction The physical understanding and scenario reasoning capabilities of OCTOPI depends on its initial physical property predictions. We evaluated OCTOPI’s physical property prediction on the PHYSICLEAR test set and show the results in Table IX. Both OCTOPI-7b and OCTOPI-13b perform well above the random baseline for combined and individual property prediction and have similar performance to the fine-tuned CLIP model, indicating that OCTOPI can be used for object property prediction. OCTOPI-13b has a higher combined accuracy (i.e. all three physical properties are correctly predicted for a given object) when compared to OCTOPI-7b, suggesting there are performance gains with larger LLMs for tactile signal grounding (apart from the bumpiness property). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Arweave

Scenario

Can Smaller AI Outperform the Giants?

Your Next Robot Might Think with Its Fingers

How the Most Important Models Actually Work

Too Long; Didn't Read

Make resilience your competitive advantage

AI Learns Common Sense from Touch, Not Just Vision

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Prompt Patterns That Decide If an AI Is “Correct” or “Wrong”

Unlocking the Future of AI: Active Inference vs. LLMs

Is 2024 The Year of Advanced Robotics?

Building Embodied Conversational AI: How We Taught a Robot to Understand, Navigate, and Interact

Researchers Combine Touch and Language to Boost Robotic Reasoning

The Prompt Patterns That Decide If an AI Is “Correct” or “Wrong”

Unlocking the Future of AI: Active Inference vs. LLMs

Is 2024 The Year of Advanced Robotics?

Building Embodied Conversational AI: How We Taught a Robot to Understand, Navigate, and Interact

Researchers Combine Touch and Language to Boost Robotic Reasoning

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps