Authors:
(1) Arkadiy Saakyan, Columbia University (a.saakyan@cs.columbia.edu);
(2) Shreyas Kulkarni, Columbia University;
(3) Tuhin Chakrabarty, Columbia University;
(4) Smaranda Muresan, Columbia University. Authors: Authors: (1) Arkadiy Saakyan, Columbia University (a.saakyan@cs.columbia.edu); (2) Shreyas Kulkarni, Columbia University; (3) Tuhin Chakrabarty, Columbia University; (4) Smaranda Muresan, Columbia University. Editor's note: this is part 2 of 6 of a study looking at how well large AI models handle figurative language. Read the rest below. Editor's note: this is part 2 of 6 of a study looking at how well large AI models handle figurative language. Read the rest below. Editor's note: this is part 2 of 6 of a study looking at how well large AI models handle figurative language. Read the rest below. Editor's note: this is part 2 of 6 of a study looking at how well large AI models handle figurative language. Read the rest below. Table of Links Abstract and 1. Introduction
2. Related Work
3. V-FLUTE Task and Dataset

3.1 Metaphors and Similes
3.2 Idioms and 3.3 Sarcasm
3.4 Humor and 3.5 Dataset Statistics


4. Experiments and 4.1 Models

4.2 Automatic Metrics and 4.3 Automatic Evaluation Results
4.4 Human Baseline


5. Human Evaluation and Error Analysis

5.1 How Do Models Perform According to Humans?
5.2 What Errors Do Models Make? and 5.3 How Well Does the Explanation Score Predict Human Judgment on Adequacy?


6. Conclusions and References

A Dataset Statistics
B API models Hyperparameters
C Fine-tuning Hyperparameters
D Prompts for LLMs
E Model Taxonomy
F By-Phenomenon Performance
G Annotator Recruitment and Compensation Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. V-FLUTE Task and Dataset

3.1 Metaphors and Similes
3.2 Idioms and 3.3 Sarcasm
3.4 Humor and 3.5 Dataset Statistics 3. V-FLUTE Task and Dataset 3.1 Metaphors and Similes
3.2 Idioms and 3.3 Sarcasm
3.4 Humor and 3.5 Dataset Statistics 3.1 Metaphors and Similes 3.2 Idioms and 3.3 Sarcasm 3.4 Humor and 3.5 Dataset Statistics 4. Experiments and 4.1 Models

4.2 Automatic Metrics and 4.3 Automatic Evaluation Results
4.4 Human Baseline 4. Experiments and 4.1 Models 4.2 Automatic Metrics and 4.3 Automatic Evaluation Results
4.4 Human Baseline 4.2 Automatic Metrics and 4.3 Automatic Evaluation Results 4.4 Human Baseline 5. Human Evaluation and Error Analysis

5.1 How Do Models Perform According to Humans?
5.2 What Errors Do Models Make? and 5.3 How Well Does the Explanation Score Predict Human Judgment on Adequacy? 5. Human Evaluation and Error Analysis 5.1 How Do Models Perform According to Humans?
5.2 What Errors Do Models Make? and 5.3 How Well Does the Explanation Score Predict Human Judgment on Adequacy? 5.1 How Do Models Perform According to Humans? 5.2 What Errors Do Models Make? and 5.3 How Well Does the Explanation Score Predict Human Judgment on Adequacy? 6. Conclusions and References

A Dataset Statistics
B API models Hyperparameters
C Fine-tuning Hyperparameters
D Prompts for LLMs
E Model Taxonomy
F By-Phenomenon Performance
G Annotator Recruitment and Compensation 6. Conclusions and References A Dataset Statistics
B API models Hyperparameters
C Fine-tuning Hyperparameters
D Prompts for LLMs
E Model Taxonomy
F By-Phenomenon Performance
G Annotator Recruitment and Compensation A Dataset Statistics B API models Hyperparameters C Fine-tuning Hyperparameters D Prompts for LLMs E Model Taxonomy F By-Phenomenon Performance G Annotator Recruitment and Compensation 2 Related Work Textual entailment (MacCartney and Manning, 2008; Bowman et al., 2015) and visual entailment (Xie et al., 2019) tasks have been proposed to measure language and multimodal understanding. However, models trained to simply improve label accuracy on these data can be brittle and suffer from spurious correlations (Poliak et al., 2018; Gururangan et al., 2018; McCoy et al., 2019; Gardner et al., 2021). Datasets such as e-SNLI (Camburu et al., 2018) and e-SNLI-VE (Kayser et al., 2021) augment existing entailment datasets with natural language explanations and train models to not only predict the label, but also generate a textual explanation for the reason behind the prediction. Such approach has been further adopted for a variety of tasks, such as commonsense reasoning (Rajani et al., 2019; Aggarwal et al., 2021) and social norm understanding (CHWang et al., 2023) among others (Wiegreffe and Marasovic, 2021). This approach has been extended to assess LLMs’ capabilities on understanding figurative language through the FLUTE dataset (Chakrabarty et al., 2022). FLUTE frames figurative language understanding as an explainable textual entailment task. Recent progress in multimodal models (Li et al., 2022; Alayrac et al., 2022; OpenAI, 2023; Team et al., 2023; Liu et al., 2023b; Anthropic, 2024) prompts us to asses similar capabilities when extended to multimodal setting, testing the understanding of nonliteral meaning contained in both images and text. We present an equivalent of the FLUTE dataset for the visual modality: V-FLUTE. A number of previous works has focused on modeling figurative phenomena beyond text. Chakrabarty et al. (2023) use a human-AI collaboration framework to generate visual metaphors from linguistic metaphors (HAIVMet dataset) and propose a visual entailment task as an extrinsic evaluation of dataset quality. The dataset contains images, claims, and labels, but no textual explanations. Yosef et al. (2023) proposed a benchmark (IRFL) where given an idiom, metaphor, or simile the model has to distinguish which of the four associated images implies the figurative meaning of the expression. This dataset focuses on the figurative meaning in the textual modality and does not contain textual explanations. There has also been work on understanding multimodal sarcasm with explanations (Desai et al., 2022), mostly containing noisy user-generated text and crowdworkerwritten explanations. Other line of work has focused on understanding humor with multimodal models. MemeCap (Hwang and Shwartz, 2023) is a dataset for understanding memes. Hessel et al. (2023) release a corpus of annotated New Yorker Caption Contest entries, where the goal is to come up with a humorous captions for an image, with high-quality explanations for why the caption is humorous. The dataset is relatively limited in size containing only 520 unique instances in its training set. We leverage all these benchmarks to build V-FLUTE. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Researchers Combine GPT-4 and Human Experts to Train AI on Visual Figurative Reasoning

Can AI Understand a Joke? New Dataset Tests Bots on Metaphors, Sarcasm, and Humor

AI Models on HackerNoon

Too Long; Didn't Read

Make resilience your competitive advantage

New Dataset Challenges AI to Explain the Humor and Sarcasm It ‘Sees’ and ‘Reads’

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Beyond Credit Scores: Exploring the Potential of Verifiable Models in Diverse Industries

Building Machine Learning Algorithms That We Can Trust

Can AI Be Trusted?

Explainability Concerns Hide Poor AI

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Beyond Credit Scores: Exploring the Potential of Verifiable Models in Diverse Industries

Building Machine Learning Algorithms That We Can Trust

Can AI Be Trusted?

Explainability Concerns Hide Poor AI

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps