Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing, pages 3214–3225

August 1–6, 2021. ©2021 Association for Computational Linguistics

3214

MultiMET: A Multimodal Dataset for Metaphor Understanding

Dongyu Zhang1, Minghao Zhang1, Heting Zhang1, Liang Yang2, Hongfei LIN2

1Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province,School of Software, Dalian University of Technology, China

2School of Computer Science and Technology, Dalian University of Technology, Chinahflin@dlut.edu.cn

Abstract

Metaphor involves not only a linguistic phe-nomenon, but also a cognitive phenomenonstructuring human thought, which makes un-derstanding it challenging. As a means ofcognition, metaphor is rendered by more thantexts alone, and multimodal information inwhich vision/audio content is integrated withthe text can play an important role in ex-pressing and understanding metaphor. How-ever, previous metaphor processing and under-standing has focused on texts, partly due tothe unavailability of large-scale datasets withground truth labels of multimodal metaphor.In this paper, we introduce MultiMET, anovel multimodal metaphor dataset to facil-itate understanding metaphorical informationfrom multimodal text and image. It con-tains 10,437 text-image pairs from a range ofsources with multimodal annotations of theoccurrence of metaphors, domain relations,sentiments metaphors convey, and author in-tents. MultiMET opens the door to automaticmetaphor understanding by investigating mul-timodal cues and their interplay. Moreover, wepropose a range of strong baselines and showthe importance of combining multimodal cuesfor metaphor understanding. MultiMET willbe released publicly for research.

1 Introduction

Metaphor is frequently employed in human lan-guage and its ubiquity in everyday communicationhas been established in empirical studies (Cameron,2003; Steen, 2010; Shutova et al., 2010). SinceLakoff and Johnson (1980) introduced conceptualmetaphor theory (CMT), metaphor has been re-garded as not only a linguistic, but also a cogni-tive phenomenon for structuring human thought.Individuals use one usually concrete concept inmetaphors to render another usually abstract onefor reasoning and communication. For example,

(a) A fire in the sky tonight. (b) Smoking causes lung cancer.

Figure 1: Examples of multimodal metaphor

in the metaphorical utterance “knowledge is trea-sure,” knowledge is viewed in terms of treasureto express that knowledge can be valuable. Ac-cording to CMT, metaphor involves the mappingprocess by which a target domain is conceptualizedor understood in terms of a source domain.

As a means of cognition and communication,metaphor can occur in more modes than text alone.Multimodal information in which vision/audio con-tent is integrated with the text can also contributeto metaphoric conceptualization (Forceville andUrios-Aparisi, 2009; Ventola et al., 2004). A mul-timodal metaphor is defined as a mapping of do-mains from different modes such as text and image,text and sound, or image and sound (Forcevilleand Urios-Aparisi, 2009). For example, in Figure1 (a), the metaphorical message of fire in the skyis conveyed by a mapping between the target do-main “sky” (sunset) and the source domain “fire”from two modalities. Figure 1 (b) offers anotherexample with the metaphor of lungs made fromcigarettes so a relation is triggered between twodifferent entities, lung and cigarette, with the per-ceptual idea that smoking causes lung cancer. Thesource domain “cigarette” comes from the image,while the target domain “lung” appears in both textand image. Understanding multimodal metaphor

3215

requires decoding metaphorical messages and in-volves many cognitive efforts such as identifyingthe semantic relationship between two domains(Coulson and Van Petten, 2002; Yang et al., 2013),interpreting authorial intent from multimodal mes-sages (Evan Nelson, 2008), analyzing the sentimentmetaphors convey (Ervas, 2019), which might bedifficult for computers to do.

Qualitative studies have investigated the in-terplay between different modes underlying theunderstanding of multimodal metaphors in com-municative environments such as advertisements(Forceville et al., 2017; Urios-Aparisi, 2009),movies (Forceville, 2016; Kappelhoff and Muller,2011), songs (Forceville and Urios-Aparisi, 2009;Way and McKerrell, 2017), and cartoons (Refaie,2003; Xiufeng, 2013). In particular, with the de-velopment of mass communication, texts nowa-days are often combined with other modalities suchas images and videos to achieve a vivid, appeal-ing, persuasive, or aesthetic effect for the audience.This rapidly growing trend toward multimodalityrequires a shift to extend metaphor studies frommonomodality to multimodality, as well as fromtheory-driven analysis to data-driven empirical test-ing for in-depth metaphor understanding.

Despite the potential and importance of multi-modal information for metaphor research, therehas been little work on the automatic understand-ing of multimodal metaphors. While a numberof approaches to metaphor processing have beenproposed with a focus on text in the NLP com-munity (Shutova et al., 2010; Mohler et al., 2013;Jang et al., 2015, 2017; Shutova et al., 2017; Pra-manick et al., 2018; Liu et al., 2020), multimodalmetaphors have not received the full attention theydeserve, partly due to the severe lack of multimodalmetaphor datasets with their challenging and time-and labor-consuming creation.

To overcome the above limitations, we proposea novel multimodal metaphor dataset (MultiMET)consisting of text-image pairs (text and its corre-sponding image counterparts) manually annotatedfor metaphor understanding. MultiMET will ex-pand metaphor understanding from monomodalityto multimodality and help to improve the perfor-mance of automatic metaphor comprehension sys-tems by investigating multimodal cues. Our maincontributions are as follows:

• We create a novel multimodal dataset consist-ing of 10,437 text-image pair samples from

a range of resources including social media(Twitter and Facebook), and advertisements.MultiMET will be released publicly for re-search.

• We present fine-grain manual multimodal an-notations of the occurrence of metaphors,metaphor category, what sentiment metaphorsevoke, and author intent. The quality controland agreement analyses for multiple annota-tors are described.

• We quantitatively show the role of textualand visual modalities for metaphor detection;whether and to what extent metaphor affectsthe distribution of sentiment and intention,which quantitatively explores the mechanismof multimodal metaphor.

• We propose three tasks to evaluate fine-grained multimodal metaphor understandingabilities, including metaphor detection, senti-ment analysis, and intent detection in multi-modal metaphor. A range of baselines withbenchmark results are reported to show thepotential and usefulness of the MultiMET forfuture research.

2 Related Work

2.1 Metaphor Datasets

Although datasets of multimodal metaphors arescarce, a variety of monomodal datasets formetaphor studies have been created in recent years.Table 1 lists these datasets with their properties.

Numerous text metaphor datasets have been pub-lished for metaphor processing in the NLP commu-nity including several popular ones, e.g., the VUAmsterdam Metaphor Corpus (VUAMC) (Steen,2010), TroFi Example Base (Birke and Sarkar,2006), and MOH-X (Mohammad et al., 2016). Thelargest one, VUAMC, consists of over 10,000 sam-ples spread across 16,000 sentences, while otherscontain less than 5,000 samples. However, most ex-isting metaphor datasets contain only textual data.Image metaphor datasets are few and they are prettylimited in the size and the scope of the data, such asVisMet (Steen, 2018), which is a visual metaphoronline resource consisting of only 353 image sam-ples. Although Shutova et al. (2016) constructedboth text and image samples, their images were ob-tained by using a given phrase and queried Google

3216

Metaphor Dataset Sample Size(%Metaphor)

Modality Data Source Annotation

TroFi (Birke and Sarkar, 2006) 3,737 (44%) Text WSJ metaphor (metaphoricity)VUAMC (Steen, 2010) 16,000 (12.5%) Text BNC Baby metaphorTSV (Tsvetkov et al., 2014) 3,334 (50%) Text Web metaphor, affectLCC (Mohler et al., 2016) 16,265 (19%) Text ClueWeb09 metaphorMOH (Mohammad et al., 2016) 1,639 (25%) Text WordNet metaphorZayed’s Tweets (Zayed et al., 2019) 2,500 (54%) Text Twitter metaphorVisual Met (Steen, 2018) 353 (100%) Image Adv, Arts, Cartoons metaphorShutova et al. (2016) 2,415 (50%) Text,Image WordNet metaphorMultiMET (Ours) 10,437 (58%) Text,Image Social Media, Adv metaphor, sentiment, intent

Table 1: Comparison of various metaphor datasets

images. In that way, words and images in theirwork may be not suitably presented by each other.

The cognitive nature of metaphor implies thatnot only one modal isolation, but rather inte-grated multimodal information may contribute tometaphor expression and understanding, whichmakes our dataset MultiMET, which is large scaleand contains both natural text and image mes-sages and their annotations, different from existingdatasets and more important for metaphor studies.

2.2 Metaphor Understanding

Automatic metaphor understanding requires ac-complishing certain tasks to decode metaphoricalmessages. In this paper, we focus on three im-portant tasks for NLP in understanding metaphor:metaphor detection, sentiment analysis, and authorintent detection. There has been increasing interestin NLP in various approaches to metaphor detec-tion based on monomodal text. Early metaphorstudies have focused on hand-constructed knowl-edge and machine learning techniques (Mason,2004; Turney et al., 2011; Tsvetkov et al., 2014;Hovy et al., 2013). Others have also used distribu-tional clustering (Shutova et al., 2013) and unsuper-vised approaches (Shutova et al., 2017; Mao et al.,2018). More recently, deep learning models havebeen explored to understand metaphor. However,little has been explored in multimodal metaphordetection except by Shutova et al. (2016), who areamong the very few to explore the fusion of tex-tual and image modalities to detect multimodalmetaphor. Their results demonstrate the positiveeffect of combining textual and image features formetaphor detection.

However, in their work, image features are ex-tracted from a small size of constructed examplesrather than natural samples of texts integrated withimages, like MultiMET in our work. In addi-

tion, apart from multimodal metaphor detection,the tasks related to metaphor understanding likesentiment detection and author intent detection inmultimodal metaphor also have rarely been stud-ied, although there exist similar multimodal studiesin different tasks (Wang et al., 2017; Zadeh et al.,2017; Kruk et al., 2019).

3 The MultiMET Dataset

3.1 Data Collection

With the goal of creating a large-scale multimodalmetaphor dataset to support research on understand-ing metaphors, we collect data that contains bothtext and image from a range of sources includingsocial media (Twitter and Facebook), and advertise-ments. Table 2 shows an overview for the statisticsof the dataset.

Social Media. To collect potential metaphoricalsamples from Twitter and Facebook, we retrievedposts by querying hashtags metaphor or metaphor-ical. We collected publicly available Twitter andFacebook posts using Twitter and Facebook APIscomplying with Twitter and Facebook’s terms ofservice. What the author labels as metaphoricalis not always aligned with the actual definitionof metaphor in our study. To collect metaphorswhose nature accorded with what we define as mul-timodal metaphors, we re-annotated “metaphoricalor literal” in the below section to potential Twitterand Facebook posts that other authors annotated asmetaphor with hashtags.

Advertisements. Based on our review of lin-guistic literature on multimodal metaphor, we fo-cused on an important source that is the main con-text of study: advertisements. Metaphorical mes-sages abound in advertisements , which offer a nat-ural and rich resource of data on metaphor and howtextual and visual factors combine and interact (So-brino, 2017; Forceville et al., 2017). We collected

3217

(a) Life is a highway. (b) Sometimes, with-out knowing why, yourheart beats faster. NewBeetle.

(c) A kitten is kissinga flower. Butterflies arenot insects.

Figure 2: Examples of metaphor categories

Item SocialMedia

Adv Total

Total Samples 6,109 4,328 10,437Metaphorical Samples 3,489 2,537 6,026Literal Samples 2,620 1,791 4,411Total Words 79,417 51,936 131,353Avg Words of Samples 13 12 13Train Set Size 2,791 2,029 4,820Validation Set Size 349 254 603Test Set Size 349 254 603

Table 2: MultiMET dataset statistics

potential metaphorical samples of advertising froma large, publicly released dataset of 64,832 imageadvertisements that contain both images and insidetext (Ye et al., 2019). To obtain the textual infor-mation, we extracted inside text from images usingthe API provided by Baidu AI. After that, humanannotators rectified the extracted inaccurate text, re-moved any blurred text, and obtained text + imagepairs from advertisements.

3.2 Data Filter

For text data, we removed external links and men-tions (@username); we removed non-English textusing the LANGID (Lui and Baldwin, 2012) li-brary to label each piece of data with a languagetag; we removed strange symbols such as emo-jis; we removed “metaphor” or “metaphoric” whenthey were regular words rather than hashtags, be-cause explicit metaphorical expressions are not ourinterest (e.g., “This metaphor is very appropriate”);we removed text with fewer than 3 words or morethan 40 words. For image data, we removed text-based images (all the words are in the image), aswell as images with low resolution. Because thistask is about multimodal metaphor, it is necessaryto maintain consistency of data between models.In other words, either both the image data and thetext data should be removed, or neither. In addition,in the de-duplication step, we considered removal

only when both text and images were repeated.

3.3 Annotation Model

We annotated the text-image pairs with the occur-rence of metaphors (literal or metaphorical); (ifmetaphorical) relations of target and source do-main (target/source: target/source vocabulary intext or verbalized target/source vocabulary in im-age); target/source modality (text, image, or text +image), metaphor category (text-dominant, image-dominant, or complementary); sentiment category(the sentiment metaphors evoke, namely very neg-ative, negative, neutral, positive, or very positive),and author intents (descriptive, expressive, persua-sive, or other). The annotation model was Anno-tationModel = (Occurrence, Target, Source, Tar-getModality, SourceModality, MetaphorCategory,SentimentCategory, Intent, DataSource). Figure 3is an annotation example.

Figure 3: An example of a text+image annotation

3.4 Metaphor Annotation

Metaphor category. There are a variety of waysin which texts and images are combined in multi-modal content (Hendricks et al., 2016; Chen et al.,2017). Based on our review of the literature and ob-servation of the samples in our dataset, we followTasic and Stamenkovic (2015) and divide multi-modal metaphor into three categories: text domi-nant, image dominant, and complementary. Some-times metaphors are expressed through texts with amapping between source and target domains whilethe accompanying images serve as a visual illus-tration of the metaphors in the text, which is textdominant. As in Figure 2 (a), the text itself is suffi-cient to convey metaphorical information and canbe identified as metaphorical expressions. “High-way” is a visual illustration of the source domainin a textual modality. By contrast, in the imagedominant category, images play the dominant rolein conveying metaphorical information and theyprovide sufficient information for readers to under-

3218

stand the metaphors. In Figure 2 (b), where wesee the metaphorical message “Beetle (cars) areblood cells,” the text enriches the understandingof metaphorical meaning by adding an explana-tion “your heart beats faster” to the visual mani-festation. The complementary category involves aroughly equal role of texts and images in renderingmetaphorical information. The understanding ofmetaphor depends on the interaction of and balancebetween different modalities. If texts and imagesare interpreted separately, metaphors cannot be un-derstood. In Figure 2 (c), when people read thetext, “A kitten is kissing a flower,” and the insidetext “Butterflies are not insects,” they do not realizethe metaphorical use until they observe the butter-fly in the corresponding image and infer that thetarget “butterfly” is expressed in term of the source“flower”.

Metaphorical or literal. Our annotations fo-cus on the dimension of expression, which in-volves identification of metaphorical and literalexpressions by verbal means and visual means(Forceville, 1996; Phillips and McQuarrie, 2004).The metaphor annotation takes place at the rela-tional level, which involves the identification ofmetaphorical relations between source and targetdomain expressions. For text modality, source andtarget domain expressions mean source and tar-get domain words used in metaphorical texts. Forimage modality, source and target domain expres-sions mean words’ verbalized source and targetdomain in the visual modality. That is, the anno-tation of metaphorical relations represented in themodality of image involve the verbalization of themetaphor’s domains. Annotations involve namingand labeling what is linguistically familiar. Unliketext modality, which relies on explicit linguisticcues, for image modality, metaphorical relationsare annotated based on perceptions of visual uni-ties, and they determine the linguistic familiarity ofimages as well as existing words in the metaphor’sdomains. Following Sorm and Steen (2018), anno-tators identified the metaphorical text+image pairsby looking at the incongruous units and explain-ing one non-reversible “A is B” identity relation,where two domains were expressed by differentmodalites.

3.5 Intent and Sentiment Annotation

Interpreting authorial intent from multimodal mes-sages in metaphor seems to be important for under-

standing metaphors. As mentioned above, withinCMT, the essence of metaphor is using one thingfrom a source domain to express and describe an-other from a target domain. This implies thatone important intent of creating metaphor couldbe to enable readers to understand the entities be-ing described better. “Perceptual resemblance” isa major means of triggering a metaphorical rela-tion between two different entities (Forceville andUrios-Aparisi, 2009). We name it descriptive intent,which involves visual and textual representationsregarding the object, event, concept, information,action or character, etc. Moreover, in modern times,the increasing ubiquity of multimodal metaphorsmeans that people cannot ignore its power of per-suasion (Urios-Aparisi, 2009). People often lever-age metaphor in communication environments suchas advertisements and social media to persuadereaders to buy or do things. We name this in-tent as persuasive. In addition, inspired by a vari-ety of arousing, humorous, or aesthetic effects ofmetaphors (Christmann et al., 2011), the expres-sive is included in our intent annotation within theenlarged definition: expressing attitude, thought,emotion, feeling, attachment, etc. Based on thesefactors as well as investigation of the samples in ourdatasets, we generalized their taxonomy and listedthe categories of the author intent in metaphor asdescriptive,persuasive, expressive, and others.

Numerous studies show that metaphorical lan-guage frequently expresses sentiments or emotionsimplicitly (Goatly, 2007; Kovecses, 1995, 2003).Compared to literal expressions, metaphors elicitmore emotional activation of the human brain in thesame context (Citron and Goldberg, 2014). Thuswe also added the sentiment in our annotation, totest whether the sentiment impact of metaphors isstronger than literary messages from a multimodalperspective. The sentiment was placed in one of thefive categories of very negative, negative, neutral,positive, or very positive.

3.6 Annotation Process

We took two independent annotation approachesfor two different types of tasks: selecting typesof sentiment and intent and the annotation ofmetaphor. To select the options for sentiment andintent, we used a majority vote through Crowd-Flower, the crowdsourcing platform. The partici-pants were randomly presented with both the textand vision components with the instruction on the

3219

(a) Metaphor Category. (b) Metaphor and non-Metaphor Intent. (c) Metaphor and non-Metaphor Sentiment.

Figure 4: Dataset Distribution

top of each text + image pair for options.The annotation of metaphors includes metaphor

occurrence, metaphor category and domain rela-tion annotation. For metaphor annotation, we usedexpert annotators to complete the challenging an-notation task, which required relatively deep under-standing of metaphorical units and the completetask of verbalization of domains in image. Theannotator team comprised five annotators who arepostgraduate student researchers majoring in com-putational linguistics with metaphor study back-grounds. The annotators formed groups of two,plus one extra person. Using cross-validation, thetwo-member groups annotated, and the fifth personintervened if they disagreed.

3.7 Quality Control and Inner Agreement

Annotations of multimodal metaphors rely on an-notators’ opinions and introspection, which mightbe subjective. Thus we took corresponding, dif-ferent measures for different types of annotationsto achieve high-quality annotation. To select op-tions, we established strict criteria for the choiceof category. Each text-image pair was annotatedby at least 10 annotators and we used a major-ity vote through CrowdFlower, the crowdsourcingplatform. Following Shutova (2017), we chose thecategory of annotated options on which 70% ormore annotators agreed as the answer to each ques-tion (final decision) to provide high confidence ofannotation. For metaphor annotation, we added aguideline course, detailed instruction, and manysamples, and we held regular meetings to discussannotation problems and matters that needed at-tention. The guidelines changed three times whennew problems emerged or good improvement meth-ods were found. The kappa score, κ, was used tomeasure inter-annotator agreements (Fleiss, 1971).The agreement on the identification of literal ormetaphorical was κ = 0.67; identification of textdominant, image dominant or complementary was

κ = 0.79; the identification of source and targetdomain relation was κ = 0.58, which means theyare substantially reliable.

4 Dataset Analysis

Metaphor Category. We analyzed the role oftextual and visual modalities to detect metaphors.From Figure 4 (a), we can see a complementarycategory among the three kinds of multimodalmetaphors, which requires the interplay of textualand visual modality to understand the metaphori-cal meaning. It accounts for the largest proportionof metaphors, followed by the text-dominant andimage-dominant categories. It shows the contribu-tion of visual factors, which are similarly impor-tant in detecting metaphors. We therefore present aquantitative study of the role of textual and visualmodalities in metaphor detection through humanannotations and confirm the role and contributionof visuals in metaphor occurrence in natural lan-guage.

Author Intent. Figure 4 (b) shows that ex-pressive and persuasive intentions occur most fre-quently in the metaphorical data. However, de-scriptive intention occurs most frequently in thenon-metaphorical data. This suggests that on theone hand, we are more likely to use metaphoricalexpressions when expressing our feelings, express-ing emotions, or trying to persuade others. On theother hand, we tend to use literal expressions tomake relatively objective statements.

Sentiment. Figure 4 (c) shows that there aresome differences in the distribution of sentiment be-tween the metaphorical data and non-metaphoricaldata. In the non-metaphorical data, neutral senti-ment accounted for the largest proportion of 51%,followed by positive sentiment (33%), strong posi-tive sentiment (7%), negative sentiment (7%), andstrong negative sentiment (2%). In the metaphor-ical data, positive sentiment accounted for thelargest proportion of 42%, followed by neutral sen-

3220

Hyper-Parameter ValueWord embedding size 300Hidden size of LSTM 256Dropout 0.4Text padding 30Batch size 48Learning rate 5e-4Gradient clipping 10Early stop patience 10

Table 3: Hyperparameters.

timents (39%), strong positive sentiment (8%), neg-ative sentiment (8%), and strong negative sentiment(3%). It turns out that there are more non-neutralsentiments in metaphor expression than in non-metaphorical expression, and that metaphors aremore frequently used to convey sentiments. Ourfindings accord with the results of previous studieson monomodal textual metaphors that metaphorsconvey more sentiments or emotions than literarytext (Mohammad et al., 2016). We confirm thestronger emotional impact of metaphors than liter-ary messages from a multimodal perspective.

In positive sentiment, the most common wordsin the source domain are person, face, and flower;the most common words in the target domain arelove, life, and success. In negative sentiment, heart,food, and smoke are the most common words inthe source domain, and the world, disaster, and lifeare the most common words in the target domain.This shows that sentiment tendency can influencethe category in the source and target domains tosome extent.

5 Experiment

For the dataset constructed for this paper, wepropose three tasks and provide their baselines,namely multimodal metaphor detection, multi-modal metaphor sentiment analysis, and multi-modal metaphor author intent detection.

We used the model shown in Figure 5 to detectmetaphors, metaphorical sentiments, and metaphor-ical intentions. For text input, we used a text en-coder to encode the text and to get the feature vectorof the text. This paper used two different methodsto encode the text, namely the pre-trained Bidirec-tional Encoder Representations from Transform-ers (BERT) model (Devlin et al., 2019) and Bi-directional Long-Short Term Memory (Bi-LSTM)networks (Medsker and Jain, 2001). Similarly, forimage input, we used an image encoder to extractimage features. We used three different image pre-

training models: VGG16 (Simonyan and Zisser-man, 2014), ResNet50 (He et al., 2016), and Effi-cientNet (Tan and Le, 2019). These methods havebeen widely used by researchers in feature extrac-tion for various tasks.

After obtaining the text feature vector and theimage feature vector, we used four different fea-ture fusion methods to combine the vectors, namelyconcatenation (Suryawanshi et al., 2020), element-wise multiply (Mai et al., 2020), element-wise add(Cai et al., 2019), and maximum (Das, 2019). Fi-nally, we inputted the fusion vector into a fullyconnected layer and obtained the probabilities ofdifferent categories through the softmax activationfunction.

Figure 5: Multimodal model for integrating text andimage data.

5.1 Experiment SettingsWe used Pytorch (Paszke et al., 2019) to build themodel. The pre-trained models are available in Py-torch. The word embeddings have been trained ona Wikipedia dataset by Glove (Pennington et al.,2014). In the training process, we did not updatethe parameters in the pre-training models. Whenthe model gradually tended to converge, we up-dated the parameters of the pre-training modelswith training data to avoid overfitting. We usedthe Adam optimizer (Kingma and Ba, 2014) to op-timize the loss function, and the training methodof gradient clipping (Zhang et al., 2019) to avoidgradient explosion. Other hyper-parameter settingsare shown in Table 3.

5.2 ResultsThe classification results are shown in Table 4.“Random” means that random predictions weremade using the data as a baseline. In general,the model performed best on metaphor detection,followed by metaphor intention detection, and fi-nally metaphor sentiment detection. For image and

3221

Metaphor Sentiment IntentionType Text Image Validation Test Validation Test Validation Test

Random - - 0.5063 0.4923 0.2222 0.2023 0.3416 0.3609

Text Bi-LSTM - 0.7458 0.7434 0.5705 0.5714 0.6597 0.6593BERT - 0.7742 0.7736 0.5958 0.5927 0.6794 0.6720

Image- VGG16 0.7315 0.7345 0.5953 0.5914 0.6672 0.6658- EfficientNet 0.7467 0.7405 0.5563 0.5548 0.6441 0.6324- ResNet50 0.7677 0.7646 0.5715 0.5714 0.6658 0.6653

Text + Image

Bi-LSTM VGG16 0.7735 0.7658 0.6195 0.6157 0.6843 0.6812Bi-LSTM EfficientNet 0.7832 0.7795 0.5723 0.5714 0.6672 0.6732Bi-LSTM ResNet50 0.7988 0.7912 0.6263 0.6220 0.7036 0.6843

BERT VGG16 0.8033 0.8072 0.6289 0.6188 0.7012 0.7000BERT EfficientNet 0.7975 0.8033 0.6152 0.6125 0.6833 0.6757BERT ResNet 0.8276 0.8286 0.6462 0.6422 0.7278 0.7245

Table 4: Results on three tasks with a combination method of concatenate.

Metaphor Sentiment IntentionCombination Methods Validation Test Validation Test validation Test

Add 0.7868 0.7834 0.6205 0.6186 0.6827 0.6779Multiply 0.7596 0.7583 0.5685 0.5636 0.6442 0.6457

Maximum 0.7827 0.7759 0.6113 0.6074 0.7035 0.6993Concatenate 0.8276 0.8286 0.6462 0.6422 0.7278 0.7245

Table 5: Results on different multimodal combinations for BERT + ResNet.

multimodal classification, the ResNet50 performedbest, followed by VGG16, and finally EfficientNet.Because ResNet solved the problem of gradientdisappearance through the method of residual con-nection, the classification performance was betterthan VGG16 and EfficientNet. For text and multi-modal classification, BERT performed better thanBi-LSTM. BERT has been fully trained in a large-scale corpus, using transfer learning technology tofine-tune our three tasks and data, so it can achievebetter performance. From the perspective of differ-ent features, multimodal features perform best, fol-lowed by text-only features, and finally image-onlyfeatures. Multimodal fusion helps to improve theclassification performance by 6%. This shows thatthe combination of image and text features is in-deed helpful for the detection and understanding ofmetaphors, especially the detection of sentimentsand intentions in metaphors. In addition, the im-portance of text modal data is explained. Withouttext description, it is difficult to detect metaphorscorrectly using only visual modal data.

To verify the influence of feature fusion on classi-fication, we compared four different feature fusionmethods. The results are shown in Table 5. Theconcatenate method to merge image and text fea-tures produces the highest accuracy. It shows thatconcatenate can make full use of the complemen-tarity between different modal data, eliminate thenoise generated by the fusion of different modaldata, and improve the detection effect. In contrast,

the other three fusion methods cannot effectivelyeliminate the influence of noise introduced by dif-ferent modal data, and it therefore interferes withthe training of the model. Overall, the multimodemodel that combines the BERT text function andthe ResNet50 image function through the concate-nation method performs best on our three tasks.

6 Conclusion

This paper presents the creation of a novel re-source, a large-scale multimodal metaphor dataset,MultiMET, with manual fine-gained annotationfor metaphor understanding and research. Ourdataset enables the quantitative study of the inter-play of multimodalities for metaphor detection andconfirms the contribution of visuals in metaphoroccurrence in natural language. It also offers aset of baseline results of various tasks and showsthe importance of combining multimodal cues formetaphor understanding. We hope MultiMETprovides future researchers with valuable multi-modal training data for the challenging tasks ofmultimodal metaphor processing and understand-ing ranging from metaphor detection to sentimentanalysis of metaphor. We also hope that Multi-MET will help to expand metaphor research frommonomodality to multimodality and improve theperformance of automatic metaphor understandingsystems and contribute to the in-depth understand-ing and research development of metaphors. Thedataset will be publicly available for research.

3222

Ethical Considerations

This research was granted ethical approval by ourInstitutional Review Board (Approval code: DU-TIEE190725 01). We collected publicly availableTwitter and Facebook data using Twitter and Face-book APIs complying with Twitter and Facebook’sterms of service. We did not store any personaldata (e.g., user IDs, usernames) and we annotatedthe data without knowledge of individual identities.

We annotated all our data using two indepen-dent approaches (expert based and crowdsourcingbased) for two different types of tasks: the anno-tation of metaphor and the selection of types ofsentiment and intent. For metaphor annotation, adeep understanding of metaphorical units was nec-essary. This challenging task was completed byfive researchers who involved in this project. To an-notate sentiment and intent, we used CrowdFlower,the crowdsourcing platform. To ensure that crowdworkers were fairly compensated, we paid themat an hourly rate of 15 USD per hour, which isa fair and reasonable rate of pay for crowdsourc-ing (Whiting et al., 2019). We launched small pi-lots through CrowdFlower. The pilot for sentimentoptions took on average 43 seconds, and crowdworkers were thus paid 0.18 USD per judgment, inaccordance with an hourly wage of 15 USD. At thesame time, the annotation of author intent took onaverage 23 seconds, and we thus paid 0.10 USDper judgment, corresponding to an hourly wage of15 USD.

Acknowledgments

We would like to thank the anonymous review-ers for their insightful and valuable comments.This work is supported by NSFC Programs(No.62076051).

References

Julia Birke and Anoop Sarkar. 2006. A clustering ap-proach for nearly unsupervised recognition of non-literal language. In Proceedings of the 11th Confer-ence of the European Chapter of the Association forComputational Linguistics, pages 329–336, Trento,Italy.

Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi-modal sarcasm detection in twitter with hierarchicalfusion model. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 2506–2515, Florence, Italy.

Lynne Cameron. 2003. Metaphor in educational dis-course. A&C Black, London, UK.

Tseng-Hung Chen, Yuan-Hong Liao, Ching-YaoChuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun.2017. Show, adapt and tell: Adversarial trainingof cross-domain image captioner. In Proceedingsof the 2017 IEEE International Conference on Com-puter Vision, pages 521–530, Venice, Italy.

Ursula Christmann, Lena Wimmer, and NorbertGroeben. 2011. The aesthetic paradox in process-ing conventional and non-conventional metaphors:A reaction time study. Scientific Study of Literature,1(2):199–240.

Francesca MM Citron and Adele E Goldberg. 2014.Metaphorical sentences are more emotionally engag-ing than their literal counterparts. Journal of Cogni-tive Neuroscience, 26(11):2585–2595.

Seana Coulson and Cyma Van Petten. 2002. Concep-tual integration and metaphor: An event-related po-tential study. Memory & Cognition, 30(6):958–968.

Dipto Das. 2019. A multimodal approach to sarcasmdetection on social media. Ph.D. thesis, MissouriState University.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186, Minneapolis, USA.

Francesca Ervas. 2019. Metaphor, ignorance and thesentiment of (ir) rationality. Synthese, pages 1–25.

Mark Evan Nelson. 2008. Multimodal synthesis andthe voice of the multimedia author in a japaneseefl context. Innovation in Language Learning andTeaching, 2(1):65–82.

Joseph L Fleiss. 1971. Measuring nominal scale agree-ment among many raters. Psychological Bulletin,76(5):378–382.

Charles Forceville. 1996. Pictorial metaphor in adver-tising. Psychology Press, East Sussex, UK.

Charles Forceville. 2016. Visual and multimodalmetaphor in film. In Embodied metaphors in film,television, and video games: Cognitive approaches,pages 17–32. Routledge, Abingdon, USA.

Charles Forceville and Eduardo Urios-Aparisi. 2009.Multimodal metaphor, volume 11. Walter deGruyter, Berlin, Germany.

Charles Forceville et al. 2017. Visual and multi-modal metaphor in advertising: Cultural perspec-tives. Styles of Communication, 9(2):26–41.

3223

Andrew Goatly. 2007. Washing the brain: Metaphorand hidden ideology, volume 23. John BenjaminsPublishing, Amsterdam, Netherlands.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep residual learning for image recog-nition. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition 2016,pages 770–778, Las Vegas, USA.

Lisa Anne Hendricks, Subhashini Venugopalan, Mar-cus Rohrbach, Raymond Mooney, Kate Saenko, andTrevor Darrell. 2016. Deep compositional caption-ing: Describing novel object categories withoutpaired training data. In Proceedings of the IEEEConference on Computer Vision and Pattern Recog-nition 2016, pages 1–10, Las Vegas, USA.

Dirk Hovy, Shashank Srivastava, Sujay Kumar Jauhar,Mrinmaya Sachan, Kartik Goyal, Huying Li, Whit-ney Sanders, and Eduard Hovy. 2013. Identifyingmetaphorical word use with tree kernels. In Proceed-ings of the 1st Workshop on Metaphor in NLP, pages52–57, Atlanta, Georgia.

Hyeju Jang, Keith Maki, Eduard Hovy, and CarolynRose. 2017. Finding structure in figurative lan-guage: Metaphor detection with topic-based frames.In Proceedings of the 18th Annual SIGDIAL Meet-ing on Discourse and Dialogue, pages 320–330,Saarbrucken, Germany.

Hyeju Jang, Seungwhan Moon, Yohan Jo, and Car-olyn Rose. 2015. Metaphor detection in discourse.In Proceedings of the 16th Annual Meeting of theSpecial Interest Group on Discourse and Dialogue,pages 384–392, Prague, Czech Republic.

Hermann Kappelhoff and Cornelia Muller. 2011. Em-bodied meaning construction: Multimodal metaphorand expressive movement in speech, gesture, andfeature film. Metaphor and the Social World,1(2):121–153.

Diederik P Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. arXiv e-prints,page arXiv:1412.6980.

Zoltan Kovecses. 1995. Anger: Its language, con-ceptualization, and physiology in the light of cross-cultural evidence. Language and the Cognitive Con-strual of the World, pages 181–196.

Zoltan Kovecses. 2003. Metaphor and emotion: Lan-guage, culture, and body in human feeling. Cam-bridge University Press, Cambridge, UK.

Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, DanJurafsky, and Ajay Divakaran. 2019. Integratingtext and image: Determining multimodal documentintent in instagram posts. In Proceedings of the2019 Conference on Empirical Methods in Natu-ral Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 4614–4624, Hong Kong,China.

George Lakoff and Mark Johnson. 1980. Metaphorswe live by. University of Chicago press, Chicago,USA.

Jerry Liu, Nathan O’Hara, Alexander Rubin, RachelDraelos, and Cynthia Rudin. 2020. Metaphor detec-tion using contextual word embeddings from trans-formers. In Proceedings of the 2nd Workshop onFigurative Language Processing, pages 250–255,Online.

Marco Lui and Timothy Baldwin. 2012. langid. py:An off-the-shelf language identification tool. In Pro-ceedings of the ACL 2012 System Demonstrations,pages 25–30, Jeju Island, Korea.

Sijie Mai, Haifeng Hu, Jia Xu, and Songlong Xing.2020. Multi-fusion residual memory network formultimodal human sentiment comprehension. IEEETransactions on Affective Computing, pages 1–15.

Rui Mao, Chenghua Lin, and Frank Guerin. 2018.Word embedding and wordnet based metaphor iden-tification and interpretation. In Proceedings of the56th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), pages1222–1231, Melbourne, Australia.

Zachary J Mason. 2004. Cormet: a computational,corpus-based conventional metaphor extraction sys-tem. Computational Linguistics, 30(1):23–44.

Larry R Medsker and LC Jain. 2001. Recurrent neuralnetworks. Design and Applications, 5:1–391.

Saif Mohammad, Ekaterina Shutova, and Peter Turney.2016. Metaphor as a medium for emotion: An em-pirical study. In Proceedings of the 15th Joint Con-ference on Lexical and Computational Semantics,pages 23–33, Berlin, Germany.

Michael Mohler, David Bracewell, Marc Tomlinson,and David Hinote. 2013. Semantic signatures forexample-based linguistic metaphor detection. InProceedings of the 1st Workshop on Metaphor inNLP, pages 27–35, Atlanta, USA.

Michael Mohler, Mary Brunson, Bryan Rink, and MarcTomlinson. 2016. Introducing the lcc metaphordatasets. In Proceedings of the 10th InternationalConference on Language Resources and Evaluation,pages 4221–4227, Portoroz, Slovenia.

Adam Paszke, Sam Gross, Francisco Massa, AdamLerer, James Bradbury, Gregory Chanan, TrevorKilleen, Zeming Lin, Natalia Gimelshein, LucaAntiga, et al. 2019. Pytorch: An imperative style,high-performance deep learning library. arXiv e-prints, page arXiv:1912.01703.

Jeffrey Pennington, Richard Socher, and Christopher DManning. 2014. Glove: Global vectors for word rep-resentation. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural Language Process-ing, pages 1532–1543, Doha, Qatar.

3224

Barbara J Phillips and Edward F McQuarrie. 2004.Beyond visual metaphor: A new typology of vi-sual rhetoric in advertising. Marketing Theory, 4(1-2):113–136.

Malay Pramanick, Ashim Gupta, and Pabitra Mitra.2018. An lstm-crf based approach to token-levelmetaphor detection. In Proceedings of the Workshopon Figurative Language Processing 2018, pages 67–75, New Orleans, USA.

Elisabeth El Refaie. 2003. Understanding visualmetaphor: The example of newspaper cartoons. Vi-sual Communication, 2(1):75–95.

Ekaterina Shutova. 2017. Annotation of linguistic andconceptual metaphor. In Handbook of linguisticannotation, pages 1073–1100. Springer, New York,USA.

Ekaterina Shutova, Douwe Kiela, and Jean Maillard.2016. Black holes and white rabbits: Metaphor iden-tification with visual features. In Proceedings of the2016 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 160–170, SanDiego, USA.

Ekaterina Shutova, Lin Sun, Elkin Darıo Gutierrez, Pa-tricia Lichtenstein, and Srini Narayanan. 2017. Mul-tilingual metaphor processing: Experiments withsemi-supervised and unsupervised learning. Com-putational Linguistics, 43(1):71–123.

Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010.Metaphor identification using verb and noun cluster-ing. In Proceedings of the 23rd International Con-ference on Computational Linguistics, pages 1002–1010, Beijing, China.

Ekaterina Shutova, Simone Teufel, and Anna Korho-nen. 2013. Statistical metaphor processing. Compu-tational Linguistics, 39(2):301–353.

Karen Simonyan and Andrew Zisserman. 2014. Verydeep convolutional networks for large-scale imagerecognition. arXiv e-prints, page arXiv:1409.1556.

Paula Perez Sobrino. 2017. Multimodal metaphor andmetonymy in advertising, volume 2. John BenjaminsPublishing Company, Amsterdam, Netherlands.

Esther Sorm and Gerard Steen. 2018. Towards amethod for visual metaphor identification. In Visualmetaphor: Structure and process, volume 18, pages47–88. John Benjamins Publishing Company, Ams-terdam, Netherlands.

Gerard Steen. 2010. A method for linguistic metaphoridentification: From MIP to MIPVU, volume 14.John Benjamins Publishing, Amsterdam, Nether-lands.

Gerard J Steen. 2018. Visual metaphor: Structure andprocess, volume 18. John Benjamins PublishingCompany, Amsterdam, Netherlands.

Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mi-hael Arcan, and Paul Buitelaar. 2020. Multimodalmeme dataset (multioff) for identifying offensivecontent in image and text. In Proceedings of the 2ndWorkshop on Trolling, Aggression and Cyberbully-ing, pages 32–41, Marseille, France.

Mingxing Tan and Quoc Le. 2019. Efficientnet: Re-thinking model scaling for convolutional neural net-works. In Proceedings of International Conferenceon Machine Learning 2019, pages 6105–6114, Cali-fornia, USA.

Milos Tasic and Dusan Stamenkovic. 2015. The inter-play of words and images in expressing multimodalmetaphors in comics. Procedia-Social and Behav-ioral Sciences, 212:117–122.

Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman,Eric Nyberg, and Chris Dyer. 2014. Metaphor detec-tion with cross-lingual model transfer. In Proceed-ings of the 52nd Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 248–258, Maryland, USA.

Peter Turney, Yair Neuman, Dan Assaf, and Yohai Co-hen. 2011. Literal and metaphorical sense identifica-tion through concrete and abstract context. In Pro-ceedings of the 2011 Conference on Empirical Meth-ods in Natural Language Processing, pages 680–690, Scotland, UK.

Eduardo Urios-Aparisi. 2009. Interaction of multi-modal metaphor and metonymy in tv commercials:Four case studies. Multimodal Metaphor, 11:95–116.

Eija Ventola, Cassily Charles, and Martin Kaltenbacher.2004. Perspectives on multimodality, volume 6.John Benjamins Publishing, Amsterdam, Nether-lands.

Haohan Wang, Aaksha Meghawat, Louis-PhilippeMorency, and Eric P Xing. 2017. Select-additivelearning: Improving generalization in multimodalsentiment analysis. In Proceeding of 2017 IEEEInternational Conference on Multimedia and Expo,pages 949–954, Hong Kong, China.

Lyndon CS Way and Simon McKerrell. 2017. Music asmultimodal discourse: Semiotics, power and protest.Bloomsbury Publishing, London, UK.

Mark E Whiting, Grant Hugh, and Michael S Bernstein.2019. Fair work: Crowd work minimum wage withone line of code. In Proceedings of the AAAI Con-ference on Human Computation and Crowdsourcing,pages 197–206, Hilversum, The Netherlands.

Zhao Xiufeng. 2013. The conceptual integration modelof multimodal metaphor construction: A case studyof a political cartoon. Foreign Languages Research,5:1–8.

3225

Fan-Pei Gloria Yang, Kailyn Bradley, Madiha Huq,Dai-Lin Wu, and Daniel C Krawczyk. 2013. Contex-tual effects on conceptual blending in metaphors: anevent-related potential study. Journal of Neurolin-guistics, 26(2):312–326.

Keren Ye, Narges Honarvar Nazari, James Hahn, Za-eem Hussain, Mingda Zhang, and Adriana Ko-vashka. 2019. Interpreting the rhetoric of visual ad-vertisements. IEEE Transactions on Pattern Analy-sis and Machine Intelligence, pages 1–16.

Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cam-bria, and Louis-Philippe Morency. 2017. Tensorfusion network for multimodal sentiment analysis.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages1103–1114, Copenhagen, Denmark.

Omnia Zayed, John P McCrae, and Paul Buitelaar.2019. Crowd-sourcing a high-quality dataset formetaphor identification in tweets. In Proceedings ofthe 2nd Conference on Language, Data and Knowl-edge, pages 1–17, Leipzig, Germany.

Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jad-babaie. 2019. Why gradient clipping acceleratestraining: A theoretical justification for adaptivity.arXiv e-prints, page arXiv:1905.11881.