Komugi

komugi

How Technology & Storytelling Dominate in a Decentralized World Algorithms, data-analytics and machine learning increasingly determine who and what we trust in a digital age. Clearly, we should add blockchain technology to this list. This became very clear when I visited Tbilisi, the capital of the Georgia, last week. I was in Tbilisi to present to more than 600 medium-sized and large companies about the importance of storytelling in building reputation and trust in a fast-changing, technology-driven world. This message might sound a little counterintuitive. How can one of the oldest social activities — storytelling — be so important in a digital age? I think there are several reasons. Technology can actually help us tell “better” (i.e. high-impact) stories and, more importantly, “hook in” a wider audience. The best businesses and entrepreneurs know how to use technology to build, connect and share a reputation or brand. Conversely, storytelling is vital to explaining new technologies and their benefits. In a globally competitive economy, storytelling becomes a key skill for differentiating (and “selling”) your product/service. Storytelling and new technologies working together is the key to building reputation and trust in a decentralized world. And people instantly “get” this. As soon as I raised the issue of storytelling in the crypto and blockchain economy, a burst of energy swept through the room. Previous presentations — on reporting requirements and conventional forms of information disclosure — seemed to have the opposite effect. This recognition and appreciation of the power of storytelling in the context of a technology-driven economy wasn’t surprising. After all, Georgia is among the top-3 countries in the world for Bitcoin mining. They understand new technology and the digital culture emerging around such technology. This is largely due to , one of the leaders in Bitcoin mining hardware (ASICs) and chips, which also operates from Georgia. What is more important is that collaborates with the Georgian government in a project to register land titles using the Bitcoin blockchain. Georgia was quick to recognize the benefits of confidence in the distributed ledger technology. BitFury BitFury _Land-Titling Project will Extend to Other Government Departments and Increase Blockchain Capabilities for Georgian…_medium.com The Bitfury Group and Government of Republic of Georgia Expand Historic Blockchain Land-Titling… Why We Need More Trust I am a believer in new technologies. Yet, it isn’t always easy to convince my colleagues or students. Volatility, hacks and other incidents and scandals in the cryptocurrency space don’t help. One of the latest that received a lot of media attention this week was the hack of the Japanese cryptocurrency exchange . Coincheck _The plunder of more than $500 million worth of digital coins from the Japanese cryptocurrency exchange Coincheck last…_www.technologyreview.com What the Coincheck hack means for the future of blockchain security At the end of January 2018, a hacker penetrated ’s security and stole 520 million XEM (approximately US$500 million). Coincheck XEM is the native cryptocurrency of ( ). ’s blockchain platform allows users to configure their own “smart assets” (which could represent a signature, a coin, a deed or any other asset) and applications. NEM New Economy Movement NEM Because had XEM sitting in a “hot wallet” (a computer connected to the Internet), it was relatively easy for hackers to steal them in what the called “one of history’s biggest heists”. Coincheck Financial Times This brings us back to the need for trust and security, the value of reputation and the importance of storytelling in building reputation. Initially, Coincheck was widely trusted in Japan. A free app with friendly UI facilitated consumers in trading. The exchange was advertised on primetime TV by one of Japan’s most famous comedians. After the “heist” you might expect that trust in blockchain platforms and cryptocurrency would decrease. But nothing seems further from the truth. As one of the “investors” (Komugi Yamazaki) in XEM through Coincheck put it: . “We are still crazy for crypto” Investors only hope that they will be able to recover their XEM (they aren’t interested in Japan’s currency Yen). And, recognizing this, Coincheck has promised to fully reimburse victims for the lost tokens. Technology + Storytelling = A Formula for Security & Trust The hack of could have been avoided by keeping the XEM in a “cold wallet”, isolated from the Internet. Most commentators seem to agree that avoiding this type of event is relatively simple. Coincheck Yet, the heist has rekindled discussion regarding the role and responsibilities of governments and regulators. The question is whether increased regulation by the state and supervision will create the trust necessary to make these new “decentralized” systems work. As I have written , regulators will always be “behind” the current pace of innovation and decentralization. before Ironically, the blockchain platform has built in some technological features that helps build trust in this “cryptocurrency and smart contracts” environment: NEM Even though the platform cannot recover the “stolen” XEM, it has used technology to “tag” them, making it very difficult for the hacker to use them without alerting investigators. NEM uses a Proof of Importance algorithm to protect against attacks and to offer more security for its blockchain transactions. NEM uses a version of the algorithm. This enables nodes to smartly assess the “reputation” of other nodes and sanction bad actors. NEM Eigentrust++ In short, can be viewed as a system that has “embedded regulation in technology”. NEM And, this (to me) is the future. In a decentralized world, security and trust can only be maintained via a combination of technology and storytelling. In order to be smart about trusting the new technologies, we shouldn’t think about applying usually “old-school” rules and regulations to build trust. The benefits of traditional forms of information disclosure, for example, have been greatly exaggerated. We have to find other technologies that “replace traditional rules and regulations” (or in technical terms come up with a secure solution to validate the blockchain). So, What’s Next? In this respect, I am very excited to be involved in an early stage startup company, . SEMADA.io In this Chicago-based company, technologists and non-technologists work together to create a blockchain-based “reputation verification platform” that rewards trustful parties and punishes bad actors. Trust will be created through an “eternal reputational record”, which is open to review and driven by proper incentives. The platform is relying on “storytelling” and “cryptocurrencies” to earn “reputation tokens”. The company understands that the real challenge is to find a “proof of stake” protocol that offers both trust and security (without unintentionally creating just another centralized validation system). The “ Proof of Stake protocol” (SPoS) uses its reputation-verification platform to solve the centralization, efficiency, and security problems that afflict existing blockchain consensus protocols. SEMADA And this is a just another step closer to building trust and security in our decentralized future. 👏 Thank you for reading! Please click and hold the below, or leave a comment. There is a new story every week. So if you follow me you won’t miss my latest insights about how the digital age is changing the way we live and work.

Building Trust & Security in Our Future

Scene Graphs & WikiData in Meme Analysis: Advancing Hate Speech Detection

Human Augmentation in Meme Analysis: Learnings and Challenges

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References 4 Benchmarking 4.1 Models To evaluate the results of our automatic augmentation and how it can affect the performance of hateful memes classification, we employed a text-based Transformer with and without the MemeGraphs information as well as a multimodal model. Previous work has shown that using only the text of the memes for classification gives results highly competitive with multimodal methods [14, 13, 28]. On the other hand, unimodal image-based models have lowest results. We use a pre-trained BERT [9] model that takes as input the text of each meme (TxtBERT) [14] and different variants of this approach that include the MemeGraphs input. In order to feed the graphs as input to the model, we represent them as a text sequence (see Section 3), which is given as an extra text input after the [SEP] token (Fig. 3). In the model that we call MemeGraphs[SceneGr], this sequence contained only the scene graphs, which were represented by triplets of the detected objects and the relations between them (Eq. 2), e.g., “0-man has 11-eye. 0-man wearing 12-shirt.”. In the model called MemeGraphs[Know], the text descriptions from the knowledge base corresponding to the detected entities are added as extra input (Eq. 3). For example, the second input sequence for the meme shown in Figure 3 will be “American politician, businessman, and 29th Governor of New Mexico.”. While in the MemeGraphs[SceneGr+Know] model, information from the whole knowledge graphs is added, comprising the scene graphs and the background knowledge concatenated with a full stop into one sequence (see Subsection 3.3). In all the models, the [CLS] token is fed into a final linear layer with a sigmoid activation function that produced the probability of the meme being hateful. In order to compare our MemeGraphs approach with a method that employs learned visual representations, we use ImgBERT, an early fusion multimodal model (see Section 2). 4.2 Experimental Setup Each meme was labeled in the original dataset as offensive or non-offensive. The classification was based on the offensiveness labels that exist in the MultiOFF dataset. The dataset is slightly imbalanced with 42% of the samples being offensive. For training and testing the models, we used the split provided by the authors of the dataset. The split consisted of training, validation, and test sets, which contained 445, 149, and 149 memes respectively. We employed the pre-trained BERT model provided by Hugging Face and fine-tuned it on our dataset.[10] To obtain image embeddings, we experimented with several pre-trained CNN models and based on the best results we chose DenseNet with 161 layers. The embeddings were extracted from the last pooling layer. The max length was set as the average token length of the training texts, depending on the input of each model. The models were trained using batch size 16, the weighted binary cross entropy loss, and AdamW optimizer with an initial learning rate of 2e-5. We used early stopping based on the validation loss with a patience of 3 epochs. We trained all models 20 times with different seeds and the training lasted between 4 and 6 epochs in each run. During inference, a threshold was used to determine if a meme is hateful or not based on the produced probability. This threshold was set to 0.5 for all the models. 4.3 Benchmarking results For each model, we obtained predictions for the test set from all twenty differently initialized runs. We evaluated each prediction set by calculating the F1 score defining the minority class (offensive class) as positive. In Table 1, we report the average F1 score over the twenty runs for each method and the corresponding standard error. In Table 2, the test score of the model that achieved the best score on the development set is shown. We observe that MemeGraphs[SceneGr+Know] outperforms the other models when predicting with the best checkpoint (Table 2). On the other hand, when looking at the average, the model with only the background knowledge as input achieves the best F1 score (MemeGraphs[Know]). We generally observe that the methods that incorporate our automatic augmentation (MemeGraphs) outperform the simple text-based fine-tuned BERT. Also, ImgBERT is outperformed showing that when the visual information is employed in the form of scene graphs the performance is improved, compared to when using only the image embeddings. [10] https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForSequenceClassification This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References 4 Benchmarking 4.1 Models To evaluate the results of our automatic augmentation and how it can affect the performance of hateful memes classification, we employed a text-based Transformer with and without the MemeGraphs information as well as a multimodal model. Previous work has shown that using only the text of the memes for classification gives results highly competitive with multimodal methods [14, 13, 28]. On the other hand, unimodal image-based models have lowest results. We use a pre-trained BERT [9] model that takes as input the text of each meme (TxtBERT) [14] and different variants of this approach that include the MemeGraphs input. In order to feed the graphs as input to the model, we represent them as a text sequence (see Section 3), which is given as an extra text input after the [SEP] token (Fig. 3). In the model that we call MemeGraphs[SceneGr], this sequence contained only the scene graphs, which were represented by triplets of the detected objects and the relations between them (Eq. 2), e.g., “0-man has 11-eye. 0-man wearing 12-shirt.”. In the model called MemeGraphs[Know], the text descriptions from the knowledge base corresponding to the detected entities are added as extra input (Eq. 3). For example, the second input sequence for the meme shown in Figure 3 will be “American politician, businessman, and 29th Governor of New Mexico.”. While in the MemeGraphs[SceneGr+Know] model, information from the whole knowledge graphs is added, comprising the scene graphs and the background knowledge concatenated with a full stop into one sequence (see Subsection 3.3). In all the models, the [CLS] token is fed into a final linear layer with a sigmoid activation function that produced the probability of the meme being hateful. In order to compare our MemeGraphs approach with a method that employs learned visual representations, we use ImgBERT, an early fusion multimodal model (see Section 2). 4.2 Experimental Setup Each meme was labeled in the original dataset as offensive or non-offensive. The classification was based on the offensiveness labels that exist in the MultiOFF dataset. The dataset is slightly imbalanced with 42% of the samples being offensive. For training and testing the models, we used the split provided by the authors of the dataset. The split consisted of training, validation, and test sets, which contained 445, 149, and 149 memes respectively. We employed the pre-trained BERT model provided by Hugging Face and fine-tuned it on our dataset.[10] To obtain image embeddings, we experimented with several pre-trained CNN models and based on the best results we chose DenseNet with 161 layers. The embeddings were extracted from the last pooling layer. The max length was set as the average token length of the training texts, depending on the input of each model. The models were trained using batch size 16, the weighted binary cross entropy loss, and AdamW optimizer with an initial learning rate of 2e-5. We used early stopping based on the validation loss with a patience of 3 epochs. We trained all models 20 times with different seeds and the training lasted between 4 and 6 epochs in each run. During inference, a threshold was used to determine if a meme is hateful or not based on the produced probability. This threshold was set to 0.5 for all the models. 4.3 Benchmarking results For each model, we obtained predictions for the test set from all twenty differently initialized runs. We evaluated each prediction set by calculating the F1 score defining the minority class (offensive class) as positive. In Table 1, we report the average F1 score over the twenty runs for each method and the corresponding standard error. In Table 2, the test score of the model that achieved the best score on the development set is shown. We observe that MemeGraphs[SceneGr+Know] outperforms the other models when predicting with the best checkpoint (Table 2). On the other hand, when looking at the average, the model with only the background knowledge as input achieves the best F1 score (MemeGraphs[Know]). We generally observe that the methods that incorporate our automatic augmentation (MemeGraphs) outperform the simple text-based fine-tuned BERT. Also, ImgBERT is outperformed showing that when the visual information is employed in the form of scene graphs the performance is improved, compared to when using only the image embeddings. [10] https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForSequenceClassification This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Comparing Meme Analysis Methods: MemeGraphs vs. ImgBERT

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References 3 MemeGraphs In this section, we describe our method for automatic augmentation of a memes dataset, an approach we call MemeGraphs. Our proposed method consists of three steps: 1. Scene graph construction, 2. Knowledge linking to detected entities and 3. The construction of the final MemeGraphs input. The first two steps are performed automatically by using off-the-shelf models. Hence, the result of MemeGraphs are knowledge graphs representing the meme as a whole, which can be used to classify them, e.g., for hate detection. We build on the MultiOFF dataset, which contains memes extracted from social media during the 2016 U.S. Presidential Election [28]. In what follows, we present the three individual steps of MemeGraphs in detail and how the final result is constructed. 3.1 Scene graphs For 67 memes, Schemata did not produce any output. For the remaining 676 memes, 10,426 relations and 9,666 objects were detected in total. The number of unique objects and relations was 142 and 970 respectively. And while the number of objects detected in an image ranged from 2 to 16, the number of detected relations ranged from 1 to 40. The top 15 most frequently detected object and relation types are shown in Fig. 2. 3.2 Knowledge linking 3.3 Knowledge graph input [6] This number was chosen after observation of the scene graphs resulting from the memes in order to avoid having an overcrowded scene graph with multiple objects. [7] https://spacy.io/universe/project/spacy-transformers [8] https://www.wikidata.org/wiki/Wikidata:Main_Page [9] All the texts are concatenated with a full stop. This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References 3 MemeGraphs In this section, we describe our method for automatic augmentation of a memes dataset, an approach we call MemeGraphs. Our proposed method consists of three steps: 1. Scene graph construction, 2. Knowledge linking to detected entities and 3. The construction of the final MemeGraphs input. The first two steps are performed automatically by using off-the-shelf models. Hence, the result of MemeGraphs are knowledge graphs representing the meme as a whole, which can be used to classify them, e.g., for hate detection. We build on the MultiOFF dataset, which contains memes extracted from social media during the 2016 U.S. Presidential Election [28]. In what follows, we present the three individual steps of MemeGraphs in detail and how the final result is constructed. 3.1 Scene graphs For 67 memes, Schemata did not produce any output. For the remaining 676 memes, 10,426 relations and 9,666 objects were detected in total. The number of unique objects and relations was 142 and 970 respectively. And while the number of objects detected in an image ranged from 2 to 16, the number of detected relations ranged from 1 to 40. The top 15 most frequently detected object and relation types are shown in Fig. 2. 3.2 Knowledge linking 3.3 Knowledge graph input [6] This number was chosen after observation of the scene graphs resulting from the memes in order to avoid having an overcrowded scene graph with multiple objects. [7] https://spacy.io/universe/project/spacy-transformers [8] https://www.wikidata.org/wiki/Wikidata:Main_Page [9] All the texts are concatenated with a full stop. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

MemeGraphs: Automated Augmentation for Hate Detection in Memes

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References 2 Related Work Combining text and image inputs is crucial for many tasks, e.g. for image search or (visual) question answering, relying on an image and its caption. Research has shown that adding images to text-based tasks (e.g., machine translation) improves the performance of the models [31]. However, the meaningful interpretation of text and image and, in particular, the relations between them, still remains challenging [17, 5]. Commonly used approaches rely on Transformer models that are pre-trained on image+text pairs [18, 20, 6, 12, 27, 19]. A step towards better scene understanding is to generate scene graphs [15]. Scene graphs provide structured knowledge about an image, e.g., objects, relations, and attributes. Recent works have shown that we can improve scene graph generation using message propagation between entities [32, 29], and by employing background knowledge in form of knowledge graphs [26], texts [25], or using feedback connections. In [26], the authors proposed Schemata, a scene graph generation model consisting of two parts: the backbone module and the relational reasoning component. Additionally, Schemata uses feedback connections to further encourage the propagation of higher-level, class-based knowledge to each neighbor. The backbone is pre-trained on ImageNet [8] and the whole network is fine-tuned on Visual Genome [15] on the scene graph classification task. Scene graphs can help to achieve state-of-the-art results in several visual tasks [30, 22]. Inspired by these approaches, we generate scene graphs to represent the visual information contained in memes by using the Schemata model (see Section 3). A specific instance of vision and language tasks is memes classification. To address the need for automatic means that can detect hateful content in memes, datasets and models [28, 3, 23] were published in the last couple of years, and shared tasks [13, 21, 11] were organized to attract interest on this task. Methods that have been implemented for hateful memes detection can be grouped into three categories: 1. Unimodal methods that use either only the text or the image as input, 2. Multimodal approaches, where image embeddings from an image encoder are fed to a text model and both models are trained separately, and 3. Multimodal methods, consisting of vision+language Transformers, being pre-trained in a multimodal fashion. Current methods experiment with models from all three categories and focus on improving models from the third category by adding extra features [3, 23, 4, 16]. These features can be visual attributes extracted from CNNs [13, 16, 1] (e.g. objects, entities or demographics), representations from CLIP [24, 23], automatically generated captions [7, 4], etc. The Hateful Memes Challenge, hosted in 2020 by Facebook, was a binary classification task of hate detection [13].[3] Kiela et al. [13] created a dataset with 10,000 memes to which they added counterfactual examples in order to make the task more challenging for unimodal approaches. They experimented with several different settings and found that multimodal methods worked best. An extended version of the Hateful Memes Challenge was included as a shared task in the Workshop on Online Abuse and Harms (WOAH) [21]. The same dataset was used but it now included new fine-grained labels for two categories: protected category and attack type. In this shared task, multimodal approaches were dominant as well. A multimodal method introduced by [13] and subsequently also used for the shared task in WOAH [14] incorporated image embeddings as inputs to a text classifier.[4] This method belongs to the second category and is an early fusion approach meaning that the image embedding and the text embedding are concatenated before feeding them to the classifier. Different types of image and text components are employed in different works. Specifically, in ImgBERT [14], first, they feed the memes images to a convolutional neural network (CNN) and extract their embeddings. Then, they provide the text of the meme as input to BERT [9] and extract the [CLS] token representation. They concatenate the [CLS] token representation with the embedding of the meme’s image and use the result as input to the classifier. During training only the text-based BERT part of ImgBERT is trained, while the image embeddings remain frozen. Another dataset for detecting hateful content in memes is the MultiOFF dataset [28]. It contains memes that were extracted from social media during the 2016 U.S. presidential elections. The dataset was first shared on Kaggle and consisted of the image URL for each meme, its text and metadata, e.g., timestamp, author, likes, etc.[5] The authors obtained the images from the URLs and discarded any metadata. In total, this dataset contains 743 memes, which were annotated as hateful or non-hateful. In [28] the authors experimented with unimodal (text only) and multimodal (text and image) approaches, and the model with the highest F1 score was a CNN operating only on the text of the memes. The above mentioned datasets focus on hate and offensive speech, but there are also datasets that cover other aspects of harmful content in memes. In [10], the authors focused on detecting propaganda in memes. They created and released a dataset with 950 memes extracted from Facebook groups, annotated for 22 different propaganda techniques. In their experiments they used existing unimodal and multimodal models and found that the latter, especially multimodally pre-trained Transformers perform best in their setting. Recently, a challenge called Multimedia Automatic Misogyny Identification (MAMI) focused on detecting misogyny and its exact form, i.e., stereotype, shaming, objectification and violence in memes [11]. In [23], they studied harm in memes and proposed a framework to detect harmful memes and the entities targeted. The authors also released their dataset with 7,096 memes in total about politics and COVID-19. The existing challenge sets, resources and models show the importance of analyzing internet memes and the challenge to combine all the modalities that form a meme. However, current works focus on incorporating visual information in the form of individual features like the ones described above or automatically generated captions. We propose a novel approach to represent the visual content of memes using scene graphs, hence "translating" them into text form. Furthermore, current methods only extract entities from the images, but not from the captions or texts. We argue that often this is not sufficient (or feasible), since memes can also incorporate screenshots of text, as it is the case in the MultiOFF dataset. Hence, we approach this problem by extracting the entities from the text in order to obtain more information. We further retrieve background knowledge for each extracted entity, and show that this approach is worthwhile to explore, since it allows for a more grounded and comprehensive automatic interpretation of memes. [3] https://www.drivendata.org/competitions/64/hateful-memes/ [4] The method was called Concat BERT in [13] and ImgBERT in [14]. Here we call it ImgBERT because we use their implementation. [5] https://www.kaggle.com/datasets/SIZZLE/2016electionmemes This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References 2 Related Work Combining text and image inputs is crucial for many tasks, e.g. for image search or (visual) question answering, relying on an image and its caption. Research has shown that adding images to text-based tasks (e.g., machine translation) improves the performance of the models [31]. However, the meaningful interpretation of text and image and, in particular, the relations between them, still remains challenging [17, 5]. Commonly used approaches rely on Transformer models that are pre-trained on image+text pairs [18, 20, 6, 12, 27, 19]. A step towards better scene understanding is to generate scene graphs [15]. Scene graphs provide structured knowledge about an image, e.g., objects, relations, and attributes. Recent works have shown that we can improve scene graph generation using message propagation between entities [32, 29], and by employing background knowledge in form of knowledge graphs [26], texts [25], or using feedback connections. In [26], the authors proposed Schemata, a scene graph generation model consisting of two parts: the backbone module and the relational reasoning component. Additionally, Schemata uses feedback connections to further encourage the propagation of higher-level, class-based knowledge to each neighbor. The backbone is pre-trained on ImageNet [8] and the whole network is fine-tuned on Visual Genome [15] on the scene graph classification task. Scene graphs can help to achieve state-of-the-art results in several visual tasks [30, 22]. Inspired by these approaches, we generate scene graphs to represent the visual information contained in memes by using the Schemata model (see Section 3). A specific instance of vision and language tasks is memes classification. To address the need for automatic means that can detect hateful content in memes, datasets and models [28, 3, 23] were published in the last couple of years, and shared tasks [13, 21, 11] were organized to attract interest on this task. Methods that have been implemented for hateful memes detection can be grouped into three categories: 1. Unimodal methods that use either only the text or the image as input, 2. Multimodal approaches, where image embeddings from an image encoder are fed to a text model and both models are trained separately, and 3. Multimodal methods, consisting of vision+language Transformers, being pre-trained in a multimodal fashion. Current methods experiment with models from all three categories and focus on improving models from the third category by adding extra features [3, 23, 4, 16]. These features can be visual attributes extracted from CNNs [13, 16, 1] (e.g. objects, entities or demographics), representations from CLIP [24, 23], automatically generated captions [7, 4], etc. The Hateful Memes Challenge, hosted in 2020 by Facebook, was a binary classification task of hate detection [13].[3] Kiela et al. [13] created a dataset with 10,000 memes to which they added counterfactual examples in order to make the task more challenging for unimodal approaches. They experimented with several different settings and found that multimodal methods worked best. An extended version of the Hateful Memes Challenge was included as a shared task in the Workshop on Online Abuse and Harms (WOAH) [21]. The same dataset was used but it now included new fine-grained labels for two categories: protected category and attack type. In this shared task, multimodal approaches were dominant as well. A multimodal method introduced by [13] and subsequently also used for the shared task in WOAH [14] incorporated image embeddings as inputs to a text classifier.[4] This method belongs to the second category and is an early fusion approach meaning that the image embedding and the text embedding are concatenated before feeding them to the classifier. Different types of image and text components are employed in different works. Specifically, in ImgBERT [14], first, they feed the memes images to a convolutional neural network (CNN) and extract their embeddings. Then, they provide the text of the meme as input to BERT [9] and extract the [CLS] token representation. They concatenate the [CLS] token representation with the embedding of the meme’s image and use the result as input to the classifier. During training only the text-based BERT part of ImgBERT is trained, while the image embeddings remain frozen. Another dataset for detecting hateful content in memes is the MultiOFF dataset [28]. It contains memes that were extracted from social media during the 2016 U.S. presidential elections. The dataset was first shared on Kaggle and consisted of the image URL for each meme, its text and metadata, e.g., timestamp, author, likes, etc.[5] The authors obtained the images from the URLs and discarded any metadata. In total, this dataset contains 743 memes, which were annotated as hateful or non-hateful. In [28] the authors experimented with unimodal (text only) and multimodal (text and image) approaches, and the model with the highest F1 score was a CNN operating only on the text of the memes. The above mentioned datasets focus on hate and offensive speech, but there are also datasets that cover other aspects of harmful content in memes. In [10], the authors focused on detecting propaganda in memes. They created and released a dataset with 950 memes extracted from Facebook groups, annotated for 22 different propaganda techniques. In their experiments they used existing unimodal and multimodal models and found that the latter, especially multimodally pre-trained Transformers perform best in their setting. Recently, a challenge called Multimedia Automatic Misogyny Identification (MAMI) focused on detecting misogyny and its exact form, i.e., stereotype, shaming, objectification and violence in memes [11]. In [23], they studied harm in memes and proposed a framework to detect harmful memes and the entities targeted. The authors also released their dataset with 7,096 memes in total about politics and COVID-19. The existing challenge sets, resources and models show the importance of analyzing internet memes and the challenge to combine all the modalities that form a meme. However, current works focus on incorporating visual information in the form of individual features like the ones described above or automatically generated captions. We propose a novel approach to represent the visual content of memes using scene graphs, hence "translating" them into text form. Furthermore, current methods only extract entities from the images, but not from the captions or texts. We argue that often this is not sufficient (or feasible), since memes can also incorporate screenshots of text, as it is the case in the MultiOFF dataset. Hence, we approach this problem by extracting the entities from the text in order to obtain more information. We further retrieve background knowledge for each extracted entity, and show that this approach is worthwhile to explore, since it allows for a more grounded and comprehensive automatic interpretation of memes. [3] https://www.drivendata.org/competitions/64/hateful-memes/ [4] The method was called Concat BERT in [13] and ImgBERT in [14]. Here we call it ImgBERT because we use their implementation. [5] https://www.kaggle.com/datasets/SIZZLE/2016electionmemes This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Advancements in Memes Analysis: Scene Graphs and Multimodal Approaches

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References ABSTRACT Memes are a popular form of communicating trends and ideas in social media and on the internet in general, combining the modalities of images and text. They can express humor and sarcasm but can also have offensive content. Analyzing and classifying memes automatically is challenging since their interpretation relies on the understanding of visual elements, language, and background knowledge. Thus, it is important to meaningfully represent these sources and the interaction between them in order to classify a meme as a whole. In this work, we propose to use scene graphs, that express images in terms of objects and their visual relations, and knowledge graphs as structured representations for meme classification with a Transformer-based architecture. We compare our approach with ImgBERT, a multimodal model that uses only learned (instead of structured) representations of the meme, and observe consistent improvements. We further provide a dataset with human graph annotations that we compare to automatically generated graphs and entity linking. Analysis shows that automatic methods link more entities than human annotators and that automatically generated graphs are better suited for hatefulness classification in memes. Keywords hate speech · internet memes · knowledge graphs · multimodal representations. 1 Introduction Internet memes are items such as images, videos, or twitter posts that are widely shared on social media and typically relate to several subjects, such as politics, social, news, and current internet trends.[1] Memes are a popular form of communication and they are often used as a means to express an opinion or stance in a humorous or sarcastic manner, but they can also be hateful and promote problematic content that is likely to hurt specific groups of people and hence be harmful to society in general [2]. Thus, analyzing trending memes can provide insight into people’s reactions and opinions to important societal matters, as well as to the traits of different groups. This can help in tasks like filtering out harmful memes from internet platforms or extracting user opinions for socioeconomic studies. This work focuses on memes in the form of images with some form of superimposed text (sometimes referred to with the technical term image macros) with the goal of detecting hateful content. Lately, there has been increased interest in deep learning models for analyzing memes and classifying them [23, 10, 11], for example in the context of the Hateful Memes competition organized by Facebook [13] and the shared task in the Workshop on Online Abuse and Harms (WOAH) 2021 that was the continuation of the first competition [21]. Another shared task in Semeval 2022 aimed at detecting misogyny in memes [11]. Alongside these shared tasks, the corresponding datasets were published with human annotations capturing important properties such as hatefulness and misogyny. Recent works use multimodal representation learning based on image features from Convolutional Neural Networks (CNNs), Transformer-based language models for the text [13, 3, 14] or multimodal (vision+language) Transformer models in order to classify memes [13]. Some works additionally incorporate specifically extracted image  features such as race [33], person attributes [1, 23], and automatic image captioning output [7, 4]. However, none of these works have included the relations between objects in the form of scene graphs or background knowledge in the form of a text description for the objects depicted in the image. In this paper, we address the issue of classifying hateful memes by performing an automated augmentation in order to represent the visual information and background knowledge (Fig. 1). We built on the MultiOFF dataset, which contains memes extracted from social media during the 2016 U.S. Presidential Election [28]. An off-the-shelf scene graph generation model was employed to produce scene graphs, which contain detected visual objects and relations between them for each meme, and a NER model to detect entities that we linked to background knowledge. The scene graphs were serialized as text resulting in a unified way to represent all the modalities expressed in a meme. This allows using a (unimodal) text classifier to classify the multimodal memes. Hence, we incorporated the automatically produced augmentations to classify the memes using a text-based Transformer model and show that they can improve its performance. The explicit representation of the image as serialized tokens also provides a more interpretable intermediate representation (compared to hidden layers in multimodal models such as ImgBERT [13, 14]). Furthermore, in order to examine how the results of the automatic augmentation would deviate from human ones, we performed manual augmentation. Two human evaluators corrected the automatically produced scene graphs and manually added background knowledge. We compare our automatic MemeGraphs method with models operating on the manually augmented data (only for training or for both training and inference) and find that automatic augmentations assist in achieving better results than manual ones. Contribution Our contributions can be summarized as follows: • We propose MemeGraphs, a novel method for classifying memes utilizing scene graphs augmented with knowledge and providing insights for processing multimodal documents. • We show that adding this kind of knowledge to a text-based Transformer model can improve its classification performance. Furthermore, we show that this yields improvements compared to a simple model only using learned representations to classify the memes, such as ImgBERT. • We conduct extensive experiments with manual and automatic settings for obtaining this knowledge and show that the automatic setting of our MemeGraphs method provides more meaningful information. In the following, we discuss related work and present our MemeGraphs method. Subsequently, in Section 4, we describe the models we implemented and report their results. Finally, we provide a qualitative analysis, including a discussion of the findings of human augmentation, and compare this with the automatic MemeGraphs method.[2] [1] Disclaimer: This paper contains examples of hateful content. [2] The code and data for the MemeGraphs method are available on: https://github.com/vasilikikou/memegraphs. This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References ABSTRACT Memes are a popular form of communicating trends and ideas in social media and on the internet in general, combining the modalities of images and text. They can express humor and sarcasm but can also have offensive content. Analyzing and classifying memes automatically is challenging since their interpretation relies on the understanding of visual elements, language, and background knowledge. Thus, it is important to meaningfully represent these sources and the interaction between them in order to classify a meme as a whole. In this work, we propose to use scene graphs, that express images in terms of objects and their visual relations, and knowledge graphs as structured representations for meme classification with a Transformer-based architecture. We compare our approach with ImgBERT, a multimodal model that uses only learned (instead of structured) representations of the meme, and observe consistent improvements. We further provide a dataset with human graph annotations that we compare to automatically generated graphs and entity linking. Analysis shows that automatic methods link more entities than human annotators and that automatically generated graphs are better suited for hatefulness classification in memes. Keywords hate speech · internet memes · knowledge graphs · multimodal representations. Keywords Keywords 1 Introduction Internet memes are items such as images, videos, or twitter posts that are widely shared on social media and typically relate to several subjects, such as politics, social, news, and current internet trends.[1] Memes are a popular form of communication and they are often used as a means to express an opinion or stance in a humorous or sarcastic manner, but they can also be hateful and promote problematic content that is likely to hurt specific groups of people and hence be harmful to society in general [2]. Thus, analyzing trending memes can provide insight into people’s reactions and opinions to important societal matters, as well as to the traits of different groups. This can help in tasks like filtering out harmful memes from internet platforms or extracting user opinions for socioeconomic studies. This work focuses on memes in the form of images with some form of superimposed text (sometimes referred to with the technical term image macros) with the goal of detecting hateful content. Lately, there has been increased interest in deep learning models for analyzing memes and classifying them [23, 10, 11], for example in the context of the Hateful Memes competition organized by Facebook [13] and the shared task in the Workshop on Online Abuse and Harms (WOAH) 2021 that was the continuation of the first competition [21]. Another shared task in Semeval 2022 aimed at detecting misogyny in memes [11]. Alongside these shared tasks, the corresponding datasets were published with human annotations capturing important properties such as hatefulness and misogyny. Recent works use multimodal representation learning based on image features from Convolutional Neural Networks (CNNs), Transformer-based language models for the text [13, 3, 14] or multimodal (vision+language) Transformer models in order to classify memes [13]. Some works additionally incorporate specifically extracted image  features such as race [33], person attributes [1, 23], and automatic image captioning output [7, 4]. However, none of these works have included the relations between objects in the form of scene graphs or background knowledge in the form of a text description for the objects depicted in the image. In this paper, we address the issue of classifying hateful memes by performing an automated augmentation in order to represent the visual information and background knowledge (Fig. 1). We built on the MultiOFF dataset, which contains memes extracted from social media during the 2016 U.S. Presidential Election [28]. An off-the-shelf scene graph generation model was employed to produce scene graphs, which contain detected visual objects and relations between them for each meme, and a NER model to detect entities that we linked to background knowledge. The scene graphs were serialized as text resulting in a unified way to represent all the modalities expressed in a meme. This allows using a (unimodal) text classifier to classify the multimodal memes. Hence, we incorporated the automatically produced augmentations to classify the memes using a text-based Transformer model and show that they can improve its performance. The explicit representation of the image as serialized tokens also provides a more interpretable intermediate representation (compared to hidden layers in multimodal models such as ImgBERT [13, 14]). Furthermore, in order to examine how the results of the automatic augmentation would deviate from human ones, we performed manual augmentation. Two human evaluators corrected the automatically produced scene graphs and manually added background knowledge. We compare our automatic MemeGraphs method with models operating on the manually augmented data (only for training or for both training and inference) and find that automatic augmentations assist in achieving better results than manual ones. Contribution Our contributions can be summarized as follows: Contribution • We propose MemeGraphs, a novel method for classifying memes utilizing scene graphs augmented with knowledge and providing insights for processing multimodal documents. • We show that adding this kind of knowledge to a text-based Transformer model can improve its classification performance. Furthermore, we show that this yields improvements compared to a simple model only using learned representations to classify the memes, such as ImgBERT. • We conduct extensive experiments with manual and automatic settings for obtaining this knowledge and show that the automatic setting of our MemeGraphs method provides more meaningful information. In the following, we discuss related work and present our MemeGraphs method. Subsequently, in Section 4, we describe the models we implemented and report their results. Finally, we provide a qualitative analysis, including a discussion of the findings of human augmentation, and compare this with the automatic MemeGraphs method.[2] [1] Disclaimer: This paper contains examples of hateful content. [2] The code and data for the MemeGraphs method are available on: https://github.com/vasilikikou/memegraphs. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

MemeGraphs: Linking Memes to Knowledge Graphs

The current communication model in the creative industries stagnates because of its own shortcomings, while not being fully interactive, it remains hierarchical and mediated. This research contributes to a deep understanding of the features of new decentralized technologies, and in particular, non-fungible tokens (NFTs), and gives an overview of ​​their potential positive impact on the communication process in the creative industries. The term ‘non-fungible’ is used in economics to denote the possession of unique objects and to describe things that cannot be replaced by others because they have a set of unique properties. A ‘token’ as a unit of account is a record in a distributed blockchain that is controlled by a computer algorithm of a smart contract, in which the values of the balances on the accounts of token holders are recorded, making it possible to transfer them from one wallet to another. Thus, refer to cryptographic tokens that represent digital files, images, audio, video, video game collectibles, and other creative products. Unlike cryptocurrency, which requires all tokens to be identical, each collected NFT token is unique or limited in quantity. non-fungible tokens (NFTs) The academic literature lacks a concrete and systematic review of the current state-of-the-art non-fungible tokens, and their implications for communication in creative industries, a limitation that was the primary driver for conducting this research. In particular, I try to address this by answering the following questions: What are the potential and benefits of distributed ledger technologies, blockchain, and NFTs for creative industries? How has their emergence and spread in recent years transformed the principles of communication? (i) (ii) Creative industries: approaches to definition and communication models The concept of creative industries has been used since the 1990s and marks the broad wave of the digital era. It usually refers to a variety of economic activities which involve the generation or use of knowledge and information and include art, design, music, film, fashion, etc. The term is often associated with the cultural industries or the creative economy . [6] [7] Considering different approaches to defining creative industries, the researchers distinguish four groups. The first approach is that the spectrum of creative industries is composed of specific sectors of creative industries. This approach is supported by researchers who put the creative industries on a par with other industries The second approach considers the creative industries as a group of certain occupations - this approach is reflected mainly in the concept of the ‘creative class’, in which the sphere of creative industries is associated with professions and fields of activity . The third approach is the approach of holistic urban development, which includes the concept of ‘creative city’ proposed by C. Landry The fourth approach views the creative industries as an important part of the broader economic system, supporting the new economy and composed of a constellation of workers, firms, institutions, infrastructures, communication channels, and other active components [6; 12]. [4] [8]. [14]. Modern researchers emphasize that creative industries should not be viewed simply as a sector of the economy or an isolated area, because there are forms and practices with which creative industries are inextricably linked, such as creative networks, linkages, creative spaces, and events Today, art, design, music, video, and other products of the creative industries, as well as the process of their creation, are often present in digital forms and spaces, greatly simplifying the interaction of industry actors. However, the situation opens the door to new complexities and opportunities for shenanigans. Part of the interaction is still the shadow activity, and there are no generally accepted industry standards. Established in recent decades, the communication model remains indirect and beneficial only to third parties, remaining problematic and dishonest to content creators and their audiences. The ability to remove an intermediary and to create direct open interaction between all actors appears with the proliferation of new distributed ledger technologies such as blockchain in recent years. [11]. Blockchain: the possibility of direct and open communication New technologies which are now widely used in creative industries, becoming a means to directly foster creativity, while contributing to the life and culture of society as a whole, as well as to find ways to overcome obstacles and solve specific industries’ problems . Primarily the distributed ledger technologies (DLTs), which had a strong influence on the industries lately, are here to rescue them. [9] Conceptualized by a person, or group of people, using the name Satoshi Nakamoto in 2008 to serve as the public transaction ledger of the cryptocurrency Bitcoin, is considered one of the DLTs, a distributed append-only time-stamped data structure that contains an ordered list of various records and these records are connected together through links called chains It allows for a peer-to-peer network where non-trusting members can verifiably interact with each without the need for a trusted authority blockchain [17;18]. [1]. Blockchain which is expected to reshape human social activities, and communication specifically, is on the way to mass adoption. According to Accenture, an application of the diffusion of innovations theory suggests that blockchains attained a 13.5% adoption rate within financial services in 2016, therefore reaching the early adopters phase that year. It has been seen as an emerging technology that enhances decentralized computation in 2018 . In 2020, research and advisory firm Gartner listed blockchain as one of the top ten strategic technologies. Currently, interest from market leaders contributes to the adoption of DLT technology, especially to the development and widespread of NFT-related solutions. [13] The phenomenon of non-fungible tokens: genesis, features, and standards A token as a unit of account is a record in a distributed blockchain and is controlled by a computer algorithm of a smart contract, in which the values ​​of the balances on the accounts of token holders are recorded, making it possible to transfer them from one wallet to another. Non-fungible tokens were originally created as special token standards to support the use of blockchain in computer games, which include the Ethereum ERC-721 standard (which is representing non-fungible digital assets) and the more recent ERC-1155 standard (which offers ‘semi-fungibility’). The first prototypes of NFTs were experimental assets created on the Bitcoin network in 2012 named Colored coins. The pioneer of NFTs based on the Ethereum blockchain was the CryptoPunks project. At the end of 2017, an online game that allows users to breed digital cats - the CryptoKitties startup gained great popularity, opening the way to a wide audience of NFTs: for the first time, a popular non-financial application was deployed on the basis of blockchain, the community realized the potential of NFTs, and their supply grew significantly in the following years: in 2018 and 2019, more than 100 projects were presented. In 2021, the NFTs ecosystem grew at an unprecedented rate. According to the report from NonFungible.com with NFTs industry statistics, the market of non-fungible tokens surpassed $250m in 2020, up nearly 300% from the previous 2019. Of all NFTs segments, virtual worlds and artworks occupied the most part, but computer games were ahead in the number of transactions (almost 50% of NFTs transactions in 2020 were associated with them), and collectibles were not far behind. The most used platform for hosting NFTs is OpenSea, a universal platform that allows users to trade NFTs, create them, view data on them, and check statistics. Another popular marketplace for buying, selling, and creating NFTs, is Mintbase which excels in covering niche NFTs categories such as music, photography, and news. Rarible and Zora platforms won great success among fans of crypto-art since they are not curated and in order to upload the art-NFTs on them, users only need to register and pay the transaction fee to mint a token. Forby, the communication of creative industries is increasingly moving into the digital space. For example, many digital artworks are already placed in online collections of virtual worlds. The first blockchain-based virtual world - Cryptovoxels has several art galleries where users can buy art in the form of NFTs. As the virtual reality space becomes more popular, displaying digital art and the direct communication it offers is becoming more and more familiar. Non-fungible tokens as a new communication paradigm for creative industries Discussing the use of communication models in the social sciences, including economic, cultural and creative fields, researchers identified their three advantages. First, they perform an organizing function, giving the opportunity to understand how a particular system works. Second, the communication models serve an explanatory function, indicating in a simplified way how the system operates. Third, models allow us to predict the outcome of actions and events . The evolution of these communication models can be distinguished into three main stages. In the first stage, the emphasis was clearly on linear one-way communication, and the main term was manipulation. This type of communication is common in Western culture (Aristotle, Lasswell, Shannon and Weaver, Berlo). The communication models developed in the second phase were largely aimed at addressing some of the previous shortcomings, conceptualizing communication as a two-way process, the interaction between the communicator and the receiver (Schramm, Gerbner, Riley and Riley), and the third model (Barnlund, Kripendorff, Kincaid, Derwen, Clark and Delia) sees communication as interaction in specific socio-cultural contexts through shared meanings. [2] The emergence of distributed ledger technologies changed the landscape drastically, as the advantages they propose for changing the communication paradigm in social fields and hence in creative industries have already manifested themselves in recent years , locally transforming an indirectly hierarchical model. [15] First and foremost, DLTs have created the possibility to eliminate intermediaries from the communication process, allowing content creators to connect directly with their audience and honestly monetize their work. Thus, it became possible to automatically pay royalties, fairly distributed in accordance with the contribution to the creative process of each of the participants. There are no more third parties in the face of labels, large distributors, or art portals, restricting the rights of creators of digital goods. This improvement is to lead to a more diverse and transparent industry. Due to blockchain technology, it became possible to create large distributed databases of digital rights, allowing participants in the creative industries to interact with each other and with materials protected by copyright, as well as with all their own metadata. It is now possible to manage tokenized rights with the deployment of smart contracts that actually manage them and maximize their value. In addition, open blockchains are distributed, so each actor can now see the exact time of creation of each digital object and the number of interactions with it, what income it brings, and who gets what percentage, which makes communication across industries direct and potentially fair. Moreover, NFTs open up new monetization opportunities for artists. If in the traditional art world, content creators usually did not receive a percentage of secondary sales, thanks to NFTs, today a digital artist can receive them in a fully automated way. What’s more, blockchain platforms are designed to make the experience interactive and profitable for the audience as well, rewarding users not only with exclusive content and lower prices, but by offering all kinds of incentives e.g. for compiling personalized collections, for contributing to increasing the database of artists, venues, or events, etc. Conclusion Crypto enthusiasts see a future where content creators will be paid fairly and the value of digital items will grow. Taking into account all the advantages that DLTs offer, following the widespread of the NFTs phenomenon and the increase in the number of users of crypto platforms for trading digital objects, today it seems like a very real prospect. Blockchain is close to solving the main industries' problem of the dominance of the hierarchical structure of communication and the mediated relationship between the actors. Only a small breakthrough is needed in a long-expected understanding that the creative industries are capable of honesty towards content creators, transparency, and hence even more impressive prosperity that they have shown recently. Nevertheless, some issues still remain unsolved. For example, there are still no official public registers of copyright, and no global standards of licenses, since different rules apply in different regions. Also, for the audience and consumers, it is still more common to use familiar major services, and many representatives of the creative industries have yet to meet the basic advantages of blockchain platforms and thus understand the benefits that NFTs can offer them. The topic of the potential of non-fungible tokens in the context of ongoing communication transformation in creative industries is open for further research. References (2019) A systematic literature review of blockchain-based applications: Current status, classification and open issues. Telematics and Informatics. Vol.36, 55-81. Casino F., Dasaklis T. K., Patsakis C. (1966) Nerves of Government: Models of Political Communication by   Paperback. K. W. Deutsch (1984) Communication Models and Knowledge Dissemination, Media Asia, 11:3, 123-128, DOI: 10.1080/01296612.1984.11726150 Dissanayake W. (2002). The Rise of the Creative Class. And How It’s Transforming Work, Leisure, Community and Everyday Life. New York: Basic Books. Florida, R. (2007) Defining Strategic Communication. International Journal of Strategic Communication 1(1):3-35. Hallahan K., Holtzhausen D. R. et al. (2002), The Cultural Industries, SAGE. Hesmondhalgh, D. (2001) The Creative Economy: How People Make Money From Ideas, Penguin. Howkins, J. (2012) The Creative City: A Toolkit for Urban Innovators, Earthscan. Landry, C. (2006) Literature Review in Creativity, New Technologies and Learning, Report 4, Future Lab, School of Education. University of Brighton. Loveless, A. M. (2018) Blockchain and the related issues: a review of current research topics. Journal of Management Analytics 5(3):1-25. Lu, Y. (2014) Cultural and Creative Industries Concept – A Historical Perspective. Procedia - Social and Behavioral Sciences, Vol. 110. 738-746. Moore, I. (2008) Four models of the creative industries. International Journal of Cultural Policy, 14(3), 233–247. Potts, J., & Cunningham, S. (2018) Everything You wanted to know about the blockchain: Its promise, components. Processes, and Problems. IEEE Consumer Electronics Magazine, 7(4), 6–14. Puthal, D., Malik, N., Mohanty, S. P., Kougianos, E., & Das, G. (2006) Creative Cities: Conceptual Issues and Policy Questions Capitalism, cities, and the production of symbolic forms. Centre for Globalization and Policy Research, School of Public Policy and Social Research, University of California-Los Angeles, USA. Scott, A. J. (2021) DLT and Blockchain in the 2021 Music Industry: State of the Art. Medium. URL: https://medium.com/paradigm-fund/dlt-and-blockchain-in-the-2021-music-industry-state-of-the-art-5e14c8ff9dc0 [accessed 25.04.2021] Shilina S. (2015) Blockchain: Blueprint for a new economy. Beijing: O’Reilly Media. Swan, M. (2017). MeDShare: Trustless medical data sharing among cloud service providers via blockchain. IEEE Access, 5, 14757–14767. Xia, Q., Sifah, E. B., Asamoah, K. O., Gao, J., Du, X., & Guizani, M. (2020) A review of research relevant to the emerging industry trends: Industry 4.0, IoT, blockchain, and business analytics. Journal of Industrial Integration and Management, 5(1), 165–180. Zhang, C., & Chen, Y.

Stories, News & Mentions about KOMUGI

Komugi’s Stories on HackerNoon