Kiefa is purpose built to enable data driven decisions and support your growth. Reach out to us at success@kiefa.io

Kiefa

This is by ( ). Drew is a writer. Twitter thread Kieran Drew @ItsKieranDrew source: 02-18-2023 https://twitter.com/ItsKieranDrew/status/1626931820267728902?embedable=true&thread=true The lead image for this article was generated by HackerNoon's via the prompt "Road ahead.” AI Image Generator

Kieran Drew: 10 Incredible Lessons I Learned After I Broke My Neck 

Hi Kieran, welcome to our series “Behind the Startup.” Please tell us about yourself and the story behind Railgun? Ishan Pandey: Railgun was ideated and founded by Emmanuel Goldstein as a way to enable privacy-preserving smart contract interactions on Ethereum. I believe privacy and self-determination are fundamental human rights that have been heavily eroded with the advent of modern technology. While Ethereum has great potential to reclaim the right to self-determination, its public nature coupled with critical design shortfalls (account-model) that make it trivial to track user actions should be cause for concern, especially when dealing with money. As a lifelong believer in the right to privacy, Mr Goldstein’s pitch made perfect sense to me. Kieran Mesquita: Prior to Railgun, I was already heavily involved in multiple blockchain projects. I started in the early days of Bitcoin trying to get an edge in CPU and later GPU mining, and later I engaged in multiple blockchain projects covering things like pushing the boundaries on running wallets on lower and lower-cost devices, permissionless crowdfunding, consensus algorithms, cross-chain bridges, and DEX’s. Besides blockchain, I’ve dabbled few other fields, namely astronomy and hobby electronics. I’m currently most active developing on Ethereum and the smart contract space. Can you explain how Zero-knowledge proof technology works? Ishan Pandey: At a high level, zero-knowledge proofs are a system whereby a prover can prove something to a verifier without revealing the details about the thing being verified. For example, an ID checks where the prover wishes to prove they are over 18 without revealing the details on their ID or revealing their exact age can be done with zero-knowledge proof. Kieran Mesquita: Getting into the implementation details, a zero-knowledge proving system takes a program represented as an arithmetic circuit, a set of public inputs, and a set of private inputs, and outputs an execution proof that the program was successfully executed with the given inputs. The verifier can then check that the program was executed successfully with the public inputs and execution proof. In the above example, the public inputs could be a hash of the ID data (perhaps publicly published by the issuer of the ID so that authenticity can be verified) and today’s date. The private inputs would then be the fields on the ID, such as date of birth and name. The arithmetic circuit would then check that A) the hash of the ID fields matches the hash that was given as public input and B) the date of birth on the ID is at least 18 years earlier than today’s date. For example, there is a number of different zero-knowledge proving systems like plonk, marlin, halo, and the one used in Railgun – groth16. These properties are incredibly useful in privacy-orientated blockchain applications where actions need to be publically verifiable without revealing their contents. We can look at how zero-knowledge proofs are applied to private cryptocurrencies – note that I have used “owner” to mean the individual or software that has access to the private key required to take actions such as spending coins. Privacy-focused transaction systems generally use UTXO accounting and apply a zero-knowledge proof system to transactions. Transactions in a UTXO (Unspent Transaction Outputs) accounting system specify outputs from previous transactions as inputs and define a new set of outputs. Each output contains the parameters that specify how it can be spent (for standard wallets, this is simply a check that the transaction is signed with a particular key pair corresponding to a wallet address) and a value for the amount of coins that output represents. In order for a transaction to be valid, the spending requirements for each input needs to be satisfied and the sum of the outputs of the transaction cannot exceed the sum of its inputs. The zero-knowledge proof variation of the UTXO model uses some blinded form of the UTXO such as its hash or encrypted so that only the owner of the UTXO can decrypt it and the UTXOs are stored in a cryptographic accumulator such as a MerkleTree. UTXOs exist perpetually in the accumulator with a nullifier (a value that can be deterministically generated from the UTXO only by the owner, usually a hash of the private key plus some values contained in the UTXO) used to prevent double spends. The private inputs to the zero-knowledge proof for a transaction would be the input and output UTXOs. The public inputs are the current UTXO accumulator state (eg. the MerkleTree root), nullifiers for the input UTXOs and the blinded output UTXOs. The arithmetic circuit would verify that A) the UTXOs exist in the accumulator, B) the spender is the owner of the UTXOs, C) the nullifiers correctly correspond to the UTXOs, and D) the sum of the outputs is less than or equal to the sum of the inputs. The verifier would then simply need to check that the nullifiers had not been used in a previous transaction to prevent double-spending, that the accumulator state used is the current or previous state of the UTXO accumulator and then add the blinded UTXOs to the UTXO accumulator. As UTXOs are only stored in blinded form, the amounts and owners are hidden from public view and using nullifiers to prevent double-spending instead of removing UTXOs from the current state hides what UTXOs are spent in any particular transaction. Miners worldwide have just authorized the first Bitcoin upgrade in four years and it is called Taproot, effective from November this year. What does this upgrade mean for the cryptocurrency in terms of privacy and efficiency? Further, how do you think it will affect the market? Ishan Pandey: Taproot consists of two upgrades, schnorr and tapscript. Schnorr signatures are a lightweight signature scheme that has a number of advantages over ECDSA signatures but have had their adoption slowed due to being covered by a patent. The three advantages of schnorr signatures that are most applicable to Bitcoin are their size, non-malleability, and linearity. Kieran Mesquita: The smaller Schorr signature sizes are an obvious benefit in Bitcoin where space on the blockchain is a premium. Schnorr public keys are 32 bytes instead of 33 bytes and signatures are 65 bytes instead of 71 or 72 bytes with sighash flags. Schnorr signatures can be aggregated via an interactive aggregation scheme allowing transactions to be created with a signature that is simultaneously valid for all inputs, further compounding the space savings. ECDSA signatures are malleable, meaning someone can take a valid signature A of value X and alter it to produce a valid signature B of value X. Since changing the signature text changes the TxID of a transaction, it can lead to a number of attacks, for example: If an exchange was tracking withdrawals by TxID and an attacker was to alter the TxID via signature malleability and have the altered transaction mined, the exchange might never recognise that the withdraw transaction has been mined (this was the attack used against Mt.Gox). 1. Chained transactions in the mempool can be invalidated if the TxID of the parent transaction is altered since transactions reference UTXOs by TxID of the previous transaction. 2. Segwit partially solves transaction malleability by not including the transaction signature as part of the TxID hash preimage, ensuring the TxID is not changeable; however, witness data is still susceptible. Schnorr signatures ensure the entire transaction is non-malleable. This is useful for applications like the Lightning Network, which requires non-malleable to function securely. Linearity allows a group keys to be efficiently aggregated into a single public key that represents the group, where group keys can collaborate to produce a single signature that is valid for the group public key. Verification of schnorr signatures on-chain only requires storing/verifying a single signature, making multisigs cheaper. Tapscript aggregates multiple spend scripts for a UTXO into a merkle tree. Transactions spending from tapscript UTXOs need only reveal the spend script they are executing. This allows large scripts to be broken down into smaller spend scripts, reducing the size of the redeem script for these inputs and, therefore, the transaction cost to spend them. While Taproot decreases the size of Bitcoin transactions, it only marginally increases pseudonymity by obfuscating or revealing less of the redeem script information which provides some privacy benefits to institutions like Coinbase, such as hiding which signers in a multisig have authorized a particular transaction. It doesn’t provide any pseudonymity benefits for the average user and doesn’t do anything to increase anonymity. I’d consider taproot to be an upgrade to efficiency and not privacy primarily. I try to stay away from making predictions on market/price performance, especially as it is driven by narrative rather than the technical viability and merits of any proposed or implemented upgrade. Several experts predict that the cryptocurrency Ethereum will eventually overtake Bitcoin especially given the current volatility in the market and that Ethereum is programmable and faster than Bitcoin. What are your views on this prediction? Ishan Pandey: Ethereum has shown a willingness to acknowledge shortfalls and improve, whereas Bitcoin has stagnated with the community actively ignoring issues or labelling them as ‘features.’ In almost all metrics but price Ethereum has already overtaken Bitcoin. Kieran Mesquita: DeFi has ushered in an open alternative to the present banking system that promises greater financial inclusion. In your opinion, how has DeFi revolutionized the virtual assets economy, and do you think it should be more heavily regulated? Ishan Pandey: DeFi has brought a much-needed explosion of utility to the cryptocurrency space. If it is the goal of cryptocurrency to build a new financial system many more financial primitives need to be available. Financial systems consist of a lot more than just cash. Kieran Mesquita: As for regulators, they need to be careful not to stifle the innovation we see in the space. The most valuable thing the SEC did in 2018 was to create the HoweyCoins ICO. The messaging that we often see that blanket labels everything as a scam simply serves to make it harder to differentiate genuine scams from legitimate projects in the cryptocurrency space. I think education is a lot more valuable than regulation, and I can name a number of educational YouTubers that have done more for DeFi on a shoestring budget than regulators. Facebook intends to launch its own cryptocurrency, Diem, to make digital payments more convenient. Can Facebook be trusted to put the public’s interests ahead of its own in light of the accusations against the company on issues such as user privacy and data mining of users? Ishan Pandey: Diem is closer to PayPal with scripting capabilities than a decentralized cryptocurrency. Kieran Mesquita: An inclusive global financial system (as is Diem’s stated purpose) must respect the fundamental human rights to privacy and self-determination. These goals are not achievable through a currency whose governing body is composed of corporates whose business model stands in stark opposition to these goals. Instead, we need to look to currencies like Bitcoin, like Ethereum, where users need not trust any single entity to have faith in its integrity. Improving the privacy of the applications being built on these public chains is fundamental to realizing the goal of a truly global, fair, and inclusive financial system. CBDC projects are gaining traction around the world, and the ASEAN region will be no exception, with numerous central banks, commercial banks, and FinTechs promoting CBDC pilots, such as Singapore’s Project Ubin. What are your views on the ongoing CBDC developments and what is your take on it? Ishan Pandey: CBDCs are an incremental improvement to efficiency in the existing financial system but come with valid economic policy and privacy concerns. Historically there has been a layer of insulation between the wider economy which is influenced by fiscal policy, and the nation’s central bank, which has solely been responsible for balancing interest rates. The danger is central banks could slip away from the role of managing monetary health and become more akin to a branch of law enforcement, making decisions based on influences and goals that might be in conflict with maintaining a healthy economy. Kieran Mesquita: After China, Thailand has become the latest Asian jurisdiction to impose trading restrictions on highly speculative cryptocurrencies and NFTs, including meme coins like Dogecoin. What is your opinion on the move of the regulators and the rationale behind it? Ishan Pandey: Modern-day societies have long been skewed in favour of the de facto elite that has access to hedge funds and other financial institutions. The average person has been disenfranchised and ignored for so long that things like the GameStop/AMC phenomenon and now ‘joke coins’ like Dogecoin have begun to serve as an outlet to protest against Wall Street and the large institutions. While well-intended perhaps, regulators taking actions to shutter this escape valve instead of addresses the wider systemic problems does little to protect the little guy as they are chartered to do. Kieran Mesquita: According to you, what new trends will we see in the cryptocurrency industry? Ishan Pandey: Narratives around cryptocurrency change so quickly that predicting future trends requires pulling out your crystal ball. However, I can talk about where I would hope to see cryptocurrency move in the future: privacy, privacy, privacy. While privacy coins have been around for 6-7 years at this point, an eon in cryptocurrency terms, they’ve been mostly restricted to purely transactional use cases. Historically we have not seen privacy solutions applied to the areas where the most economic activity is happening (like DeFi and NFTs). I would hope to see a renewed focus on privacy in other areas where blockchain is seeing applications. Kieran Mesquita: Disclaimer: The purpose of this article is to remove informational asymmetry existing today in our digital markets by performing due diligence by asking the right questions and equipping readers with better opinions to make informed decisions. The material does not constitute any investment, financial, or legal advice. Please do your research before investing in any digital assets or tokens, etc. The writer does not have any vested interest in the company. Ishan Pandey.

"Privacy and Zero-Knowledge Proof Are Going To Be Critical for A Free Society" - Kieran Mesquita

Scene Graphs & WikiData in Meme Analysis: Advancing Hate Speech Detection

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References 2 Related Work Combining text and image inputs is crucial for many tasks, e.g. for image search or (visual) question answering, relying on an image and its caption. Research has shown that adding images to text-based tasks (e.g., machine translation) improves the performance of the models [31]. However, the meaningful interpretation of text and image and, in particular, the relations between them, still remains challenging [17, 5]. Commonly used approaches rely on Transformer models that are pre-trained on image+text pairs [18, 20, 6, 12, 27, 19]. A step towards better scene understanding is to generate scene graphs [15]. Scene graphs provide structured knowledge about an image, e.g., objects, relations, and attributes. Recent works have shown that we can improve scene graph generation using message propagation between entities [32, 29], and by employing background knowledge in form of knowledge graphs [26], texts [25], or using feedback connections. In [26], the authors proposed Schemata, a scene graph generation model consisting of two parts: the backbone module and the relational reasoning component. Additionally, Schemata uses feedback connections to further encourage the propagation of higher-level, class-based knowledge to each neighbor. The backbone is pre-trained on ImageNet [8] and the whole network is fine-tuned on Visual Genome [15] on the scene graph classification task. Scene graphs can help to achieve state-of-the-art results in several visual tasks [30, 22]. Inspired by these approaches, we generate scene graphs to represent the visual information contained in memes by using the Schemata model (see Section 3). A specific instance of vision and language tasks is memes classification. To address the need for automatic means that can detect hateful content in memes, datasets and models [28, 3, 23] were published in the last couple of years, and shared tasks [13, 21, 11] were organized to attract interest on this task. Methods that have been implemented for hateful memes detection can be grouped into three categories: 1. Unimodal methods that use either only the text or the image as input, 2. Multimodal approaches, where image embeddings from an image encoder are fed to a text model and both models are trained separately, and 3. Multimodal methods, consisting of vision+language Transformers, being pre-trained in a multimodal fashion. Current methods experiment with models from all three categories and focus on improving models from the third category by adding extra features [3, 23, 4, 16]. These features can be visual attributes extracted from CNNs [13, 16, 1] (e.g. objects, entities or demographics), representations from CLIP [24, 23], automatically generated captions [7, 4], etc. The Hateful Memes Challenge, hosted in 2020 by Facebook, was a binary classification task of hate detection [13].[3] Kiela et al. [13] created a dataset with 10,000 memes to which they added counterfactual examples in order to make the task more challenging for unimodal approaches. They experimented with several different settings and found that multimodal methods worked best. An extended version of the Hateful Memes Challenge was included as a shared task in the Workshop on Online Abuse and Harms (WOAH) [21]. The same dataset was used but it now included new fine-grained labels for two categories: protected category and attack type. In this shared task, multimodal approaches were dominant as well. A multimodal method introduced by [13] and subsequently also used for the shared task in WOAH [14] incorporated image embeddings as inputs to a text classifier.[4] This method belongs to the second category and is an early fusion approach meaning that the image embedding and the text embedding are concatenated before feeding them to the classifier. Different types of image and text components are employed in different works. Specifically, in ImgBERT [14], first, they feed the memes images to a convolutional neural network (CNN) and extract their embeddings. Then, they provide the text of the meme as input to BERT [9] and extract the [CLS] token representation. They concatenate the [CLS] token representation with the embedding of the meme’s image and use the result as input to the classifier. During training only the text-based BERT part of ImgBERT is trained, while the image embeddings remain frozen. Another dataset for detecting hateful content in memes is the MultiOFF dataset [28]. It contains memes that were extracted from social media during the 2016 U.S. presidential elections. The dataset was first shared on Kaggle and consisted of the image URL for each meme, its text and metadata, e.g., timestamp, author, likes, etc.[5] The authors obtained the images from the URLs and discarded any metadata. In total, this dataset contains 743 memes, which were annotated as hateful or non-hateful. In [28] the authors experimented with unimodal (text only) and multimodal (text and image) approaches, and the model with the highest F1 score was a CNN operating only on the text of the memes. The above mentioned datasets focus on hate and offensive speech, but there are also datasets that cover other aspects of harmful content in memes. In [10], the authors focused on detecting propaganda in memes. They created and released a dataset with 950 memes extracted from Facebook groups, annotated for 22 different propaganda techniques. In their experiments they used existing unimodal and multimodal models and found that the latter, especially multimodally pre-trained Transformers perform best in their setting. Recently, a challenge called Multimedia Automatic Misogyny Identification (MAMI) focused on detecting misogyny and its exact form, i.e., stereotype, shaming, objectification and violence in memes [11]. In [23], they studied harm in memes and proposed a framework to detect harmful memes and the entities targeted. The authors also released their dataset with 7,096 memes in total about politics and COVID-19. The existing challenge sets, resources and models show the importance of analyzing internet memes and the challenge to combine all the modalities that form a meme. However, current works focus on incorporating visual information in the form of individual features like the ones described above or automatically generated captions. We propose a novel approach to represent the visual content of memes using scene graphs, hence "translating" them into text form. Furthermore, current methods only extract entities from the images, but not from the captions or texts. We argue that often this is not sufficient (or feasible), since memes can also incorporate screenshots of text, as it is the case in the MultiOFF dataset. Hence, we approach this problem by extracting the entities from the text in order to obtain more information. We further retrieve background knowledge for each extracted entity, and show that this approach is worthwhile to explore, since it allows for a more grounded and comprehensive automatic interpretation of memes. [3] https://www.drivendata.org/competitions/64/hateful-memes/ [4] The method was called Concat BERT in [13] and ImgBERT in [14]. Here we call it ImgBERT because we use their implementation. [5] https://www.kaggle.com/datasets/SIZZLE/2016electionmemes This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References 2 Related Work Combining text and image inputs is crucial for many tasks, e.g. for image search or (visual) question answering, relying on an image and its caption. Research has shown that adding images to text-based tasks (e.g., machine translation) improves the performance of the models [31]. However, the meaningful interpretation of text and image and, in particular, the relations between them, still remains challenging [17, 5]. Commonly used approaches rely on Transformer models that are pre-trained on image+text pairs [18, 20, 6, 12, 27, 19]. A step towards better scene understanding is to generate scene graphs [15]. Scene graphs provide structured knowledge about an image, e.g., objects, relations, and attributes. Recent works have shown that we can improve scene graph generation using message propagation between entities [32, 29], and by employing background knowledge in form of knowledge graphs [26], texts [25], or using feedback connections. In [26], the authors proposed Schemata, a scene graph generation model consisting of two parts: the backbone module and the relational reasoning component. Additionally, Schemata uses feedback connections to further encourage the propagation of higher-level, class-based knowledge to each neighbor. The backbone is pre-trained on ImageNet [8] and the whole network is fine-tuned on Visual Genome [15] on the scene graph classification task. Scene graphs can help to achieve state-of-the-art results in several visual tasks [30, 22]. Inspired by these approaches, we generate scene graphs to represent the visual information contained in memes by using the Schemata model (see Section 3). A specific instance of vision and language tasks is memes classification. To address the need for automatic means that can detect hateful content in memes, datasets and models [28, 3, 23] were published in the last couple of years, and shared tasks [13, 21, 11] were organized to attract interest on this task. Methods that have been implemented for hateful memes detection can be grouped into three categories: 1. Unimodal methods that use either only the text or the image as input, 2. Multimodal approaches, where image embeddings from an image encoder are fed to a text model and both models are trained separately, and 3. Multimodal methods, consisting of vision+language Transformers, being pre-trained in a multimodal fashion. Current methods experiment with models from all three categories and focus on improving models from the third category by adding extra features [3, 23, 4, 16]. These features can be visual attributes extracted from CNNs [13, 16, 1] (e.g. objects, entities or demographics), representations from CLIP [24, 23], automatically generated captions [7, 4], etc. The Hateful Memes Challenge, hosted in 2020 by Facebook, was a binary classification task of hate detection [13].[3] Kiela et al. [13] created a dataset with 10,000 memes to which they added counterfactual examples in order to make the task more challenging for unimodal approaches. They experimented with several different settings and found that multimodal methods worked best. An extended version of the Hateful Memes Challenge was included as a shared task in the Workshop on Online Abuse and Harms (WOAH) [21]. The same dataset was used but it now included new fine-grained labels for two categories: protected category and attack type. In this shared task, multimodal approaches were dominant as well. A multimodal method introduced by [13] and subsequently also used for the shared task in WOAH [14] incorporated image embeddings as inputs to a text classifier.[4] This method belongs to the second category and is an early fusion approach meaning that the image embedding and the text embedding are concatenated before feeding them to the classifier. Different types of image and text components are employed in different works. Specifically, in ImgBERT [14], first, they feed the memes images to a convolutional neural network (CNN) and extract their embeddings. Then, they provide the text of the meme as input to BERT [9] and extract the [CLS] token representation. They concatenate the [CLS] token representation with the embedding of the meme’s image and use the result as input to the classifier. During training only the text-based BERT part of ImgBERT is trained, while the image embeddings remain frozen. Another dataset for detecting hateful content in memes is the MultiOFF dataset [28]. It contains memes that were extracted from social media during the 2016 U.S. presidential elections. The dataset was first shared on Kaggle and consisted of the image URL for each meme, its text and metadata, e.g., timestamp, author, likes, etc.[5] The authors obtained the images from the URLs and discarded any metadata. In total, this dataset contains 743 memes, which were annotated as hateful or non-hateful. In [28] the authors experimented with unimodal (text only) and multimodal (text and image) approaches, and the model with the highest F1 score was a CNN operating only on the text of the memes. The above mentioned datasets focus on hate and offensive speech, but there are also datasets that cover other aspects of harmful content in memes. In [10], the authors focused on detecting propaganda in memes. They created and released a dataset with 950 memes extracted from Facebook groups, annotated for 22 different propaganda techniques. In their experiments they used existing unimodal and multimodal models and found that the latter, especially multimodally pre-trained Transformers perform best in their setting. Recently, a challenge called Multimedia Automatic Misogyny Identification (MAMI) focused on detecting misogyny and its exact form, i.e., stereotype, shaming, objectification and violence in memes [11]. In [23], they studied harm in memes and proposed a framework to detect harmful memes and the entities targeted. The authors also released their dataset with 7,096 memes in total about politics and COVID-19. The existing challenge sets, resources and models show the importance of analyzing internet memes and the challenge to combine all the modalities that form a meme. However, current works focus on incorporating visual information in the form of individual features like the ones described above or automatically generated captions. We propose a novel approach to represent the visual content of memes using scene graphs, hence "translating" them into text form. Furthermore, current methods only extract entities from the images, but not from the captions or texts. We argue that often this is not sufficient (or feasible), since memes can also incorporate screenshots of text, as it is the case in the MultiOFF dataset. Hence, we approach this problem by extracting the entities from the text in order to obtain more information. We further retrieve background knowledge for each extracted entity, and show that this approach is worthwhile to explore, since it allows for a more grounded and comprehensive automatic interpretation of memes. [3] https://www.drivendata.org/competitions/64/hateful-memes/ [4] The method was called Concat BERT in [13] and ImgBERT in [14]. Here we call it ImgBERT because we use their implementation. [5] https://www.kaggle.com/datasets/SIZZLE/2016electionmemes This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Advancements in Memes Analysis: Scene Graphs and Multimodal Approaches

Hateful Meme Detection: Leveraging PVLMs for Zero-shot VQA Probing

Hybrid Meta-Heuristic for Efficient Course Timetabling

Is GPT Powerful Enough to Analyze the Emotions of Memes?: Discussion and References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Muzhaffar Hazman, University of Galway, Ireland; (2) Susan McKeever, Technological University Dublin, Ireland; (3) Josephine Griffith, University of Galway, Ireland. Table of Links Abstract and Introduction Related Works Methodology Results Limitations and Future Works Conclusion, Acknowledgments, and References A Hyperparameters and Settings B Metric: Weighted F1-Score C Architectural Details D Performance Benchmarking E Contingency Table: Baseline vs. Text-STILT 6 Conclusion In this work, we addressed the challenge of training multimodal meme sentiment classifiers on a limited number of labelled memes by incorporating unimodal sentiment analysis data. We did so by proposing the first instance of STILT that applies unimodal intermediate tasks to a multimodal target task. Specifically, we tested image-only and text-only sentiment classification as intermediate tasks in training a meme sentiment classifier. We showed that this approach worked – unimodal text improved meme classification performance to a statistically significant degree. This novel approach allowed us to train a meme classifier that outperforms meme-only finetuning with only 60% as many labelled meme samples. As possible explanations for our observations, we discuss apparent similarities and differences in the roles of image and text modalities between unimodal and multimodal sentiment analysis tasks. Acknowledgments This work was conducted with the financial support of the Science Foundation Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224; and the provision of computational facilities and support from the Irish Centre for High-End Computing (ICHEC). References Christian Bauckhage. 2011. Insights into internet memes. Proceedings of the International AAAI Conference on Web and Social Media, 5(1):42–49. Efrat Blaier, Itzik Malkiel, and Lior Wolf. 2021. Caption enriched samples for improving hateful memes detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9350–9358, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Ana-Maria Bucur, Adrian Cosma, and Ioana Iordache. 2022. BLUE at memotion 2.0 2022: You have my image, my text and my transformer. In De-Factify @ AAAI 2022. First Workshop on Multimodal FactChecking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. CrowdFlower. 2016. Image sentiment polarity - dataset. Available at https://data.world/ crowdflower/image-sentiment-polarity. Accessed: 2023-01-15. Baishan Duan and Yuesheng Zhu. 2022. Browallia at memotion 2.0 2022 : Multimodal memotion analysis with modified ogb strategies (short paper). In DeFactify @ AAAI 2022. First Workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Noam Gal, Limor Shifman, and Zohar Kampf. 2016. “it gets better”: Internet memes and the construction of collective identity. New Media & Society, 18(8):1698–1714. Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018. Hybrid Attention based Multimodal Network for Spoken Language Classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2379–2390, Santa Fe, New Mexico, USA. Association for Computational Linguistics. Muzhaffar Hazman, Susan McKeever, and Josephine Griffith. 2023. Meme sentiment analysis enhanced with multimodal spatial encoding and face embedding. In Artificial Intelligence and Cognitive Science, pages 318–331, Cham. Springer Nature Switzerland. Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. 2022. Transferability in deep learning: A survey. arXiv preprint, arXiv:2201.05867. Vishal Keswani, Sakshi Singh, Suryansh Agarwal, and Ashutosh Modi. 2020. IITK at SemEval-2020 task 8: Unimodal and bimodal sentiment analysis of Internet memes. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1135–1140, Barcelona (online). International Committee for Computational Linguistics. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, et al. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33:2611–2624. Liisi Laineste and Piret Voolaid. 2017. Laughing across borders: Intertextuality of internet memes. The European Journal of Humour Research, 4(4):26–49. Gwang Gook Lee and Mingwei Shen. 2022. Amazon pars at memotion 2.0 2022: Multi-modal multi-task learning for memotion 2.0 challenge. In De-Factify @ AAAI 2022. First Workshop on Multimodal FactChecking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. Shreyash Mishra, S Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. 2023. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes. arXiv preprint, arXiv:2303.09892. Niklas Muennighoff. 2020. Vilio: state-of-the-art visiolinguistic models applied to hateful memes. arXiv preprint arXiv:2012.07788. Thanh Van Nguyen, Nhat Truong Pham, Ngoc Duy Nguyen, Hai Nguyen, Long H. Nguyen, and YongGuk Kim. 2022. Hcilab at memotion 2.0 2022: Analysis of sentiment, emotion and intensity of emotion classes from meme images using single and multi modalities (short paper). In De-Factify @ AAAI 2022. First Workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Asaf Nissenbaum and Limor Shifman. 2017. Internet memes as contested cultural capital: The case of 4chan’s /b/ board. New Media & Society, 19(4):483– 501. Parth Patwa, Sathyanarayanan Ramamoorthy, Nethra Gunti, Shreyash Mishra, S Suryavardan, Aishwarya Reganti, et al. 2022. Findings of memotion 2: Sentiment and emotion analysis of memes. In De-Factify @ AAAI 2022. First Workshop on Multimodal FactChecking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Kim Ngan Phan, Gueesang Lee, Hyung-Jeong Yang, and Soo-Hyung Kim. 2022. Little flower at memotion 2.0 2022 : Ensemble of multi-modal model using attention mechanism in memotion analysis (short paper). In De-Factify @ AAAI 2022. First Workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Jason Phang, Thibault Fevry, and Samuel R. Bowman. ´ 2019. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint, arXiv:1811.01088. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics. Clifton Poth, Jonas Pfeiffer, Andreas Ruckl ¨ e, and Iryna ´ Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585–10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2021. DynaSent: A dynamic benchmark for sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2388–2404, Online. Association for Computational Linguistics. Shraman Pramanick, Md Shad Akhtar, and Tanmoy Chakraborty. 2021a. Exercise? i thought you said ’extra fries’: Leveraging sentence demarcations and multi-hop attention for meme affect analysis. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):513–524. Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021b. MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4439–4455, Punta Cana, Dominican Republic. Association for Computational Linguistics. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online. Association for Computational Linguistics. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR. Sathyanarayanan Ramamoorthy, Nethra Gunti, Shreyash Mishra1, S Suryavardan, Aishwarya Reganti, Parth Patwa, Amitava Das, Tanmoy Chakraborty, Amit Sheth, Asif Ekbal, and Chaitanya Ahuja. 2022. Memotion 2: Dataset on sentiment and emotion analysis of memes. In De-Factify @ AAAI 2022. First Workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI. Elad Segev, Asaf Nissenbaum, Nathan Stolero, and Limor Shifman. 2015. Families and Networks of Internet Memes: The Relationship Between Cohesiveness, Uniqueness, and Quiddity Concreteness. Journal of Computer-Mediated Communication, 20(4):417–433. Lanyu Shang, Yang Zhang, Yuheng Zha, Yingxi Chen, Christina Youn, and Dong Wang. 2021. AOMD: An Analogy-Aware Approach to Offensive Meme Detection on Social Media. Inf. Process. Manage., 58(5). Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, et al. 2020. SemEval-2020 task 8: Memotion analysisthe visuo-lingual metaphor! In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 759–773, Barcelona (online). International Committee for Computational Linguistics. Shivam Sharma, Mohd Khizir Siddiqui, Md. Shad Akhtar, and Tanmoy Chakraborty. 2022. Domainaware self-supervised pre-training for label-efficient meme analysis. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 792–805, Online only. Association for Computational Linguistics. Limor Shifman. 2014. The cultural logic of photo-based meme genres. Journal of Visual Culture, 13(3):340–358. Pranaydeep Singh, Aaron Maladry, and Els Lefever. 2022. Combining language models and linguistic information to label entities in memes. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pages 35–42, Dublin, Ireland. Association for Computational Linguistics. Shardul Suryawanshi, Bharathi Raja Chakravarthi, Pranav Verma, Mihael Arcan, John Philip McCrae, and Paul Buitelaar. 2020. A dataset for troll classification of TamilMemes. In Proceedings of the WILDRE5– 5th Workshop on Indian Language Data: Resources and Evaluation, pages 7–13, Marseille, France. European Language Resources Association (ELRA). Cagri Toraman, Furkan S¸ahinuc¸, and Eyup Yilmaz. 2022. Large-scale hate speech detection with crossdomain transfer. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2215–2225, Marseille, France. European Language Resources Association. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465–4476, Florence, Italy. Association for Computational Linguistics. Ron Zhu. 2020. Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution. arXiv preprint, arXiv:2012.08290. Yan Zhuang and Yanru Zhang. 2022. Yet at memotion 2.0 2022 : Hate speech detection combining bilstm and fully connected layers (short paper). In De-Factify @ AAAI 2022. First Workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR Workshop Proceedings. AAAI.

KIEFA

kiefa.io

1-10 employees

Since 2022

COMPANY RANKING

KIEFA

EVERGREEN INDEX #5971

Representing this company? Click here to claim and customize this page!

Kiefa WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

KIEFA

kiefa.io

1-10 employees

Since 2022

COMPANY RANKING

KIEFA

EVERGREEN INDEX #5971

RELATED COMPANIESRANK

STORIES ABOUT KIEFA

Representing this company? Click here to claim and customize this page!

Kiefa WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!