Additionally, HiddenLayer recently announced that CRN®, a brand of The Channel Company, named HiddenLayer to its 2023 Stellar Startups list. This annual list, previously known as CRN Emerging Vendors, recognizes fast-rising technology manufacturers committed to delivering leading-edge solutions ...

HiddenLayer Launches Channel Partner Program to Secure AI and MLOps ...

WASHINGTON — Booz Allen Hamilton, one of the largest U.S. defense contractors, said it invested in HiddenLayer, a smaller company specializing in digital defenses of artificial intelligence and sensitive networks.

Booz Allen Hamilton invests in AI security startup HiddenLayer

HiddenLayer, the leading security provider for artificial intelligence (AI) models and assets, has raised $50 million in Series A funding to expand its talent base, increase go-to-market efforts, and further invest in its award-winning Machine Learning Security (MLSec) Platform. The investment marks the largest Series A funding raised by a cybersecurity company ...

HiddenLayer Raises $50M in Series A Funding to Safeguard AI

Bringing the company’s total raised to $56 million, the new funds will be put toward supporting HiddenLayer’s go-to-market efforts, expanding its headcount from 50 employees to 90 by the end of the year and further investing in R&amp;D, co-founder and CEO Chris Sestito told TechCrunch via email.

HiddenLayer raises $50M for its AI-defending cybersecurity tools ...

HiddenLayer, which emerged from stealth in July 2022 with $6 million in funding, said the latest financing was led by M12, Microsoft’s Venture Fund, and Moore Strategic Ventures. The Austin company also took on equity investments from Booz Allen Ventures, IBM Ventures, Capital One Ventures, ...

HiddenLayer Raises Hefty $50M Round for AI Security Tech - ...

HiddenLayer | 5,128 followers on LinkedIn. #Security4AI | HiddenLayer, a Gartner recognized AI Application Security company, is a provider of security solutions for artificial intelligence algorithms, models &amp; the data that power them. With a first-of-its-kind, non-invasive software approach ...

HiddenLayer | LinkedIn

Chris Sestito, Co-founder and CEO, HiddenLayer, pitches at the RSAC 2023 Innovation Sandbox Contest.For 17 years, cybersecurity&#x27;s boldest new innovators have...

HiddenLayer — RSA Conference 2023 Innovation Sandbox - YouTube

/PRNewswire/ -- HiddenLayer, a Gartner recognized AI Application Security company, is proud to announce a strategic collaboration with Intel Corporation to...

HiddenLayer Announces Machine Learning Security Collaboration with ...

Your AI algorithms represent a unique competitive advantage for your company and come at a considerable expense. A successful adversarial attack against them could cost you that advantage and you would never know it happened. HiddenLayer is the first productized solution for the next security ...

Prevent attacks against your AI with HiddenLayer

HiddenLayer helps enterprises safeguard the machine learning models behind their most important products with a comprehensive security platform. Only HiddenLayer offers turnkey AI/ML security that does not add unnecessary complexity to models and does not require access to raw data and algorithms.

HiddenLayer - Ten Eleven

HiddenLayer

hiddenlayer

Authors:
(1) Hanqing ZHAO, College of Traditional Chinese Medicine, Hebei University, Funded by National Natural Science Foundation of China (No.82004503) and Science and Technology Project of Hebei Education Department(BJK2024108) and a Corresponding Author (zhaohq@hbu.edu.cn);
(2) Yuehan LI, College of Traditional Chinese Medicine, Hebei University. Table of Links Abstract and 1. Introduction 2. Materials and Methods 2.1 Experimental Data and 2.2 Conditional random fields mode 2.3 TF-IDF algorithm and 2.4 Dependency Parser Based on Neural Network 2.5 Experimental Environment 3 Experimental results 3.1 Results of word segmentation and entity recognition 3.2 Visualization results of related entity vocabulary map 3.3 Results of dependency parsing 4 Final Remarks 5 References 2.1 Experimental Data The data of this study are the publicly available versions of the Origin of Medicine, Spleen and Stomach Theory and Yin Syndrome Lue Case. The full text content is transformed into txt documents, the table of contents is removed, only the full text title and all the main text are retained, and Spaces and blank lines are removed, and no data cleaning is performed. 2.2 Conditional random fields model Conditional Random Field (CRF) is a basic model of natural language processing, which is widely used in Chinese word segmentation, named entity recognition, part-of-speech tagging and other tagging scenarios. Chinese word segmentation uses BMES word position method, that is, word head, word middle, word end and independent word. The input sentence S is equivalent to the sequence X, and the output label sequence L is equivalent to the sequence Y. We want to train a model to find the optimal corresponding L under the premise of a given S. The keypoint of training this model is the selection of the feature function F and the determination of the weight of each feature function W. For each feature function, its input has the following four elements: ① Sentence S②i, which is used to represent the ith word in sentence S ③li, which is the part-of-speech (POS) tagged by the scoring sequence to the ith word ④li−1, which is the part-of-speech tagged by the scoring sequence to the ith word. Its output value is either 0 or 1, where 0 means that the sequence to be scored does not conform to this feature, and 1 means that the sequence to be scored does. For the sequences L and S, we can construct the conditional probability distribution model formula: IOB labeling method is used for named entity recognition on the basis of word segmentation, as shown in Figure 1, using the gram generated by the word segmentation sequence of each sentence, the tri-gram model is used to extract features, and finally input into the CRF model to complete the labeling. 2.3 TF-IDF algorithm TF-IDF (Term frequency-inverse Document Frequency) is a common weighting technique used in information retrieval and data mining, which is often used to mine keywords in articles. TF-IDF is a statistical analysis method used to evaluate the importance of a term to a document set or a corpus. Term Frequency (TF) is the number or frequency of occurrences of a term in a document. If a term appears more than once in a document, it is likely to be an important term. The formula is as follows: Term Frequency (TF) = The number of times a term appears in a document/the total number of words in the document Inverse Document Frequency (IDF) =log(total number of documents in the corpus /(number of documents containing the term +1)) TF−IDF= Term frequency (TF) × Inverse Document frequency (IDF) The importance of a term is directly proportional to the number of times it appears in the document and inversely proportional to the number of times it appears in the corpus. This calculation method can effectively avoid the influence of common words on keywords, and improve the correlation between keywords and articles. 2.4Dependency Parser Based on Neural Network Dependency parsing can help us understand the meaning of text. Grammar parsing is an important part of language understanding. The goal is to analyze the grammatical structure of a sentence and represent it into an understandable structure, usually a tree structure. The dependency syntax theory believes that there is a master-slave relationship between words. If a word modifies another word in a sentence, the modifier is called a dependent word, and the modified word is called a dominant word. Dependency parsing based on neural network converts the sequence of words in a sentence into a graph structure by analyzing the grammatical relationship inside the sentence. Common grammatical relations include verb-object relation, left-adjunction relation, right-adjunction relation, coordinate relation, determination-central relation, subject-verb relation, etc. Dependency grammar is a commonly used grammar system. Dependency arc connects two sentences in a sentence that have a certain grammatical relationship, forming a syntactic dependency tree. The dependency tree is constructed by the stack, starting from the root node, and then all the words stored in the cache are pushed into the stack one by one by using three states: Shift, Left-Reduce and Right-Reduce. Neural network consists of three layers, Input layer (Softmax layer), Hidden layer (Hidden layer) and output layer (Input layer). The model is fromthe 2014 paper "A Fast and Accurate Dependency Parser using Neural Networks" by Danqi Chen and Christopher D. anning. In this study, HanLP is used to implement a dependency parser based on neural networks. 2.5 Experimental Environment The research was implemented on a small artificial intelligence platform equipped with Intel Xeon Gold 6248R CPU@3.00Ghz*96, 256GB memory and NVIDIA A100 80G*2 GPU computing card in the Laboratory of Traditional Chinese Medicine Informatics of Hebei University. Ubuntu 18.04.6LTS, Python 3.9 environment. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Hanqing ZHAO, College of Traditional Chinese Medicine, Hebei University, Funded by National Natural Science Foundation of China (No.82004503) and Science and Technology Project of Hebei Education Department(BJK2024108) and a Corresponding Author (zhaohq@hbu.edu.cn); (2) Yuehan LI, College of Traditional Chinese Medicine, Hebei University. Authors: Authors: (1) Hanqing ZHAO, College of Traditional Chinese Medicine, Hebei University, Funded by National Natural Science Foundation of China (No.82004503) and Science and Technology Project of Hebei Education Department(BJK2024108) and a Corresponding Author (zhaohq@hbu.edu.cn); (2) Yuehan LI, College of Traditional Chinese Medicine, Hebei University. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Materials and Methods 2.1 Experimental Data and 2.2 Conditional random fields mode 2.1 Experimental Data and 2.2 Conditional random fields mode 2.3 TF-IDF algorithm and 2.4 Dependency Parser Based on Neural Network 2.3 TF-IDF algorithm and 2.4 Dependency Parser Based on Neural Network 2.5 Experimental Environment 2.5 Experimental Environment 3 Experimental results 3.1 Results of word segmentation and entity recognition 3.1 Results of word segmentation and entity recognition 3.2 Visualization results of related entity vocabulary map 3.2 Visualization results of related entity vocabulary map 3.3 Results of dependency parsing 3.3 Results of dependency parsing 4 Final Remarks 4 Final Remarks 5 References 5 References 2.1 Experimental Data The data of this study are the publicly available versions of the Origin of Medicine, Spleen and Stomach Theory and Yin Syndrome Lue Case. The full text content is transformed into txt documents, the table of contents is removed, only the full text title and all the main text are retained, and Spaces and blank lines are removed, and no data cleaning is performed. 2.2 Conditional random fields model Conditional Random Field (CRF) is a basic model of natural language processing, which is widely used in Chinese word segmentation, named entity recognition, part-of-speech tagging and other tagging scenarios. Chinese word segmentation uses BMES word position method, that is, word head, word middle, word end and independent word. The input sentence S is equivalent to the sequence X, and the output label sequence L is equivalent to the sequence Y. We want to train a model to find the optimal corresponding L under the premise of a given S. The keypoint of training this model is the selection of the feature function F and the determination of the weight of each feature function W. For each feature function, its input has the following four elements: ① Sentence S②i, which is used to represent the ith word in sentence S ③li, which is the part-of-speech (POS) tagged by the scoring sequence to the ith word ④li−1, which is the part-of-speech tagged by the scoring sequence to the ith word. Its output value is either 0 or 1, where 0 means that the sequence to be scored does not conform to this feature, and 1 means that the sequence to be scored does. For the sequences L and S, we can construct the conditional probability distribution model formula: IOB labeling method is used for named entity recognition on the basis of word segmentation, as shown in Figure 1, using the gram generated by the word segmentation sequence of each sentence, the tri-gram model is used to extract features, and finally input into the CRF model to complete the labeling. 2.3 TF-IDF algorithm TF-IDF (Term frequency-inverse Document Frequency) is a common weighting technique used in information retrieval and data mining, which is often used to mine keywords in articles. TF-IDF is a statistical analysis method used to evaluate the importance of a term to a document set or a corpus. Term Frequency (TF) is the number or frequency of occurrences of a term in a document. If a term appears more than once in a document, it is likely to be an important term. The formula is as follows: Term Frequency (TF) = The number of times a term appears in a document/the total number of words in the document Inverse Document Frequency (IDF) =log(total number of documents in the corpus /(number of documents containing the term +1)) TF−IDF= Term frequency (TF) × Inverse Document frequency (IDF) The importance of a term is directly proportional to the number of times it appears in the document and inversely proportional to the number of times it appears in the corpus. This calculation method can effectively avoid the influence of common words on keywords, and improve the correlation between keywords and articles. 2.4Dependency Parser Based on Neural Network Dependency parsing can help us understand the meaning of text. Grammar parsing is an important part of language understanding. The goal is to analyze the grammatical structure of a sentence and represent it into an understandable structure, usually a tree structure. The dependency syntax theory believes that there is a master-slave relationship between words. If a word modifies another word in a sentence, the modifier is called a dependent word, and the modified word is called a dominant word. Dependency parsing based on neural network converts the sequence of words in a sentence into a graph structure by analyzing the grammatical relationship inside the sentence. Common grammatical relations include verb-object relation, left-adjunction relation, right-adjunction relation, coordinate relation, determination-central relation, subject-verb relation, etc. Dependency grammar is a commonly used grammar system. Dependency arc connects two sentences in a sentence that have a certain grammatical relationship, forming a syntactic dependency tree. The dependency tree is constructed by the stack, starting from the root node, and then all the words stored in the cache are pushed into the stack one by one by using three states: Shift, Left-Reduce and Right-Reduce. Neural network consists of three layers, Input layer (Softmax layer), Hidden layer (Hidden layer) and output layer (Input layer). The model is fromthe 2014 paper "A Fast and Accurate Dependency Parser using Neural Networks" by Danqi Chen and Christopher D. anning. In this study, HanLP is used to implement a dependency parser based on neural networks. 2.5 Experimental Environment The research was implemented on a small artificial intelligence platform equipped with Intel Xeon Gold 6248R CPU@3.00Ghz*96, 256GB memory and NVIDIA A100 80G*2 GPU computing card in the Laboratory of Traditional Chinese Medicine Informatics of Hebei University. Ubuntu 18.04.6LTS, Python 3.9 environment. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Automated Entity and Relationship Extraction in Classical TCM Literature

Embeddings 101: Unlocking Semantic Relationships in Text

অন্তর্ভুক্ত 101: টেক্সট মধ্যে সিমেন্টিক সম্পর্ক খুলুন

Embeddings 101: Semantische Beziehungen im Text freischalten

Embeddings 101 : Déverrouiller les relations sémantiques dans le texte

Embeddings 101: पाठ में सेमेंटिक रिश्तों को अनलॉक करना

Embeddings 101: Unlocking Semantic Relationships in Text(テキスト内のセマンティックな関係を解除する)

삽입 101 : 텍스트에서 세마닉 관계를 해제

Embeddings 101: Desbloqueio de relações semânticas em texto

Встраивание 101: Разблокирование семантических отношений в тексте

Embeddings 101: Yazıda Semantik İlişkilerin Kilitlenmesi

Embeddings 101: Mở khóa các mối quan hệ ngữ nghĩa trong văn bản

嵌入式101:解锁文本中的语义关系

Authors:
(1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu);
(2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3.1. Compositionality 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 2.1. Historical foundations The origins of large language models can be traced back to the inception of AI research. The early history of natural language processing (NLP) was marked by a schism between two competing paradigms: the symbolic and the stochastic approaches. A major influence on the symbolic paradigm in NLP was Noam Chomsky’s transformational-generative grammar (Chomsky 1957), which posited that the syntax of natural languages could be captured by a set of formal rules that generated well-formed sentences. Chomsky’s work laid the foundation for the development of rule-based syntactic parsers, which leverage linguistic theory to decompose sentences into their constituent parts. Early conversational NLP systems, such as Winograd’s SHRDLU (Winograd 1971), required syntactic parsers with a complex set of ad hoc rules to process user input. In parallel, the stochastic paradigm was pioneered by researchers such as mathematician Warren Weaver, who was influenced by Claude Shannon’s information theory. In a memorandum written in 1949, Weaver proposed the use of computers for machine translation employing statistical techniques (Weaver 1955). This work paved the way for the development of statistical language models, such as n-gram models, which estimate the likelihood of word sequences based on observed frequencies of word combinations in a corpus (Jelinek 1998). Initially, however, the stochastic paradigm was lagging behind symbolic approaches to NLP, showing only modest success in toy models with limited applications. Another important theoretical stepping stone on the road to modern language models is the so-called distributional hypothesis, first proposed by the linguist Zellig Harris in the 1950s (Harris 1954). This idea was grounded in the structuralist view of language, which posits that linguistic units acquire meaning through their patterns of co-occurrence with other units in the system. Harris specifically suggested that the meaning of a word could be inferred by examining its distributional properties, or the contexts in which it occurs. Firth (1957) aptly summarized this hypothesis with the slogan “You shall know a word by the company it keeps,” acknowledging the influence of Wittgenstein (1953)’s conception of meaning-as-use to highlight the importance of context in understanding linguistic meaning. As research on the distributional hypothesis progressed, scholars began exploring the possibility of representing word meanings as vectors in a multidimensional space 1. Early empirical work in this area stemmed from psychology and examined the meaning of words along various dimensions, such as valence and potency (Osgood 1952). While this work introduced the idea of representing meaning in a multidimensional vector space, it relied on explicit participant ratings about word connotations along different scales (e.g., good–bad), rather than analyzing the distributional properties of a linguistic corpus. Subsequent research in information retrieval combined vector-based representations with a data-driven approach, developing automated techniques for representing documents and words as vectors in high-dimensional vector spaces (Salton et al. 1975). After decades of experimental research, these ideas eventually reached maturity with the development of word embedding models using artificial neural networks (Bengio et al. 2000). These models are based on the insight that the distributional properties of words can be learned by training a neural network to predict a word’s context given the word itself, or vice versa. Unlike previous statistical methods such as n-gram models, word embedding models encode words into dense, low-dimensional vector representations (Fig. 1). The resulting vector space drastically reduces the dimensionality of linguistic data while preserving information about meaningful linguistic relationships beyond simple co-occurrence statistics. Notably, many semantic and syntactic relationships between words are reflected in linear substructures within the vector space of word embedding models. For example, Word2Vec (Mikolov et al. 2013) demonstrated that word embeddings can capture both semantic and syntactic regularities, as evidenced by the ability to solve word analogy tasks through simple vector arithmetic that reveal the latent linguistic structure encoded in the vector space (e.g., 𝑘𝑖𝑛𝑔 + 𝑤𝑜𝑚𝑎𝑛 − 𝑚𝑎𝑛 ≈ 𝑞𝑢𝑒𝑒𝑛, or 𝑤𝑎𝑙𝑘𝑖𝑛𝑔 + 𝑠𝑤𝑎𝑚 − 𝑤𝑎𝑙𝑘𝑒𝑑 ≈ 𝑠𝑤𝑖𝑚𝑚𝑖𝑛𝑔). The development of word embedding models marked a turning point in the history of NLP, providing a powerful and efficient means of representing linguistic units in a continuous vector space based on their statistical distribution in a large corpus. However, these models have several significant limitations. First, they are not capable of capturing polysemy and homonymy, because they assign a single or “static” embedding to each word type, which cannot account for changes in meaning based on context; for example, “bank” is assigned a unique embedding regardless of whether it refers to the side of a river or the financial institution. Second, they rely on “shallow” artificial neural network architectures with a single hidden layer, which limits their ability to model complex relationships between words. Finally, being designed to represent language at the level of individual words, they are not well-suited to model complex linguistic expression, such as phrases, sentences, and paragraphs. While it is possible to represent a sentence as a vector by averaging the embeddings of every word in the sentence, this is a very poor way of representing sentence-level meaning, as it loses information about compositional structure reflected in word order. In other words, word embedding models merely treat language as a “bag of words”; for example, “a law book” and “a book law” are treated identically as the unordered set {'a','book','law'}. The shortcomings of shallow word embedding models were addressed with the introduction of “deep” language models, going back to recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and the gated recurrent unit (GRU) (Cho et al. 2014). These deep neural network architectures incorporate a memory-like mechanism, allowing them to remember and process sequences of inputs over time, rather than individual, isolated words. Despite this advantage over word embedding models, they suffer from their own limitations: they are slow to train and struggle with long sequences of text. These issues were addressed with the introduction of the Transformer architecture by Vaswani et al. (2017), which laid the groundwork for modern LLMs. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Authors: Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.1. Historical foundations 2.2. Transformer-based LLMs 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3. Interface with classic philosophical issues 3.1. Compositionality 3.1. Compositionality 3.2. Nativism and language acquisition 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.3. Language understanding and grounding 3.4. World models 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 4. Conclusion, Glossary, and References 2.1. Historical foundations The origins of large language models can be traced back to the inception of AI research. The early history of natural language processing (NLP) was marked by a schism between two competing paradigms: the symbolic and the stochastic approaches. A major influence on the symbolic paradigm in NLP was Noam Chomsky’s transformational-generative grammar (Chomsky 1957), which posited that the syntax of natural languages could be captured by a set of formal rules that generated well-formed sentences. Chomsky’s work laid the foundation for the development of rule-based syntactic parsers, which leverage linguistic theory to decompose sentences into their constituent parts. Early conversational NLP systems, such as Winograd’s SHRDLU (Winograd 1971), required syntactic parsers with a complex set of ad hoc rules to process user input. In parallel, the stochastic paradigm was pioneered by researchers such as mathematician Warren Weaver, who was influenced by Claude Shannon’s information theory. In a memorandum written in 1949, Weaver proposed the use of computers for machine translation employing statistical techniques (Weaver 1955). This work paved the way for the development of statistical language models, such as n-gram models, which estimate the likelihood of word sequences based on observed frequencies of word combinations in a corpus (Jelinek 1998). Initially, however, the stochastic paradigm was lagging behind symbolic approaches to NLP, showing only modest success in toy models with limited applications. Another important theoretical stepping stone on the road to modern language models is the so-called distributional hypothesis, first proposed by the linguist Zellig Harris in the 1950s (Harris 1954). This idea was grounded in the structuralist view of language, which posits that linguistic units acquire meaning through their patterns of co-occurrence with other units in the system. Harris specifically suggested that the meaning of a word could be inferred by examining its distributional properties, or the contexts in which it occurs. Firth (1957) aptly summarized this hypothesis with the slogan “You shall know a word by the company it keeps,” acknowledging the influence of Wittgenstein (1953)’s conception of meaning-as-use to highlight the importance of context in understanding linguistic meaning. As research on the distributional hypothesis progressed, scholars began exploring the possibility of representing word meanings as vectors in a multidimensional space 1. Early empirical work in this area stemmed from psychology and examined the meaning of words along various dimensions, such as valence and potency (Osgood 1952). While this work introduced the idea of representing meaning in a multidimensional vector space, it relied on explicit participant ratings about word connotations along different scales (e.g., good–bad), rather than analyzing the distributional properties of a linguistic corpus. Subsequent research in information retrieval combined vector-based representations with a data-driven approach, developing automated techniques for representing documents and words as vectors in high-dimensional vector spaces (Salton et al. 1975). After decades of experimental research, these ideas eventually reached maturity with the development of word embedding models using artificial neural networks (Bengio et al. 2000). These models are based on the insight that the distributional properties of words can be learned by training a neural network to predict a word’s context given the word itself, or vice versa. Unlike previous statistical methods such as n-gram models, word embedding models encode words into dense, low-dimensional vector representations (Fig. 1). The resulting vector space drastically reduces the dimensionality of linguistic data while preserving information about meaningful linguistic relationships beyond simple co-occurrence statistics. Notably, many semantic and syntactic relationships between words are reflected in linear substructures within the vector space of word embedding models. For example, Word2Vec (Mikolov et al. 2013) demonstrated that word embeddings can capture both semantic and syntactic regularities, as evidenced by the ability to solve word analogy tasks through simple vector arithmetic that reveal the latent linguistic structure encoded in the vector space (e.g., 𝑘𝑖𝑛𝑔 + 𝑤𝑜𝑚𝑎𝑛 − 𝑚𝑎𝑛 ≈ 𝑞𝑢𝑒𝑒𝑛, or 𝑤𝑎𝑙𝑘𝑖𝑛𝑔 + 𝑠𝑤𝑎𝑚 − 𝑤𝑎𝑙𝑘𝑒𝑑 ≈ 𝑠𝑤𝑖𝑚𝑚𝑖𝑛𝑔). The development of word embedding models marked a turning point in the history of NLP, providing a powerful and efficient means of representing linguistic units in a continuous vector space based on their statistical distribution in a large corpus. However, these models have several significant limitations. First, they are not capable of capturing polysemy and homonymy, because they assign a single or “static” embedding to each word type, which cannot account for changes in meaning based on context; for example, “bank” is assigned a unique embedding regardless of whether it refers to the side of a river or the financial institution. Second, they rely on “shallow” artificial neural network architectures with a single hidden layer, which limits their ability to model complex relationships between words. Finally, being designed to represent language at the level of individual words, they are not well-suited to model complex linguistic expression, such as phrases, sentences, and paragraphs. While it is possible to represent a sentence as a vector by averaging the embeddings of every word in the sentence, this is a very poor way of representing sentence-level meaning, as it loses information about compositional structure reflected in word order. In other words, word embedding models merely treat language as a “bag of words”; for example, “a law book” and “a book law” are treated identically as the unordered set {'a','book','law'}. The shortcomings of shallow word embedding models were addressed with the introduction of “deep” language models, going back to recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and the gated recurrent unit (GRU) (Cho et al. 2014). These deep neural network architectures incorporate a memory-like mechanism, allowing them to remember and process sequences of inputs over time, rather than individual, isolated words. Despite this advantage over word embedding models, they suffer from their own limitations: they are slow to train and struggle with long sequences of text. These issues were addressed with the introduction of the Transformer architecture by Vaswani et al. (2017), which laid the groundwork for modern LLMs. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

The Shift from Symbolic AI to Deep Learning in Natural Language Processing 

AI Simbólico ukhat Manqha Yatiqañ ukar mayjt’ayawix Procesamiento de Lenguas Naturales ukan 

El paso de la IA simbólica al aprendizaje profundo en el procesamiento del lenguaje natural

تغییر از هوش مصنوعی نمادین به یادگیری عمیق در پردازش زبان طبیعی 

Prijelaz sa simboličke umjetne inteligencije na duboko učenje u obradi prirodnog jezika 

Eltérés a szimbolikus mesterséges intelligencia felől a mélytanulás felé a természetes nyelvi feldolgozásban 

自然言語処理におけるシンボリック AI からディープラーニングへの移行

자연어 처리에서 심볼릭 AI에서 딥러닝으로의 전환

Премин од симболична вештачка интелигенција кон длабоко учење во обработка на природни јазици 

Peralihan daripada AI Simbolik kepada Pembelajaran Mendalam dalam Pemprosesan Bahasa Semulajadi 

Гузариш аз AI рамзӣ ба омӯзиши амиқ дар коркарди забони табиӣ 

Simwoliki AI-den tebigy dilleri gaýtadan işlemekde çuňňur öwrenişe geçiş 

قدرتی زبان کی پروسیسنگ میں علامتی AI سے گہری سیکھنے کی طرف شفٹ 

Authors:
(1) Soham De, Google DeepMind and with Equal contributions;
(2) Samuel L. Smith, Google DeepMind and with Equal contributions;
(3) Anushan Fernando, Google DeepMind and with Equal contributions;
(4) Aleksandar Botev, Google DeepMind and with Equal contributions;
(5) George Cristian-Muraru, Google DeepMind and with Equal contributions;
(6) Albert Gu, Work done while at Google DeepMind;
(7) Ruba Haroun, Google DeepMind;
(8) Leonard Berrada, Google DeepMind;
(9) Yutian Chen, Google DeepMind;
(10) Srivatsan Srinivasan, Google DeepMind;
(11) Guillaume Desjardins, Google DeepMind;
(12) Arnaud Doucet, Google DeepMind;
(13) David Budden, Google DeepMind;
(14) Yee Whye Teh, Google DeepMind;
(15) David Budden, Google DeepMind;
(16) Razvan Pascanu, Google DeepMind;
(17) Nando De Freitas, Google DeepMind;
(18) Caglar Gulcehre, Google DeepMind. Table of Links 1 Introduction 2 Model Architecture 3 Recurrent Models Scale as Efficiently as Transformers 3.1. Scaling curves 3.2. Evaluation on downstream tasks 4 Training Recurrent Models Efficiently on Device and 4.1. Model parallelism for large scale training 4.2. Efficient linear recurrences on device 4.3. Training speed on longer sequences 5. Inference Speed 5.1. A simple model of the decode step 5.2. Results 6. Long Context Modeling and 6.1. Improving next token prediction with longer contexts 6.2. Copy and retrieval capabilities 7. Related Works 8. Conclusion, Acknowledgements, and References A. RG-LRU Recurrence Gate B. Complex-Gated Linear Recurrent Unit (CG-LRU) C. Model Scale Hyper-Parameters D. Efficient Linear Recurrences on Device E. The Local Attention Window Size of Griffin F. Inference Speeds G. Improving Next Token Prediction with Longer Contexts: Additional Results H. Additional Details of the Copy and Retrieval Tasks 2. Model Architecture All our models contain the following components: (i) a residual block, (ii) an MLP block, and (iii) a temporal-mixing block. While (i) and (ii) are the same across all models, we consider three temporal mixing blocks: global Multi-Query Attention (MQA), local (sliding-window) MQA and our proposed recurrent block. As part of the recurrent block we use the Real-Gated Linear Recurrent Unit (RG-LRU) – a novel recurrent layer inspired by the Linear Recurrent Unit (Orvieto et al., 2023b). The residual block, as shown in Figure 2(a), defines the global structure of our models and is inspired by pre-norm Transformers (Xiong et al., 2020). After embedding the input sequence we pass it through 𝑁 such blocks (𝑁 denoting the model depth), and then we apply RMSNorm (Zhang and Sennrich, 2019) to produce the final activations. To compute the token probabilities we apply a final linear layer followed by a softmax. The weights of this layer are shared with the input embedding layer. 2.1. Residual block The residual block contains two components, applied in order. The first component takes the hidden state 𝑥 and applies an RMSNorm (Zhang and Sennrich, 2019), followed by the temporal-mixing block. We then merge the output with a skip connection from 𝑥 through addition. Similarly, the second component applies RMSNorm, followed by the MLP block and then merges its output with a skip connection from the input of the RMSNorm. This block is illustrated in Figure 2 (a). 2.2. MLP block We use a gated MLP block (Dauphin et al., 2017) (illustrated in Figure 2(b)), which creates two branches from its input of dimension 𝐷. We apply a linear layer with output dimension 𝑀𝐷 on each branch, where 𝑀 denotes the expansion factor. For simplicity, we use 𝑀 =3 throughout this work. We apply a GeLU non-linearity (Hendrycks and Gimpel, 2016) on one of the branches before merging them by element-wise multiplication, similar to GeGeLU (Shazeer, 2020). However, in our MLP block, we apply a final linear layer with output dimension 𝐷 on the outputs of the GeGeLU layer. 2.3. Temporal-mixing blocks The temporal-mixing block is the component of our model that aggregates hidden layer activations at different temporal locations in the sequence. We consider three temporal-mixing blocks: global MQA (Shazeer, 2019), local MQA (Beltagy et al., 2020) and our proposed Recurrent block. Local sliding window attention One of the key disadvantages of using global attention is that its computational complexity grows quadratically in the sequence length. To address this, several works have started to adopt local attention (Beltagy et al., 2020), also known as sliding window attention. It allows each position to attend only to a fixed number of tokens in the past. This not only reduces the computational FLOPs but also bounds the size of the KV cache to the size of window, making it no longer quadratic in the sequence length. All other details are the same as the global MQA. 2.4. Real-Gated Linear Recurrent Unit (RG-LRU) Our proposed RG-LRU layer has a simple recurrence inspired by the Linear Recurrent Unit (LRU) (Orvieto et al., 2023b), but incorporates a gating mechanism motivated by the literature on non-linear RNNs, in particular LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Chung et al., 2014). The equations describing the layer are as follows: This paper is available on arxiv under CC BY 4.0 DEED license. [1] We suggest ablating the use of complex numbers for other modalities and provide more information about the complex-valued version of the RG-LRU layer in Appendix B. Authors: (1) Soham De, Google DeepMind and with Equal contributions; (2) Samuel L. Smith, Google DeepMind and with Equal contributions; (3) Anushan Fernando, Google DeepMind and with Equal contributions; (4) Aleksandar Botev, Google DeepMind and with Equal contributions; (5) George Cristian-Muraru, Google DeepMind and with Equal contributions; (6) Albert Gu, Work done while at Google DeepMind; (7) Ruba Haroun, Google DeepMind; (8) Leonard Berrada, Google DeepMind; (9) Yutian Chen, Google DeepMind; (10) Srivatsan Srinivasan, Google DeepMind; (11) Guillaume Desjardins, Google DeepMind; (12) Arnaud Doucet, Google DeepMind; (13) David Budden, Google DeepMind; (14) Yee Whye Teh, Google DeepMind; (15) David Budden, Google DeepMind; (16) Razvan Pascanu, Google DeepMind; (17) Nando De Freitas, Google DeepMind; (18) Caglar Gulcehre, Google DeepMind. Authors: Authors: (1) Soham De, Google DeepMind and with Equal contributions; (2) Samuel L. Smith, Google DeepMind and with Equal contributions; (3) Anushan Fernando, Google DeepMind and with Equal contributions; (4) Aleksandar Botev, Google DeepMind and with Equal contributions; (5) George Cristian-Muraru, Google DeepMind and with Equal contributions; (6) Albert Gu, Work done while at Google DeepMind; (7) Ruba Haroun, Google DeepMind; (8) Leonard Berrada, Google DeepMind; (9) Yutian Chen, Google DeepMind; (10) Srivatsan Srinivasan, Google DeepMind; (11) Guillaume Desjardins, Google DeepMind; (12) Arnaud Doucet, Google DeepMind; (13) David Budden, Google DeepMind; (14) Yee Whye Teh, Google DeepMind; (15) David Budden, Google DeepMind; (16) Razvan Pascanu, Google DeepMind; (17) Nando De Freitas, Google DeepMind; (18) Caglar Gulcehre, Google DeepMind. Table of Links 1 Introduction 1 Introduction 2 Model Architecture 2 Model Architecture 3 Recurrent Models Scale as Efficiently as Transformers 3 Recurrent Models Scale as Efficiently as Transformers 3.1. Scaling curves 3.1. Scaling curves 3.2. Evaluation on downstream tasks 3.2. Evaluation on downstream tasks 4 Training Recurrent Models Efficiently on Device and 4.1. Model parallelism for large scale training 4 Training Recurrent Models Efficiently on Device and 4.1. Model parallelism for large scale training 4.2. Efficient linear recurrences on device 4.2. Efficient linear recurrences on device 4.3. Training speed on longer sequences 4.3. Training speed on longer sequences 5. Inference Speed 5. Inference Speed 5.1. A simple model of the decode step 5.1. A simple model of the decode step 5.2. Results 5.2. Results 6. Long Context Modeling and 6.1. Improving next token prediction with longer contexts 6. Long Context Modeling and 6.1. Improving next token prediction with longer contexts 6.2. Copy and retrieval capabilities 6.2. Copy and retrieval capabilities 7. Related Works 7. Related Works 8. Conclusion, Acknowledgements, and References 8. Conclusion, Acknowledgements, and References A. RG-LRU Recurrence Gate A. RG-LRU Recurrence Gate B. Complex-Gated Linear Recurrent Unit (CG-LRU) B. Complex-Gated Linear Recurrent Unit (CG-LRU) C. Model Scale Hyper-Parameters C. Model Scale Hyper-Parameters D. Efficient Linear Recurrences on Device D. Efficient Linear Recurrences on Device E. The Local Attention Window Size of Griffin E. The Local Attention Window Size of Griffin F. Inference Speeds F. Inference Speeds G. Improving Next Token Prediction with Longer Contexts: Additional Results G. Improving Next Token Prediction with Longer Contexts: Additional Results H. Additional Details of the Copy and Retrieval Tasks H. Additional Details of the Copy and Retrieval Tasks 2. Model Architecture All our models contain the following components: (i) a residual block, (ii) an MLP block, and (iii) a temporal-mixing block. While (i) and (ii) are the same across all models, we consider three temporal mixing blocks: global Multi-Query Attention (MQA), local (sliding-window) MQA and our proposed recurrent block. As part of the recurrent block we use the Real-Gated Linear Recurrent Unit (RG-LRU) – a novel recurrent layer inspired by the Linear Recurrent Unit (Orvieto et al., 2023b). The residual block, as shown in Figure 2(a), defines the global structure of our models and is inspired by pre-norm Transformers (Xiong et al., 2020). After embedding the input sequence we pass it through 𝑁 such blocks (𝑁 denoting the model depth), and then we apply RMSNorm (Zhang and Sennrich, 2019) to produce the final activations. To compute the token probabilities we apply a final linear layer followed by a softmax. The weights of this layer are shared with the input embedding layer. 2.1. Residual block The residual block contains two components, applied in order. The first component takes the hidden state 𝑥 and applies an RMSNorm (Zhang and Sennrich, 2019), followed by the temporal-mixing block. We then merge the output with a skip connection from 𝑥 through addition. Similarly, the second component applies RMSNorm, followed by the MLP block and then merges its output with a skip connection from the input of the RMSNorm. This block is illustrated in Figure 2 (a). 2.2. MLP block We use a gated MLP block (Dauphin et al., 2017) (illustrated in Figure 2(b)), which creates two branches from its input of dimension 𝐷. We apply a linear layer with output dimension 𝑀𝐷 on each branch, where 𝑀 denotes the expansion factor. For simplicity, we use 𝑀 =3 throughout this work. We apply a GeLU non-linearity (Hendrycks and Gimpel, 2016) on one of the branches before merging them by element-wise multiplication, similar to GeGeLU (Shazeer, 2020). However, in our MLP block, we apply a final linear layer with output dimension 𝐷 on the outputs of the GeGeLU layer. 2.3. Temporal-mixing blocks The temporal-mixing block is the component of our model that aggregates hidden layer activations at different temporal locations in the sequence. We consider three temporal-mixing blocks: global MQA (Shazeer, 2019), local MQA (Beltagy et al., 2020) and our proposed Recurrent block. Local sliding window attention One of the key disadvantages of using global attention is that its computational complexity grows quadratically in the sequence length. To address this, several works have started to adopt local attention (Beltagy et al., 2020), also known as sliding window attention. It allows each position to attend only to a fixed number of tokens in the past. This not only reduces the computational FLOPs but also bounds the size of the KV cache to the size of window, making it no longer quadratic in the sequence length. All other details are the same as the global MQA. Local sliding window attention 2.4. Real-Gated Linear Recurrent Unit (RG-LRU) Our proposed RG-LRU layer has a simple recurrence inspired by the Linear Recurrent Unit (LRU) (Orvieto et al., 2023b), but incorporates a gating mechanism motivated by the literature on non-linear RNNs, in particular LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Chung et al., 2014). The equations describing the layer are as follows: This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] We suggest ablating the use of complex numbers for other modalities and provide more information about the complex-valued version of the RG-LRU layer in Appendix B.

RG-LRU: A Breakthrough Recurrent Layer Redefining NLP Model Efficiency

Authors:
(1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch);
(2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch);
(3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.2 Anchor Regression 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation B Experiments 4.2 Housing nonlinear regression We extend the results from the previous section to the California and Boston housing data and compare ADA to C-Mixup [49]. We repeat the same experiments on three different regression datasets. Results are provided in Appendix B.2 and also show the superiority of ADA over C-Mixup for data augmentation in the implemented experimental setup. Data: We use the California housing dataset [19] and the Boston housing dataset [14]. The training dataset contains up to n = 406 samples, and the remaining samples are for validation. We report the results as a function of the number of training points. Models and comparisons: We fit a ridge regression model (baseline) and train a MLP with one hidden layer with a varying number of hidden units with sigmoid activation. The baseline models only use only the original data. We train the same models using C-Mixup with a Gaussian kernel and bandwidth of 1.75. We compare the previous approaches to models fitted on ADA augmented data. We generate 20 different augmentations per original observation using different values for γ controlled via α = 4 similar to what was described in Section 4.1. The Anchor matrix is constructed using k-means clustering with q = 10. Results: We report the results in Figure 3. First, we observe that the MLPs outperform Ridge regression suggesting a nonlinear data structure. Second, when the number of training samples is low, applying ADA improves the performance of all models compared to C-Mixup and the baseline. The performance gap decreases as the number of samples increases. When comparing C-Mixup and ADA, we see that using sufficiently many samples both methods achieve similar performance. While on the Boston data, the performance gap between the baseline and ADA persists, on California housing, the non-augmented model fit performs better than the augmented one when data availability increases. This suggests that there is a sweet spot where the addition of original data samples is required for better generalization, and augmented samples cannot contribute any further. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Authors: Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.1 Data Augmentation 2.2 Anchor Regression 2.2 Anchor Regression 3 Anchor Data Augmentation 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation A Additional information for Anchor Data Augmentation B Experiments B Experiments 4.2 Housing nonlinear regression We extend the results from the previous section to the California and Boston housing data and compare ADA to C-Mixup [49]. We repeat the same experiments on three different regression datasets. Results are provided in Appendix B.2 and also show the superiority of ADA over C-Mixup for data augmentation in the implemented experimental setup. Data : We use the California housing dataset [19] and the Boston housing dataset [14]. The training dataset contains up to n = 406 samples, and the remaining samples are for validation. We report the results as a function of the number of training points. Data Models and comparisons: We fit a ridge regression model (baseline) and train a MLP with one hidden layer with a varying number of hidden units with sigmoid activation. The baseline models only use only the original data. We train the same models using C-Mixup with a Gaussian kernel and bandwidth of 1.75. We compare the previous approaches to models fitted on ADA augmented data. We generate 20 different augmentations per original observation using different values for γ controlled via α = 4 similar to what was described in Section 4.1. The Anchor matrix is constructed using k-means clustering with q = 10. Models and comparisons: Results : We report the results in Figure 3. First, we observe that the MLPs outperform Ridge regression suggesting a nonlinear data structure. Second, when the number of training samples is low, applying ADA improves the performance of all models compared to C-Mixup and the baseline. The performance gap decreases as the number of samples increases. When comparing C-Mixup and ADA, we see that using sufficiently many samples both methods achieve similar performance. While on the Boston data, the performance gap between the baseline and ADA persists, on California housing, the non-augmented model fit performs better than the augmented one when data availability increases. This suggests that there is a sweet spot where the addition of original data samples is required for better generalization, and augmented samples cannot contribute any further. Results This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

ADA vs C-Mixup: Performance on California and Boston Housing Datasets

Authors:
(1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch);
(2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch);
(3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.2 Anchor Regression 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation B Experiments 4 Experiments We experimentally investigate and compare the performance of ADA. First, we use ADA in an in-distribution setting for a linear regression problem (Section 4.1), in which we show that even in this case, ADA provides improved performance in the low data regime. Second, in Section 4.2, we apply ADA and C-Mixup to the California and Boston Housing datasets as we increase the number of training samples. In the last two subsections, we replicate the in-distribution generalization (Section 4.3) and the out-of-distribution Robustness (Section 4.4) from the C-Mixup paper [49]. In [49] the authors further assess a task generalization experiment. However, the corresponding code was not publicly provided, and a comparison could not be easily made. 4.1 Linear synthetic data Using synthetic linear data, we investigate if ADA can improve model performance in an overparameterized setting compared to C-Mixup, vanilla augmentation, or classical expected risk minimization (ERM). Additionally, we analyze the sensitivity of our approach to the choice of γ and the number of augmentations. Data: The generated data follows a standard linear structure Models and Comparisons: We investigate and compare the impact of ADA using two different models with varying complexity: a linear Ridge regression and a multilayer perceptron (MLP) with one hidden layer and 10 units with ReLU activation. Using an MLP with more hidden layers shows similar results (see Appendix B.1 for details). For the Ridge regression model, we increase the dataset by a factor of 10 by sampling from the respective augmentation methods and subsequently compute the regression estimators. In contrast, for the MLP, we implement the augmentation methods on a minibatch level. Specifically, we incorporate vanilla augmentation by adding Gaussian noise to each batch, apply C-Mixup after sampling from the beta distribution in each batch, and finally, apply ADA after sampling from the defined gamma values in each batch. In summary, even in the simplest of cases, in which we should not expect gains from ADA (or C-Mixup), these data augmentation strategies provide gains in performance when the number of training examples is not sufficient to achieve the error floor. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Authors: Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.1 Data Augmentation 2.2 Anchor Regression 2.2 Anchor Regression 3 Anchor Data Augmentation 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation A Additional information for Anchor Data Augmentation B Experiments B Experiments 4 Experiments We experimentally investigate and compare the performance of ADA. First, we use ADA in an in-distribution setting for a linear regression problem (Section 4.1), in which we show that even in this case, ADA provides improved performance in the low data regime. Second, in Section 4.2, we apply ADA and C-Mixup to the California and Boston Housing datasets as we increase the number of training samples. In the last two subsections, we replicate the in-distribution generalization (Section 4.3) and the out-of-distribution Robustness (Section 4.4) from the C-Mixup paper [49]. In [49] the authors further assess a task generalization experiment. However, the corresponding code was not publicly provided, and a comparison could not be easily made. 4.1 Linear synthetic data Using synthetic linear data, we investigate if ADA can improve model performance in an overparameterized setting compared to C-Mixup, vanilla augmentation, or classical expected risk minimization (ERM). Additionally, we analyze the sensitivity of our approach to the choice of γ and the number of augmentations. Data : The generated data follows a standard linear structure Data Models and Comparisons: We investigate and compare the impact of ADA using two different models with varying complexity: a linear Ridge regression and a multilayer perceptron (MLP) with one hidden layer and 10 units with ReLU activation. Using an MLP with more hidden layers shows similar results (see Appendix B.1 for details). Models and Comparisons: For the Ridge regression model, we increase the dataset by a factor of 10 by sampling from the respective augmentation methods and subsequently compute the regression estimators. In contrast, for the MLP, we implement the augmentation methods on a minibatch level. Specifically, we incorporate vanilla augmentation by adding Gaussian noise to each batch, apply C-Mixup after sampling from the beta distribution in each batch, and finally, apply ADA after sampling from the defined gamma values in each batch. In summary, even in the simplest of cases, in which we should not expect gains from ADA (or C-Mixup), these data augmentation strategies provide gains in performance when the number of training examples is not sufficient to achieve the error floor. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

Evaluating ADA: Experimental Results on Linear and Housing Datasets

Table of Links Abstract and 1 Introduction 2 Preliminaries 3 End-to-End Adaptive Local Learning 3.1 Loss-Driven Mixture-of-Experts 3.2 Synchronized Learning via Adaptive Weight 4 Debiasing Experiments and 4.1 Experimental Setup 4.2 Debiasing Performance 4.3 Ablation Study 4.4 Effect of the Adaptive Weight Module and 4.5 Hyper-parameter Study 5 Related Work 6 Conclusion, Acknowledgements, and References 4 Debiasing Experiments In this section, we present a comprehensive set of experiments to highlight the strong debiasing performance of the proposed method, validate the effectiveness of various model components, assess the impact of the proposed adaptive weight module, and examine the impact of hyperparameters. 4.1 Experimental Setup Considering the Rawlsian Max-Min fairness principle [29], the goal of debiasing is to promote the average NDCG@20 for subgroups with low mainstream scores while preserving or even improving the utility for subgroups with high mainstream scores at the same time. Hence, we also anticipate an increase in the overall NDCG@20 of the model. Baselines. In the experiments, we compare the proposed TALL with MultVAE and four state-of-art debiasing methods: (1) MultVAE [26] is the widely used vanilla recommendation model without debiasing. (2) WL [39] is a global method designed to assign more weights to niche users in the training loss of MultVAE. (3) LOCA [12] is a local learning model that trains multiple anchor models corresponding to identified anchor users and aggregates the outputs from anchor models based on the similarity between the target user and anchor users. (4) LFT [39] is the SOTA local learning model that first trains a global model with all data and then fine-tunes a customized local model for each user using their local data for the target user. (5) EnLFT [39] is the ensembled version of LFT, which is similar to LOCA but trains the anchor models by the approach of LFT. To fairly compare the performance of different baselines and our proposed model, for all four debiasing baselines and our proposed TALL, we adopt the MultVAE as the base model (or the expert model in TALL). LOCA, EnLFT, and TALL have the same complexity with a fixed number of MultVAE in them. And owing to the end-to-end training paradigm, TALL takes less training time than other local learning baselines. Last, LFT has the largest complexity, which has an independent MultVAE for each user. Reproducibility. All models are implemented in PyTorch and optimized by the Adam algorithm [20]. For the baseline MultVAE and the MultVAE component in other models, we set one hidden layer of size 100. And we maintain the number of local models at 100 for LOCA, EnLFT, and TALL for all datasets to ensure a fair comparison. All other hyper-parameters are grid searched by the validation sets. All code and data can be found at https:// github.com/ JP-25/ end-To-end-Adaptive-Local-Leanring-TALL-. Authors:
(1) Jinhao Pan [0009 −0006 −1574 −6376], Texas A&M University, College Station, TX, USA;
(2) Ziwei Zhu [0000 −0002 −3990 −4774], George Mason University, Fairfax, VA, USA;
(3) Jianling Wang [0000 −0001 −9916 −0976], Texas A&M University, College Station, TX, USA;
(4) Allen Lin [0000 −0003 −0980 −4323], Texas A&M University, College Station, TX, USA;
(5) James Caverlee [0000 −0001 −8350 −8528]. Texas A&M University, College Station, TX, USA. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries 2 Preliminaries 3 End-to-End Adaptive Local Learning 3 End-to-End Adaptive Local Learning 3.1 Loss-Driven Mixture-of-Experts 3.1 Loss-Driven Mixture-of-Experts 3.2 Synchronized Learning via Adaptive Weight 3.2 Synchronized Learning via Adaptive Weight 4 Debiasing Experiments and 4.1 Experimental Setup 4 Debiasing Experiments and 4.1 Experimental Setup 4.2 Debiasing Performance 4.2 Debiasing Performance 4.3 Ablation Study Ablation 4.4 Effect of the Adaptive Weight Module and 4.5 Hyper-parameter Study 4.4 Effect of the Adaptive Weight Module and 4.5 Hyper-parameter Study 5 Related Work 5 Related Work 6 Conclusion, Acknowledgements, and References 6 Conclusion, Acknowledgements, and References 4 Debiasing Experiments In this section, we present a comprehensive set of experiments to highlight the strong debiasing performance of the proposed method, validate the effectiveness of various model components, assess the impact of the proposed adaptive weight module, and examine the impact of hyperparameters. 4.1 Experimental Setup Considering the Rawlsian Max-Min fairness principle [29], the goal of debiasing is to promote the average NDCG@20 for subgroups with low mainstream scores while preserving or even improving the utility for subgroups with high mainstream scores at the same time. Hence, we also anticipate an increase in the overall NDCG@20 of the model. the goal of debiasing is to promote the average NDCG@20 for subgroups with low mainstream scores while preserving or even improving the utility for subgroups with high mainstream scores at the same time. Hence, we also anticipate an increase in the overall NDCG@20 of the model. Baselines . In the experiments, we compare the proposed TALL with MultVAE and four state-of-art debiasing methods: (1) MultVAE [26] is the widely used vanilla recommendation model without debiasing. (2) WL [39] is a global method designed to assign more weights to niche users in the training loss of MultVAE. Baselines MultVAE WL (3) LOCA [12] is a local learning model that trains multiple anchor models corresponding to identified anchor users and aggregates the outputs from anchor models based on the similarity between the target user and anchor users. (4) LFT [39] is the SOTA local learning model that first trains a global model with all data and then fine-tunes a customized local model for each user using their local data for the target user. (5) EnLFT [39] is the ensembled version of LFT, which is similar to LOCA but trains the anchor models by the approach of LFT. LOCA LFT EnLFT To fairly compare the performance of different baselines and our proposed model, for all four debiasing baselines and our proposed TALL, we adopt the MultVAE as the base model (or the expert model in TALL). LOCA, EnLFT, and TALL have the same complexity with a fixed number of MultVAE in them. And owing to the end-to-end training paradigm, TALL takes less training time than other local learning baselines. Last, LFT has the largest complexity, which has an independent MultVAE for each user. Reproducibility . All models are implemented in PyTorch and optimized by the Adam algorithm [20]. For the baseline MultVAE and the MultVAE component in other models, we set one hidden layer of size 100. And we maintain the number of local models at 100 for LOCA, EnLFT, and TALL for all datasets to ensure a fair comparison. All other hyper-parameters are grid searched by the validation sets. All code and data can be found at https:// github.com/ JP-25/ end-To-end-Adaptive-Local-Leanring-TALL-. Reproducibility Authors: (1) Jinhao Pan [0009 −0006 −1574 −6376], Texas A&M University, College Station, TX, USA; (2) Ziwei Zhu [0000 −0002 −3990 −4774], George Mason University, Fairfax, VA, USA; (3) Jianling Wang [0000 −0001 −9916 −0976], Texas A&M University, College Station, TX, USA; (4) Allen Lin [0000 −0003 −0980 −4323], Texas A&M University, College Station, TX, USA; (5) James Caverlee [0000 −0001 −8350 −8528]. Texas A&M University, College Station, TX, USA. Authors: Authors: (1) Jinhao Pan [0009 −0006 −1574 −6376], Texas A&M University, College Station, TX, USA; (2) Ziwei Zhu [0000 −0002 −3990 −4774], George Mason University, Fairfax, VA, USA; (3) Jianling Wang [0000 −0001 −9916 −0976], Texas A&M University, College Station, TX, USA; (4) Allen Lin [0000 −0003 −0980 −4323], Texas A&M University, College Station, TX, USA; (5) James Caverlee [0000 −0001 −8350 −8528]. Texas A&M University, College Station, TX, USA. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Countering Mainstream Bias via End-to-End Adaptive Local Learning: Debiasing Experiments and Setup

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details 3 PRELIMINARIES A deep transformer architecture of depth L is formed by sequentially stacking L transformer blocks. The most common block is Pre-LN, depicted in Fig. 1 (left), which we treat as a baseline for comparing training speed, both in terms of per-update and runtime. It differs from the original Post-LN block only in the position of the normalisation layers relative to the skip connections, but is more popular as the Post-LN block suffers from poor training stability and signal propagation in deep layers (Xiong et al., 2020; Liu et al., 2020; Noci et al., 2022; He et al., 2023). with scalar gain weights αFF, βFF, αSA, βSA fixed to 1 by default. Here, “MHA” stands for MultiHead Attention (detailed below), and “Norm” denotes a normalisation layer (Ba et al., 2016; Zhang & Sennrich, 2019). In words, we see that the Pre-LN transformer block consists of two sequential sub-blocks (one attention and one MLP), with normalisation layers and residual connections for both sub-blocks, and crucially the normalisation layers are placed within the residual branch. The MLP is usually single hidden-layer, with hidden dimension that is some multiple of d (e.g. 4 (Vaswani et al., 2017) or 8/3 (Touvron et al., 2023)), and acts on each token in the sequence independently. The MHA sub-block allows tokens to share information between one another using self-attention. For input sequence X, the self-attention mechanism outputs: This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details 3 PRELIMINARIES A deep transformer architecture of depth L is formed by sequentially stacking L transformer blocks. The most common block is Pre-LN, depicted in Fig. 1 (left), which we treat as a baseline for comparing training speed, both in terms of per-update and runtime. It differs from the original Post-LN block only in the position of the normalisation layers relative to the skip connections, but is more popular as the Post-LN block suffers from poor training stability and signal propagation in deep layers (Xiong et al., 2020; Liu et al., 2020; Noci et al., 2022; He et al., 2023). with scalar gain weights αFF, βFF, αSA, βSA fixed to 1 by default. Here, “MHA” stands for MultiHead Attention (detailed below), and “Norm” denotes a normalisation layer (Ba et al., 2016; Zhang & Sennrich, 2019). In words, we see that the Pre-LN transformer block consists of two sequential sub-blocks (one attention and one MLP), with normalisation layers and residual connections for both sub-blocks, and crucially the normalisation layers are placed within the residual branch. The MLP is usually single hidden-layer, with hidden dimension that is some multiple of d (e.g. 4 (Vaswani et al., 2017) or 8/3 (Touvron et al., 2023)), and acts on each token in the sequence independently. The MHA sub-block allows tokens to share information between one another using self-attention. For input sequence X, the self-attention mechanism outputs: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Improving Training Stability in Deep Transformers: Pre-LN vs. Post-LN Blocks

One of the most common question asked during deep learning knowledge interviews is - “Why can’t we use a quadratic cost function to train a Neural Network?”. In this article, we’ll discuss the answer to this question. There will be a lot of Math involved but nothing crazy! I will keep things simple and precise. Let’s start with contemplating the general architecture of a Neural network The General Architecture of a Neural Network We have a series of inputs forming an “input layer”, a series of neurons in the “hidden layer” and one neuron forming an “output layer” for a binary classification problem. For this question, we will assume that we are dealing with a binary classifier, so we have just one output value out of the network. Now, look at the following figure where we’ve highlighted the input layer in green, the output neuron in red, and one neuron of the hidden layer in orange. From all greens to the orange, we see that all inputs are connected to the orange neuron. In other words, the “activation” of the orange neuron happens using the “aggregation” of all the green neurons in the input layer. This process is replicated over all neurons over all layers until we reach the final red output neuron. What if we replace the orange neuron with the red neuron, i.e., we remove the hidden layer and connect the red neuron with the green neurons directly? We will get the following: For this question, we will assume the aforementioned ‘simplistic architecture’ and the result can be generalized to the complete architecture as well. Now let’s introduce some Math step-by-step. What we see above is the basic “weight update” equation for a Neural network. I have removed the extra hyperparameters such as the learning factor and the sub-sampling (min-batch), etc. w_k is the vector of weights and the weights are the ‘parameters’ of our Neural network model. w_k comprises individual weights gathered in a column vector. These weights are associated with the inputs to the model (that is the green neurons in our architecture). We have a cost function C_i where i = 1 to n are the number of data instances in our sample. The cost function C is the “error” between the actual output y and the output from the neural network (red neuron). Each data instance will produce a predicted output as against an actual output, therefore, there will be a cost or error for every data instance. The objective of the model is to minimize this cost function on an average over the entire dataset. And as we know, the minimization step involves taking a derivative with respect to the model parameters (weights). We do this using the partial derivative of the vector w_k. All this means is that the cost C will be an expression/aggregation of weights w_1 to w_q and we will differentiate to each weight w and collect that in a vector. This is called the negative “gradient vector”. It is used to update the weight vector from the k-th iteration to the (k+1)th iteration. The methodology is Stochastic Gradient descent but we will leave that out for this article. In a nutshell, the neural network learns by an update to the weights via the negative gradient vector averaged over all the samples and calculated for w_k. This helps us move to the minimization of the cost function and helps the network to learn and improve its accuracy. It is obvious that if the updates to the weights are not happening, because the negative gradient is moving towards zero, the learning has stopped. This doesn’t necessarily imply that we have reached the minimum! Because our cost function is highly complicated and we need to find a minimum in a multi-dimensional space. Therefore, there could be many local minima where the gradient is zero and the network stops learning. Anyway, we don’t have to worry about that for this problem. Let’s look at the following expression: This expression defines z_i as a weighted sum of the inputs x_ji. Note that these inputs are the green neurons in our architecture. As we have no hidden layer, we combine the inputs x_ji and the weights w_j and add a bias term to get z_i which is what is represented by the connecting arrows from the green neurons to the red neuron in our architecture. Since we have q inputs, we have x_j and w_j where j = 1 to q But, we don’t pass z_i to the red neuron. We apply an “activation function” to it. This activation function can be different for different neural networks. For the purposes of this problem, we assume the activation function is “Sigmoid” or “logistic”. I will assume here that the reader is aware of this function and move on further. Next comes our main problem statement -  How do we define the Cost function (C)? It is well known that for binary classification, the cost function is “Cross entropy” but the question here is why can’t it be “Quadratic”. Let’s define the expressions of both the cost functions: Quadratic cost function –   Cross Entropy cost function – While the quadratic cost function is straightforward (think least squares minimization between the actual output y_i and the predicted output a_i), we can offer some explanation for the cross-entropy cost function. This is akin to negative log-likelihood in our regression models. Note that there is a negative sign outside the brackets, which is used to keep the cost positive (because a_i will be between 0 and 1 - an output of sigmoid, therefore, the term inside the brackets will be always negative). Also note that when a_i gets really close to y_i, the cost gets really close to zero. This is because, when y_i = 1 and a_i ~ 1, ln(a_i) will be approximately 0. Similarly, when y_i= 0 and a_i  ~ 0, ln(1-a_i) will be approximately 0. Thus, this function keeps the cost positive and minimal when the model is predicting well. However, the same can be said about the quadratic cost function as well. But, we don’t use it. Why? Here comes the explanation We go back to the basic weight update equation we saw earlier and input the quadratic cost function to it. We get the following Now to keep things simple, we will consider only one data point, that is i=1 and n=1. And we differentiate partially for each weight w_j. We get the following: Recall that since i = 1, we have Substituting the value of z, we get That is our gradient vector which is responsible for updating the weights of the network will have a derivative of the sigmoid function when we use a quadratic cost function. Now let’s look at the behavior of the derivative of the sigmoid function: From the above plot, it is clear that the derivative, representing the slope of the sigmoid function, approaches 0 as soon as the input z becomes large! What does this mean? This means that the gradient vector will be zero when the activation input z is large. Therefore, the network will stop learning as the weights won’t get updated. Recall that this does not mean we have reached a minimum. This means we are stuck at an undesirable point and in the function space which could be far from the minimum value. This is known as “learning slow down”. However, this does NOT occur with a cross-entropy cost function. We perform the same substitution using the cross-entropy cost function and get the following: It is interesting to note that the term: occurs in the gradient for quadratic cost as well. However, there is a trick which we will use to simplify it. The gradient of the sigmoid function can be expressed as follows We substitute that into our original expression and get the following: That is our gradient vector which is responsible for updating the weights of the network does not have a derivative of the sigmoid function when we use a cross entropy cost function. Hence, there is no slowdown in learning with this cost function. We juxtapose the gradients again for a better look: This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well! One of the most common question asked during deep learning knowledge interviews is - “ Why can’t we use a quadratic cost function to train a Neural Network? ”. In this article, we’ll discuss the answer to this question. There will be a lot of Math involved but nothing crazy! I will keep things simple and precise. Why can’t we use a quadratic cost function to train a Neural Network? Let’s start with contemplating the general architecture of a Neural network Neural network The General Architecture of a Neural Network We have a series of inputs forming an “input layer”, a series of neurons in the “hidden layer” and one neuron forming an “output layer” for a binary classification problem. For this question, we will assume that we are dealing with a binary classifier, so we have just one output value out of the network. Now, look at the following figure where we’ve highlighted the input layer in green, the output neuron in red, and one neuron of the hidden layer in orange. From all greens to the orange, we see that all inputs are connected to the orange neuron. In other words, the “activation” of the orange neuron happens using the “aggregation” of all the green neurons in the input layer. This process is replicated over all neurons over all layers until we reach the final red output neuron. What if we replace the orange neuron with the red neuron, i.e., we remove the hidden layer and connect the red neuron with the green neurons directly? We will get the following: We will get the following: For this question, we will assume the aforementioned ‘simplistic architecture’ and the result can be generalized to the complete architecture as well. Now let’s introduce some Math step-by-step. Now let’s introduce some Math step-by-step. What we see above is the basic “weight update” equation for a Neural network. I have removed the extra hyperparameters such as the learning factor and the sub-sampling (min-batch), etc. w_k is the vector of weights and the weights are the ‘parameters’ of our Neural network model. w_k comprises individual weights gathered in a column vector. These weights are associated with the inputs to the model (that is the green neurons in our architecture). We have a cost function C_i where i = 1 to n are the number of data instances in our sample. The cost function C is the “error” between the actual output y and the output from the neural network (red neuron). Each data instance will produce a predicted output as against an actual output, therefore, there will be a cost or error for every data instance. The objective of the model is to minimize this cost function on an average over the entire dataset. And as we know, the minimization step involves taking a derivative with respect to the model parameters (weights). We do this using the partial derivative of the vector w_k. All this means is that the cost C will be an expression/aggregation of weights w_1 to w_q and we will differentiate to each weight w and collect that in a vector. This is called the negative “gradient vector”. It is used to update the weight vector from the k-th iteration to the (k+1)th iteration. The methodology is Stochastic Gradient descent but we will leave that out for this article. In a nutshell, the neural network learns by an update to the weights via the negative gradient vector averaged over all the samples and calculated for w_k. This helps us move to the minimization of the cost function and helps the network to learn and improve its accuracy. It is obvious that if the updates to the weights are not happening, because the negative gradient is moving towards zero, the learning has stopped. This doesn’t necessarily imply that we have reached the minimum! Because our cost function is highly complicated and we need to find a minimum in a multi-dimensional space. Therefore, there could be many local minima where the gradient is zero and the network stops learning. Anyway, we don’t have to worry about that for this problem. Let’s look at the following expression: Let’s look at the following expression: This expression defines z_i as a weighted sum of the inputs x_ji. Note that these inputs are the green neurons in our architecture. As we have no hidden layer, we combine the inputs x_ji and the weights w_j and add a bias term to get z_i which is what is represented by the connecting arrows from the green neurons to the red neuron in our architecture. Since we have q inputs, we have x_j and w_j where j = 1 to q But, we don’t pass z_i to the red neuron. We apply an “activation function” to it. This activation function can be different for different neural networks. For the purposes of this problem, we assume the activation function is “Sigmoid” or “logistic”. I will assume here that the reader is aware of this function and move on further. Next comes our main problem statement -  How do we define the Cost function (C)? It is well known that for binary classification, the cost function is “Cross entropy” but the question here is why can’t it be “Quadratic”. Let’s define the expressions of both the cost functions: Let’s define the expressions of both the cost functions: Quadratic cost function – Cross Entropy cost function – Quadratic cost function – Cross Entropy cost function – While the quadratic cost function is straightforward (think least squares minimization between the actual output y_i and the predicted output a_i), we can offer some explanation for the cross-entropy cost function. This is akin to negative log-likelihood in our regression models. Note that there is a negative sign outside the brackets, which is used to keep the cost positive (because a_i will be between 0 and 1 - an output of sigmoid, therefore, the term inside the brackets will be always negative). Also note that when a_i gets really close to y_i, the cost gets really close to zero. This is because, when y_i = 1 and a_i ~ 1, ln(a_i) will be approximately 0. Similarly, when y_i= 0 and a_i  ~ 0, ln(1-a_i) will be approximately 0. Thus, this function keeps the cost positive and minimal when the model is predicting well. However, the same can be said about the quadratic cost function as well. But, we don’t use it. Why? Here comes the explanation We go back to the basic weight update equation we saw earlier and input the quadratic cost function to it. We get the following Now to keep things simple, we will consider only one data point, that is i=1 and n=1. And we differentiate partially for each weight w_j. We get the following: Recall that since i = 1, we have Substituting the value of z, we get That is our gradient vector which is responsible for updating the weights of the network will have a derivative of the sigmoid function when we use a quadratic cost function. Now let’s look at the behavior of the derivative of the sigmoid function: Now let’s look at the behavior of the derivative of the sigmoid function: From the above plot, it is clear that the derivative, representing the slope of the sigmoid function, approaches 0 as soon as the input z becomes large! What does this mean? This means that the gradient vector will be zero when the activation input z is large. Therefore, the network will stop learning as the weights won’t get updated. Recall that this does not mean we have reached a minimum. This means we are stuck at an undesirable point and in the function space which could be far from the minimum value. This is known as “learning slow down”. However, this does NOT occur with a cross-entropy cost function. We perform the same substitution using the cross-entropy cost function and get the following: We perform the same substitution using the cross-entropy cost function and get the following: It is interesting to note that the term: occurs in the gradient for quadratic cost as well. However, there is a trick which we will use to simplify it. The gradient of the sigmoid function can be expressed as follows We substitute that into our original expression and get the following: That is our gradient vector which is responsible for updating the weights of the network does not have a derivative of the sigmoid function when we use a cross entropy cost function. Hence, there is no slowdown in learning with this cost function. not We juxtapose the gradients again for a better look: We juxtapose the gradients again for a better look: This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well! Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well!

Why Quadratic Cost Functions Are Ineffective in Neural Network Training

কেন চতুর্মুখী খরচ ফাংশন নিউরাল নেটওয়ার্ক প্রশিক্ষণ অকার্যকর হয়

Warum quadratische Kostenfunktionen beim Training neuronaler Netze unwirksam sind

Por qué las funciones de costo cuadrático son ineficaces en el entrenamiento de redes neuronales

Pourquoi les fonctions de coût quadratiques sont inefficaces dans la formation sur les réseaux neuronaux

न्यूरल नेटवर्क प्रशिक्षण में द्विघात लागत फ़ंक्शन अप्रभावी क्यों हैं

ニューラルネットワークのトレーニングで二次コスト関数が効果がない理由

신경망 훈련에서 2차 비용 함수가 비효율적인 이유

Por que as funções de custo quadráticas são ineficazes no treinamento de redes neurais

Почему квадратичные функции стоимости неэффективны при обучении нейронных сетей

İkinci Dereceden Maliyet Fonksiyonları Sinir Ağı Eğitiminde Neden Etkisizdir?

Tại sao hàm chi phí bậc hai không hiệu quả trong đào tạo mạng lưới thần kinh

为什么二次成本函数在神经网络训练中无效

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Edson Pindza, Tshwane University of Technology; Department of Mathematics and Statistics; 175 Nelson Mandela Drive OR Private Bag X680 and Pretoria 0001; South Africa [edsonpindza@gmail.com];
(2) Jules Clement Mba, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [jmba@uj.ac.za];
(3) Sutene Mwambi, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [sutenem@uj.ac.za];
(4) Nneka Umeorah, Cardiff University; School of Mathematics; Cardiff CF24 4AG; United Kingdom [umeorahn@cardiff.ac.uk]. Table of Links Abstract and Introduction
Methodology
Neural Network Methodology
Numerical results, implementation and discussion
Conclusion, Acknowledgments, and Funding
Availability of data, code and materials, Contributions and Declarations
References 3. Neural Network Methodology The neural network (NN) algorithm is used for solving problems relating to dimensionality reduction [42, 49], visualization of data [31], clustering [11, 52] and classification issues [37, 40]. It is also used in regression [2, 21, 45] in solving regression problems since they do not require prior knowledge about the distribution of the data. The NN algorithm often has the tendency to predict with higher accuracy than other techniques due to the NN’s capability to fit any continuous functions [18]. Mathematically, the NN can be referred to as a directed graph having neurons as vertices and links as edges. Different forms of NN depend on the connectivity of their neurons, and the simplest one is the Multi-layer perceptron, also known as the feedforward NN. Basically, a NN can have only one input and one output layer in its simplest form, and this can be written as: where m is the number of input variables, Φ is the activation function, wi,k is the weight of the input layer i with respect to the output k, b is the bias, and the input vector x is connected to the output k (denoting the kth neuron in the output layer) through a biased weighted sum. In the presence of hidden layers positioned between the input layers and the output layers, the output can be written as: where m1 is the number of input variables, m2 is the number of nodes in the hidden layer, Φ : R → R refers to non-linear activation functions for each of the layers in the brackets, and Φout is the possibly new output function. Recently, the NN has become an indispensable tool for learning the solutions of differential equations, and this section will utilize the potential of the NN in solving PDE-related problems [1, 10, 20, 50]. Also, the NN has been implemented in solving equations without analytical solutions. For instance, Nwankwo et al. [38] considered the solution of the American options PDE by incorporating the Landau transformation and solving the corresponding transformed function via a neural network approach. In many scientific and industrial contexts, there is a need to solve parametric PDEs using NN techniques and Khoo et al. [24] developed such a technique which solves PDE with inhomogeneous coefficient fields. For high-dimensional parametric PDE, [14] analyzed the deep parametric PDE method to solve high-dimensional parametric PDEs with much emphasis on financial applications. In the following, we describe the Bitcoin options with the corresponding Black-Scholes pricing PDE: The above equation which is a non-homogeneous PDE can be re-written as Standard theorems guarantee a classical smooth solution exists for the pricing PDE (3.13) given the assumed dynamics. The continuity and linear growth conditions on the coefficient functions ensure they satisfy Lipschitz continuity. Contingent claims theory shows that Lipschitz continuity is sufficient for existence and uniqueness under the stochastic process specifications. In particular, the smooth past price dependence in the diffusion coefficient allows the application results for delayed Black-Scholes equations. Therefore, the Cauchy problem is well-posed under the model assumptions, and a smooth pricing solution is guaranteed to exist based on the theorems. This provides the theoretical foundation for using a neural network to approximate the price function numerically. Thus, employing the ANN to solve the PDE, we introduce an approximating function ζ(t, S : θ) with parameter set θ. The loss or cost function associated with this training is measured by how well the approximation function satisfies the differential operator, boundary conditions and the terminal condition of the option pricing PDE. These are given respectively as Suppose L(ζ) = 0, then ζ(t, S : θ) is a solution of the PDE in equation (3.13). The major aim is to obtain a set of parameters θ such that the function ζ(t, S : θ) minimizes the observed error of L(ζ). The procedure for seeking a good parameter set by minimizing this loss function using the stochastic gradient descent (SGD) optimizer is called “training”. Thus, using a Machine Learning approach, and in this case, the artificial NN, it will be feasible to minimize the function L(ζ) by applying the SGD approach on the sequence of asset and time points which are drawn at random. A part of the input samples, fully dependent on the mini-batch size, is drawn within each iteration to estimate the gradient of the given objective function. The gradient is then estimated over these mini-batches due to the limitations of the computer memory [30]. The training of NN is classified into first identifying the network’s hyperparameters, that is, the architectural structure and the loss function of the network. Next, use the SGD to optimize or estimate the loss minimizer, and then the derivatives of the loss function are computed using the backpropagation techniques. Essentially, the process of searching and identifying the best θ parameter by minimizing the loss function using the gradient descent-based optimizers is referred to as “training”. The whole procedure is well illustrated in the algorithm below [46], and can be implemented in solving the PDE in equation (3.13): The learning rate α is employed to scale the intensity of the parameter updates during the gradient descent. Choosing the learning rate, as the descent parameter α, in the network training is crucial since this factor plays a significant role in aiding the algorithm’s convergence to the true solution in the gradient descent. It equally affects the pace at which the algorithm learns and whether or not the cost function is minimized. When α implies divergence, and thus, the optimal point can be missed. Also, a fixed value of α can most likely make the local optima overshoot after some iterations, leading to divergence. Defining the learning rate as a dynamic decreasing function is preferable since it allows the algorithm to identify the needed point rapidly. Another limitation in employing the gradient descent method is obtaining the value of the initial parameters of the loss function. If the value is significantly close to the local optimum, then the slope of the loss function reduces, thereby leading to the optimal convergence. Otherwise, there will be no convergence as the solution explodes abnormally. 3.1. Solution of bitcoin option pricing PDE While the pricing PDE has a proven smooth solution under the model dynamics, obtaining the analytical form is intractable. Numerical methods must be used to approximate the solution. Neural networks provide a flexible parametric approach based on their universal approximation theoretical results. Given sufficient network capacity, neural networks can represent a wide class of functions, including solutions to PDEs like the pricing equation. However, challenges remain in finding the optimal network parameters to recover the true solution robustly. While existence is guaranteed, the non-convex optimization of neural networks does not assure convergence to global optimality. Care must be taken in specifying the network architecture, loss function, regularization, and training methodology to promote the learning of the pricing function. Subject to these caveats, neural networks present a promising computational technique for approximating smooth pricing solutions, circumventing discretization of the domain. We note the limitations of using neural networks as function approximators. However, their generalization capabilities provide a methodology for data-driven extraction of the pricing mapping across the entire domain. This avoids the constraints of methods like grid-based techniques that rely on local consistency. Future work should explore neural network training enhancements and theoretical guarantees to ensure robust solutions. This paper is available on arxiv under CC 4.0 license. Authors: (1) Edson Pindza, Tshwane University of Technology; Department of Mathematics and Statistics; 175 Nelson Mandela Drive OR Private Bag X680 and Pretoria 0001; South Africa [edsonpindza@gmail.com]; (2) Jules Clement Mba, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [jmba@uj.ac.za]; (3) Sutene Mwambi, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [sutenem@uj.ac.za]; (4) Nneka Umeorah, Cardiff University; School of Mathematics; Cardiff CF24 4AG; United Kingdom [umeorahn@cardiff.ac.uk]. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Edson Pindza, Tshwane University of Technology; Department of Mathematics and Statistics; 175 Nelson Mandela Drive OR Private Bag X680 and Pretoria 0001; South Africa [edsonpindza@gmail.com]; (2) Jules Clement Mba, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [jmba@uj.ac.za]; (3) Sutene Mwambi, University of Johannesburg; School of Economics, College of Business and Economics and P. O. Box 524, Auckland Park 2006; South Africa [sutenem@uj.ac.za]; (4) Nneka Umeorah, Cardiff University; School of Mathematics; Cardiff CF24 4AG; United Kingdom [umeorahn@cardiff.ac.uk]. Table of Links Abstract and Introduction Methodology Neural Network Methodology Numerical results, implementation and discussion Conclusion, Acknowledgments, and Funding Availability of data, code and materials, Contributions and Declarations References Abstract and Introduction Abstract and Introduction Methodology Methodology Neural Network Methodology Neural Network Methodology Numerical results, implementation and discussion Numerical results, implementation and discussion Conclusion, Acknowledgments, and Funding Conclusion, Acknowledgments, and Funding Availability of data, code and materials, Contributions and Declarations Availability of data, code and materials, Contributions and Declarations References References 3. Neural Network Methodology The neural network (NN) algorithm is used for solving problems relating to dimensionality reduction [42, 49], visualization of data [31], clustering [11, 52] and classification issues [37, 40]. It is also used in regression [2, 21, 45] in solving regression problems since they do not require prior knowledge about the distribution of the data. The NN algorithm often has the tendency to predict with higher accuracy than other techniques due to the NN’s capability to fit any continuous functions [18]. Mathematically, the NN can be referred to as a directed graph having neurons as vertices and links as edges. Different forms of NN depend on the connectivity of their neurons, and the simplest one is the Multi-layer perceptron, also known as the feedforward NN. Basically, a NN can have only one input and one output layer in its simplest form, and this can be written as: where m is the number of input variables, Φ is the activation function, wi,k is the weight of the input layer i with respect to the output k, b is the bias, and the input vector x is connected to the output k (denoting the kth neuron in the output layer) through a biased weighted sum. In the presence of hidden layers positioned between the input layers and the output layers, the output can be written as: where m1 is the number of input variables, m2 is the number of nodes in the hidden layer, Φ : R → R refers to non-linear activation functions for each of the layers in the brackets, and Φout is the possibly new output function. Recently, the NN has become an indispensable tool for learning the solutions of differential equations, and this section will utilize the potential of the NN in solving PDE-related problems [1, 10, 20, 50]. Also, the NN has been implemented in solving equations without analytical solutions. For instance, Nwankwo et al. [38] considered the solution of the American options PDE by incorporating the Landau transformation and solving the corresponding transformed function via a neural network approach. In many scientific and industrial contexts, there is a need to solve parametric PDEs using NN techniques and Khoo et al. [24] developed such a technique which solves PDE with inhomogeneous coefficient fields. For high-dimensional parametric PDE, [14] analyzed the deep parametric PDE method to solve high-dimensional parametric PDEs with much emphasis on financial applications. In the following, we describe the Bitcoin options with the corresponding Black-Scholes pricing PDE: The above equation which is a non-homogeneous PDE can be re-written as Standard theorems guarantee a classical smooth solution exists for the pricing PDE (3.13) given the assumed dynamics. The continuity and linear growth conditions on the coefficient functions ensure they satisfy Lipschitz continuity. Contingent claims theory shows that Lipschitz continuity is sufficient for existence and uniqueness under the stochastic process specifications. In particular, the smooth past price dependence in the diffusion coefficient allows the application results for delayed Black-Scholes equations. Therefore, the Cauchy problem is well-posed under the model assumptions, and a smooth pricing solution is guaranteed to exist based on the theorems. This provides the theoretical foundation for using a neural network to approximate the price function numerically. Thus, employing the ANN to solve the PDE, we introduce an approximating function ζ(t, S : θ) with parameter set θ. The loss or cost function associated with this training is measured by how well the approximation function satisfies the differential operator, boundary conditions and the terminal condition of the option pricing PDE. These are given respectively as Suppose L(ζ) = 0, then ζ(t, S : θ) is a solution of the PDE in equation (3.13). The major aim is to obtain a set of parameters θ such that the function ζ(t, S : θ) minimizes the observed error of L(ζ). The procedure for seeking a good parameter set by minimizing this loss function using the stochastic gradient descent (SGD) optimizer is called “training”. Thus, using a Machine Learning approach, and in this case, the artificial NN, it will be feasible to minimize the function L(ζ) by applying the SGD approach on the sequence of asset and time points which are drawn at random. A part of the input samples, fully dependent on the mini-batch size, is drawn within each iteration to estimate the gradient of the given objective function. The gradient is then estimated over these mini-batches due to the limitations of the computer memory [30]. The training of NN is classified into first identifying the network’s hyperparameters, that is, the architectural structure and the loss function of the network. Next, use the SGD to optimize or estimate the loss minimizer, and then the derivatives of the loss function are computed using the backpropagation techniques. Essentially, the process of searching and identifying the best θ parameter by minimizing the loss function using the gradient descent-based optimizers is referred to as “training”. The whole procedure is well illustrated in the algorithm below [46], and can be implemented in solving the PDE in equation (3.13): The learning rate α is employed to scale the intensity of the parameter updates during the gradient descent. Choosing the learning rate, as the descent parameter α, in the network training is crucial since this factor plays a significant role in aiding the algorithm’s convergence to the true solution in the gradient descent. It equally affects the pace at which the algorithm learns and whether or not the cost function is minimized. When α implies divergence, and thus, the optimal point can be missed. Also, a fixed value of α can most likely make the local optima overshoot after some iterations, leading to divergence. Defining the learning rate as a dynamic decreasing function is preferable since it allows the algorithm to identify the needed point rapidly. Another limitation in employing the gradient descent method is obtaining the value of the initial parameters of the loss function. If the value is significantly close to the local optimum, then the slope of the loss function reduces, thereby leading to the optimal convergence. Otherwise, there will be no convergence as the solution explodes abnormally. 3.1. Solution of bitcoin option pricing PDE 3.1. Solution of bitcoin option pricing PDE While the pricing PDE has a proven smooth solution under the model dynamics, obtaining the analytical form is intractable. Numerical methods must be used to approximate the solution. Neural networks provide a flexible parametric approach based on their universal approximation theoretical results. Given sufficient network capacity, neural networks can represent a wide class of functions, including solutions to PDEs like the pricing equation. However, challenges remain in finding the optimal network parameters to recover the true solution robustly. While existence is guaranteed, the non-convex optimization of neural networks does not assure convergence to global optimality. Care must be taken in specifying the network architecture, loss function, regularization, and training methodology to promote the learning of the pricing function. Subject to these caveats, neural networks present a promising computational technique for approximating smooth pricing solutions, circumventing discretization of the domain. We note the limitations of using neural networks as function approximators. However, their generalization capabilities provide a methodology for data-driven extraction of the pricing mapping across the entire domain. This avoids the constraints of methods like grid-based techniques that rely on local consistency. Future work should explore neural network training enhancements and theoretical guarantees to ensure robust solutions.

Stories, News & Mentions about HIDDENLAYER

Categories

Trending Topics

Stories, News & Mentions about HIDDENLAYER

Hiddenlayer’s Stories on HackerNoon

Hiddenlayer's News & Mentions

Categories

Trending Topics