Sigmoid is a Leading Data Solutions company offering best-in-class services in Data Engineering and Data Science

Sigmoid

sigmoid

Animate your web applications with a customizable ease-out. Ease-out animations are great for creating animations that feel like they respond instantly but still cruise to a gentle halt. Most easings specify one of a small number of easing curves: easeOutQuad, easeOutSine, easeOutCubic, etc. However, the sharpness of that curve is not configurable. Here I show how to create a configurable ease-out function that will work for animating any property you desire. How to Use It The easing function takes a single parameter , that specifies the steepness of the curve. The larger is, the steeper the curve, and the higher the velocity in the middle section relative to the beginning and ending. k k The will produce a function that takes a parameter that represents the percentage of the animation completed as a number between 0 and 1. It’s important to note that this system only produces numbers that you can use within an animation framework (that you might have to build). An example is included in the . easeOutFactory t framework jsFiddle Here is an example of the usage: // LISTING 1 (Javascript)// easeOutFactory produces a function representing an ease-out. var easing = easeOutFactory(4); // choose k = 4, see Fig. 2 easing(0.00); // 0easing(0.10); // 0.2047403251776069easing(0.25); // 0.4793609299265755easing(0.50); // 0.7900128291929867easing(0.75); // 0.9389236079466523easing(0.90); // 0.9821358147987899easing(1.00); // 1 You can your easing curve visually using this . Modify to adjust the steepness. design OS X Grapher file k A demonstration of four different Rienzi Gokea. k values implemented in the engine of mobile game Epsilon Jump ( iOS , Android ) graciously provided by The Theory This section is not required for use. Read on to understand the techniques underlying the code so that you will be able to develop your own easing curves. The basic idea is to take the ease-in-out curve developed in and use the right half of it to get our adjustable ease-out. Ease In, Out: The Sigmoid Factory At the conclusion of The Sigmoid Factory, we had this equation developed to describe a sigmoid centered at the origin: Equation 1: An instrumented sigmoid centered at x = 0. α is our knob to tune the steepness, and t is our animation progression from 0 to 1. Figure 1: The centered sigmoid function from Eqn. 2. For our animations, we constrain the useful region of the function to be between 0 and 1 on the time ( )and distance ( ) axes. Notice how the plotted function is substantially less than 1 at = 1 in Fig. 1. We can apply the same scaling trick as we used in The Sigmoid Factory to bump it up to 1. That is to say, we normalize the function by its value at = 1. x y x x Because we’re not shifting the function around, just scaling it, the math works out much more simply than before. Equation 2: The ease out equation. s(t, k) is as specified in Eqn. 1. t is time, k is a steepness constant. Figure 2: The Ease Out Function from Eqn. 2 plotted as k = 4. You can find the . grapher file here I’ll leave Ease-In as an exercise to the reader. Think about how to flip things around. 😘 Hint: Versus Traditional Ease-Outs There is a slight deficiency with this method versus easeOutQuad and easeOutCubic. Figure 3: easeOutCubic (black) and easeOutQuad (gray) You’ll notice that at = 1, both of these eases have zero vertical velocity. That is to say, their d_y_/d_x_ at 1 is 0. The adjustable sigmoid ease described here gets quite close to zero starting around = 6 or 7. If you’re using a below this range, it might look a bit nicer to use one of the traditional eases. x k k Table 1: Approximate Corresponding Eases EASING           EQN            K APPROX. sin(2πt/4)     3.25 -t^2 + 2t      3.50 (t-1)^3 + 1    5.50 (t-1)^5 + 1    8.00 easeOutSine easeOutQuad easeOutCubic easeOutQuint If you enjoyed this article, please follow me on Medium. You might also like The Sigmoid Factory , The Bounce Factory and The Spring Factory . References — Where the theory for the ease-in-out curve used in this article was developed. Ease In, Out: The Sigmoid Factory — A collection of many commonly used easing curves. http://easings.net/ — A Grapher file I developed to design my ease out functions. Ease Out Grapher File

Ease Out: The Half Sigmoid

Animate your web applications with a customizable ease-in-out. It’s very common for animations to be specified as ease-in-out. It’s a very pleasing sensation to witness an object speed up, cruise, and slow to a halt. Most easings specify one of a small number of easing curves: easeInOutQuad, easeInOutSine, easeInOutCubic, etc. However, the sharpness of that curve is not configurable. Here I show how to create a configurable ease-in-out function that will work for animating any property you desire. How to Use It The easing function takes a single parameter , that specifies the steepness of the curve. The larger is, the steeper the curve, and the higher the velocity in the middle section relative to the beginning and ending. k k The will produce a function that takes a parameter that represents the percentage of the animation completed as a number between 0 and 1. It’s important to note that this system only produces numbers that you can use within an animation framework (that you might have to build). An example framework is included in the . sigmoidFactory t jsFiddle Here is an example of the usage: var easing = sigmoidFactory(7); easing(0); // nearly 0 (5.551115123125783e-17)easing(1); // 1easing(0.5); // 0.5easing(0.25); // 0.028453023879735584easing(0.75); // 0.9715469761202644 You can your easing curve visually using this . Modify to adjust the steepness. design OS X Grapher file k A demonstration of four different Rienzi Gokea. k values implemented in the engine of mobile game Epsilon Jump ( iOS , Android ) graciously provided by The Theory This section is not required for use. Read on to understand the techniques underlying the code so that you will be able to develop your own easing curves. Firstly, we need a function that looks like . There are two that come to mind that fit the bill: (Fig. 1, which we will use) and [1] (which we will not use). easeInOut sigmoid arctangent Figure 1. The Sigmoid Function However, these curves both have some defects we’ll have to remedy. We want our animations to start at 0 and end at 1. Both functions begin in the middle of the at 0, and asymptotically approach their final values (0 and 1 in the case of sigmoid, -π/2 to π/2 in the case of arctangent). animation We’ll need to apply a number of fixes to our chosen function in order to make it ready for service. Move the center point to 0.5 along the x axis. Instrument a steepness knob. Force the function to cross 0 at 0, and 1 at 1. I’ve selected the sigmoid curve for our fixes because it’s based on which is generally easier to analyze than arctangent, though it’s nearly as easy to work with in this case. That exercise will be left to the reader. e Equation 1. The Sigmoid Function Moving the sigmoid is quite easy. Using an algebraic manipulation, if our easing function is , then moving the center point by 0.5 is accomplished by . f(t) f(t-0.5) Similarly, if we multiply by a chosen knob, , we can make the function “accelerate and condense” if > 1, or make it “slow and spread out” if < 1. t f(kt) k k Equation 2. Sigmoid augmented with a knob k and centered at t = 0.5. In Eqn. 2, we move the manipulations of into the function in order to make the function easier to use. This solves our first two problems, but take a look at the graph below where Eqn. 2 has been plotted with = 6. t k Figure 2. Eqn. 2 plotted with k = 6. Note that the ends are significantly above 0 and below 1. This is pretty close! However, the beginning and ending of the animation is respectively above 0 and below 1. This will cause an animation to exhibit a “jump” at the ends. If k is set sufficiently high (above about 11 or 12), you won’t even notice the problem. However, this restricts our range of expression as we can’t use any shallow curves, but it might be good enough for you and you can stop here if you so choose. While your first intuition, as it was mine, might be to find some way to make the sigmoid pass though zero by doing some trick like multiplying by or using some crazy conditional statements or reflections, there is a simpler way that’s kind of a hack. Notice that the error at the ends of the sigmoid is symmetric — the distance from 0 is the same as the distance from 1. What if we could just scale the result so that 0 passes through 0 and 1 passes through 1? t There are several steps to get this strategy to work, but in brief, you start with a basic sigmoid centered at 0 (so the range is -1/2 to 1/2), multiply by the correction factor (0.5 / f(1)), then figure out how to translate it up (+0.5) and to the right (t-0.5). You then condense the range of the function by 2 (it reaches -0.5 at -1 and +0.5 at 1). Equation 3. Evolved from Eqn. 2, this is our instrumented sigmoid centered at 0. Equation 4. The fully developed ease in out equation. s(t, α) comes from Eqn. 3. The first term is the correction factor, the second term is our sigmoid, and the third term moves the curve into position. Figure 3. Equation 4 graphed at k = 3. You can find the . grapher file here If you enjoyed this article, please follow me on Medium. You might also like The Bounce Factory and The Spring Factory , and others on Analytic Animations . Endnotes Here is a picture of arctangent: Inverse (Arc)tangent 2. In a discussion on , the function was brought to my attention. It has derivatives of 0 at 1 and 0 and is commonly used in shaders. Reddit smoothstep Figure 4. Smoothstep References — A collection of many commonly used easing curves. http://easings.net/ — A Grapher file I developed to design my easing functions. Ease In Out Graph

Ease In, Out: The Sigmoid Factory

Binary Classification: Understanding Activation and Loss Functions with a PyTorch Example

The brain of a human child is spectacularly amazing. Even in any previously unknown situation, the brain makes a decision based on its primal knowledge. Depending on the outcome, it learns and remembers the most optimal choices to be taken in that particular scenario. On a high level, this process of learning can be understood as a ’trial and error’ process, where the brain tries to maximise the occurrence of positive outcomes. What is Reinforcement Learning? Similar is the inception of Reinforcement Learning. An ideal machine is like a child’s brain, that can remember each and every decision taken in given tasks. Likewise, the goal is to try and optimise the results. In Reinforcement Learning, the learner isn’t told which action to take, but is instead made to try and discover actions that would yield maximum reward. In the most interesting and challenging cases, actions may not only affect the immediate reward, but also impact the next situation and all subsequent rewards. These two characteristics: ‘trial and error search’ and ‘delayed reward’ are the most distinguishing features of reinforcement learning. Machines vs Humans Many of us must have heard about the famous Alpha Go, built by Google using Reinforcement Learning. This machine has even beaten the world champion Lee Sudol in the abstract strategy board game of Go! Elon Musk in a famous debate on AI with Jack Ma, explained how machines are becoming smarter than humans. Reinforcement Learning is definitely one of the areas where machines have already proven their capability to outsmart humans. The Video Games Analogy Reinforcement Learning can be understood by an example of video games. A typical video game usually consists of: Fig: A Video Game Analogy of Reinforcement Learning An (player) who moves around doing stuffAn that the agent exists in (map, room). An that the agent takes (moves upward one space, sells cloak). A that the agent acquires (coins, killing other players). A that the agent currently exists in (on a particular square of a map, part of a room). A that the agent may have (level up, getting as many rewards as possible) agent environment action reward state goal The agent basically runs through sequences of state-action pairs in the given environment, observing the rewards that result, to figure out the best path for the agent to take in order to reach the . goal Rules of Reinforcement Learning The Markov decision process lays the foundation stone for Reinforcement Learning and formally describes an observable environment. There are two important parts of Reinforcement Learning: This is a function that maps a given state to probabilities of selecting each possible action from that state. In simple terms, it identifies the best probable set of actions an agent should take in order to maximize their reward. This policy is learnt by taking random decisions and then iterating backwards (updating weights) once there is a positive or negative end result. Policy Learning: These are functions of states, or of state-action pairs, that estimate how good is it for an agent to be in a given state, or how good is it for the agent to perform a given action in a given state. Unlike Policy Learning, Q-Learning takes two inputs: state and action, and returns a value for each pair. If you’re at an intersection, Q-Learning will tell you the expected value of each action that your agent could take (left, right, etc.). Value Functions (Q Learning): Applications of Reinforcement Learning There are numerous application areas of Reinforcement Learning. Starting from robotics and games to self-driving cars, Reinforcement Learning has found applications in many areas. Famous researchers in the likes of Andrew Ng, Andrej Karpathy and David Silverman are betting big on the future of Reinforcement Learning. Final Words It seems till date that the idea of outsmarting humans in every field is farfetched. But the seed has been sown and companies like Google and Tesla have shown that if machines and humans work together, the future has many opportunities to offer. As far as Reinforcement Learning is concerned, we at Sigmoid are excited about its future and its game changing applications. About Abhijeet is a Data Scientist at . He mainly works in the domain of Recommendation Engines, Time Series Forecasting, Reinforcement Learning and Computer Vision. Sigmoid Previously published here .

Reinforcement Learning: A Brief Introduction to Rules and Applications

In this post, we will talk about the Universal approximation theorem and we will also prove the theorem graphically. This is a follow-up post of my previous post on . Sigmoid Neuron Citation Note: The content and the structure of this article is based on the deep learning lectures from One-Fourth Labs — Padhai . Sigmoid Neuron Before we go into the actual discussion of Universal approximation theorem, I will give a brief recap on working of Sigmoid Neuron and how it will handle non-linearly separable data. A sigmoid neuron is similar to the perceptron neuron, for every input it has weight associated with the input. The weights indicate the importance of the input in the decision-making process. The output from the sigmoid is not 0 or 1 like the perceptron model instead it is a real value between 0–1 which can be interpreted as a probability. The most commonly used sigmoid function is the logistic function, which has a characteristic of an “ ” shaped curve. xi wi S Sigmoid Neuron Representation Handling Non-Linear Data From my previous post on , we have seen that even though we have introduced non-linear Sigmoid it is still not able to effectively separate ones and zeroes. The important point is that from a rigid decision boundary in , we have taken our first step in the direction of creating a decision boundary that works well for non-linearly separable data. Hence the sigmoid neuron is the building block of a deep neural network. Sigmoid Neuron perceptron Modeling Complex Relations In real life, we deal with complex functions where the relationship between input and output might be complex, unlike the simple sigmoid neuron or perceptron. Complex Functions How do we even come up with such complex functions? It’s hard to come up with such functions there could be infinite possibilities for such a function, we need a simple approach. To solve this problem, let's take an analogy of building a house. You can think of the house as a complex function, the way we will build a house is we start with a building a block i.e... brick in this case. First, we will lay the foundation and on top of that we will lay another layer and we will just keep continuing in this manner till we get very complex output i.e… house. In this process, we are combining the very simple building blocks in an effective way so that we can get this house. Modeling Complex Functions In our case instead of bricks and houses, we are interested in complex functions. The building block in building complex functions is sigmoid neuron. The way we are going to create complex functions is that we will combine the sigmoid neurons in an effective way so that there will be an interaction between the neurons present at different layers. So that the predicted output is a complex function of inputs and there will be weights associated across multiple layers. Now the question arises, how do we decide how many layers to use, how many neurons in each layer etc… these are valid questions but the discussion on these questions is out of the scope of this article. Now I need to prove to you that the output from the complex function created using the network of neurons is very close to the true relationship between input and the output. Deep neural network with a certain number of hidden layers should be able to approximate any function that exits between input and output. says:that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of , under mild assumptions on the activation function — Wikipedia Universal approximation theorem R In simple terms, UAT says that — you can always come up with a deep neural network that will approximate any complex relation between input and output. Proof of Universal Approximation Theorem In this section, we will see the illustrative proof of Universal Approximation theorem. For this illustrative proof, we will consider a simple example where we have one-dimensional input and output , the graph shown below represents the true relationship between input and the output. Let’s assume that I don’t know what that function equation between and and I can’t come up with a representation for the equation. x y x y True relationship between x and y To solve this problem, I will break this function into multiple smaller parts so that each part is represented by a simpler function. By combining the series of smaller functions (rectangular bars/towers) I can approximate the relation between and as close to the true relationship possible. x y Combination of simpler functions The key point to note in this case is that I don’t have to worry about coming up with complex equations to represent the relationship between input and output. I can just come up with a simple function and use the combination of these functions to approximate the relationship to my true relationship. The more functions that I choose in this method the better will be my approximation. How do we come up with these rectangles/towers and how it will tie back to the sigmoid neuron? Let’s take two sigmoid functions having a very steep slope and notice that they have a different place at which they peak. The left sigmoid peaks just before zero and right sigmoid peaks just after zero. If I just subtract these two functions the net effect is going to be a tower (rectangular output). If we can get the series of these towers, then we can approximate any true function between input and output. Neural Network Can we come up with a neural network to represent this operation of subtracting one sigmoid function from another?. Neural network for subtraction if we have an input and it is passed through the two sigmoid neurons and the output from these two neurons are combined in another neuron with weights +1 and -1 i.e… same is as subtracting these two outputs, then we will get our tower. Now you can see that we have our building block ready which is a connection of three sigmoid neurons. If we can construct many such building blocks and add all of them up, we can approximate any complex true relationship between input and output. This is called the , to approximate any kind of relationship between input and output. x representational power of deep neural networks Photo by on Sergey Pesterev Unsplash Conclusion In this post, we briefly looked at the working of sigmoid neuron and its ability to handle non-linear data. We then looked at how can we model complex relationships and saw the formal definition of Universal Approximation theorem and then went on to see the illustrative proof of Universal Approximation theorem. Recommended Reading _The building block of the deep neural networks is called the sigmoid neuron._towardsdatascience.com Sigmoid Neuron — Deep Neural Networks _Read writing from Niranjan Kumar in Hacker Noon. Intern at HSBC Analytics. ML and DL Enthusiast. Writer at…_hackernoon.com Niranjan Kumar - Hacker Noon In my next post, we will discuss the feed forward neural works in detail for both regression and classification tasks. Stay tuned for that. is working as an intern at HSBC Analytics division. He is passionate about deep learning and AI. He is one of the top writers at in . Connect with me on or follow me on for updates about upcoming articles on deep learning and Artificial Intelligence. Niranjan Kumar Medium Artificial Intelligence LinkedIn twitter

Illustrative Proof of Universal Approximation Theorem

Neural Network for Valuing Bitcoin: Numerical Results, Implementation and Discussion

Understanding and Generating Dialogue between Characters in Stories: Limitations and References

Authors:
(1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria;
(2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria;
(3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria;
(4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria;
(5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA;
(6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany;
(7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Related Work MemeGraphs Benchmarking Analysis Conclusion, Acknowledgments, and References 4 Benchmarking 4.1 Models To evaluate the results of our automatic augmentation and how it can affect the performance of hateful memes classification, we employed a text-based Transformer with and without the MemeGraphs information as well as a multimodal model. Previous work has shown that using only the text of the memes for classification gives results highly competitive with multimodal methods [14, 13, 28]. On the other hand, unimodal image-based models have lowest results. We use a pre-trained BERT [9] model that takes as input the text of each meme (TxtBERT) [14] and different variants of this approach that include the MemeGraphs input. In order to feed the graphs as input to the model, we represent them as a text sequence (see Section 3), which is given as an extra text input after the [SEP] token (Fig. 3). In the model that we call MemeGraphs[SceneGr], this sequence contained only the scene graphs, which were represented by triplets of the detected objects and the relations between them (Eq. 2), e.g., “0-man has 11-eye. 0-man wearing 12-shirt.”. In the model called MemeGraphs[Know], the text descriptions from the knowledge base corresponding to the detected entities are added as extra input (Eq. 3). For example, the second input sequence for the meme shown in Figure 3 will be “American politician, businessman, and 29th Governor of New Mexico.”. While in the MemeGraphs[SceneGr+Know] model, information from the whole knowledge graphs is added, comprising the scene graphs and the background knowledge concatenated with a full stop into one sequence (see Subsection 3.3). In all the models, the [CLS] token is fed into a final linear layer with a sigmoid activation function that produced the probability of the meme being hateful. In order to compare our MemeGraphs approach with a method that employs learned visual representations, we use ImgBERT, an early fusion multimodal model (see Section 2). 4.2 Experimental Setup Each meme was labeled in the original dataset as offensive or non-offensive. The classification was based on the offensiveness labels that exist in the MultiOFF dataset. The dataset is slightly imbalanced with 42% of the samples being offensive. For training and testing the models, we used the split provided by the authors of the dataset. The split consisted of training, validation, and test sets, which contained 445, 149, and 149 memes respectively. We employed the pre-trained BERT model provided by Hugging Face and fine-tuned it on our dataset.[10] To obtain image embeddings, we experimented with several pre-trained CNN models and based on the best results we chose DenseNet with 161 layers. The embeddings were extracted from the last pooling layer. The max length was set as the average token length of the training texts, depending on the input of each model. The models were trained using batch size 16, the weighted binary cross entropy loss, and AdamW optimizer with an initial learning rate of 2e-5. We used early stopping based on the validation loss with a patience of 3 epochs. We trained all models 20 times with different seeds and the training lasted between 4 and 6 epochs in each run. During inference, a threshold was used to determine if a meme is hateful or not based on the produced probability. This threshold was set to 0.5 for all the models. 4.3 Benchmarking results For each model, we obtained predictions for the test set from all twenty differently initialized runs. We evaluated each prediction set by calculating the F1 score defining the minority class (offensive class) as positive. In Table 1, we report the average F1 score over the twenty runs for each method and the corresponding standard error. In Table 2, the test score of the model that achieved the best score on the development set is shown. We observe that MemeGraphs[SceneGr+Know] outperforms the other models when predicting with the best checkpoint (Table 2). On the other hand, when looking at the average, the model with only the background knowledge as input achieves the best F1 score (MemeGraphs[Know]). We generally observe that the methods that incorporate our automatic augmentation (MemeGraphs) outperform the simple text-based fine-tuned BERT. Also, ImgBERT is outperformed showing that when the visual information is employed in the form of scene graphs the performance is improved, compared to when using only the image embeddings. [10] https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForSequenceClassification This paper is available on arxiv under CC 4.0 license. Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Authors: Authors: (1) Vasiliki Kougia, University of Vienna, Faculty of computer science, Vienna, Austria & UniVie Doctoral School Computer Science, Vienna, Austria; (2) Simon Fetze, University of Vienna, Faculty of computer science, Vienna, Austria; (3) Thomas Kirchmair, University of Vienna, Faculty of computer science, Vienna, Austria; (4) Erion Çano, University of Vienna, Faculty of computer science, Vienna, Austria; (5) Sina Moayed Baharlou, Boston University, Department of Electrical and Computer Engineering, Boston, MA, USA; (6) Sahand Sharifzadeh, Ludwig Maximilians University of Munich, Faculty of Computer Science, Munich, Germany; (7) Benjamin Roth, University of Vienna, Faculty of computer science, Vienna, Austria. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work MemeGraphs MemeGraphs Benchmarking Benchmarking Analysis Analysis Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References 4 Benchmarking 4.1 Models To evaluate the results of our automatic augmentation and how it can affect the performance of hateful memes classification, we employed a text-based Transformer with and without the MemeGraphs information as well as a multimodal model. Previous work has shown that using only the text of the memes for classification gives results highly competitive with multimodal methods [14, 13, 28]. On the other hand, unimodal image-based models have lowest results. We use a pre-trained BERT [9] model that takes as input the text of each meme (TxtBERT) [14] and different variants of this approach that include the MemeGraphs input. In order to feed the graphs as input to the model, we represent them as a text sequence (see Section 3), which is given as an extra text input after the [SEP] token (Fig. 3). In the model that we call MemeGraphs[SceneGr], this sequence contained only the scene graphs, which were represented by triplets of the detected objects and the relations between them (Eq. 2), e.g., “0-man has 11-eye. 0-man wearing 12-shirt.”. In the model called MemeGraphs[Know], the text descriptions from the knowledge base corresponding to the detected entities are added as extra input (Eq. 3). For example, the second input sequence for the meme shown in Figure 3 will be “American politician, businessman, and 29th Governor of New Mexico.”. While in the MemeGraphs[SceneGr+Know] model, information from the whole knowledge graphs is added, comprising the scene graphs and the background knowledge concatenated with a full stop into one sequence (see Subsection 3.3). In all the models, the [CLS] token is fed into a final linear layer with a sigmoid activation function that produced the probability of the meme being hateful. In order to compare our MemeGraphs approach with a method that employs learned visual representations, we use ImgBERT, an early fusion multimodal model (see Section 2). 4.2 Experimental Setup Each meme was labeled in the original dataset as offensive or non-offensive. The classification was based on the offensiveness labels that exist in the MultiOFF dataset. The dataset is slightly imbalanced with 42% of the samples being offensive. For training and testing the models, we used the split provided by the authors of the dataset. The split consisted of training, validation, and test sets, which contained 445, 149, and 149 memes respectively. We employed the pre-trained BERT model provided by Hugging Face and fine-tuned it on our dataset.[10] To obtain image embeddings, we experimented with several pre-trained CNN models and based on the best results we chose DenseNet with 161 layers. The embeddings were extracted from the last pooling layer. The max length was set as the average token length of the training texts, depending on the input of each model. The models were trained using batch size 16, the weighted binary cross entropy loss, and AdamW optimizer with an initial learning rate of 2e-5. We used early stopping based on the validation loss with a patience of 3 epochs. We trained all models 20 times with different seeds and the training lasted between 4 and 6 epochs in each run. During inference, a threshold was used to determine if a meme is hateful or not based on the produced probability. This threshold was set to 0.5 for all the models. 4.3 Benchmarking results For each model, we obtained predictions for the test set from all twenty differently initialized runs. We evaluated each prediction set by calculating the F1 score defining the minority class (offensive class) as positive. In Table 1, we report the average F1 score over the twenty runs for each method and the corresponding standard error. In Table 2, the test score of the model that achieved the best score on the development set is shown. We observe that MemeGraphs[SceneGr+Know] outperforms the other models when predicting with the best checkpoint (Table 2). On the other hand, when looking at the average, the model with only the background knowledge as input achieves the best F1 score (MemeGraphs[Know]). We generally observe that the methods that incorporate our automatic augmentation (MemeGraphs) outperform the simple text-based fine-tuned BERT. Also, ImgBERT is outperformed showing that when the visual information is employed in the form of scene graphs the performance is improved, compared to when using only the image embeddings. [10] https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForSequenceClassification This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Comparing Meme Analysis Methods: MemeGraphs vs. ImgBERT

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Ghazaleh H. Torbati, Max Planck Institute for Informatics Saarbrucken, Germany & ghazaleh@mpi-inf.mpg.de;
(2) Andrew Yates, University of Amsterdam Amsterdam, Netherlands & a.c.yates@uva.nl;
(3) Anna Tigunova, Max Planck Institute for Informatics Saarbrucken, Germany & tigunova@mpi-inf.mpg.de;
(4) Gerhard Weikum, Max Planck Institute for Informatics Saarbrucken, Germany & weikum@mpi-inf.mpg.de. Table of Links Abstract and Introduction
Related Work
Methodology
Experimental Design
Experimental Results
Conclusion
Ethics Statement and References III. METHODOLOGY A. System Architecture The CUP framework is based on a two-tower architecture for representation learning (one ”tower” for users, the other for items, following the prevalent architecture in neural information retrieval with query and document/passage encodings). The two towers are jointly trained, coupled by the shared loss function and ground truth. Figure 1 shows a pictorial overview. User reviews and items descriptions are fed into BERT followed by a feed-forward network to learn latent representations. Downstream, the vectors are simply compared by a dot product for scores that indicate whether the user likes an item or not. Importantly, unlike prior works, we allow the top-most layer of BERT itself to be fine-tuned as part of the end-to-end training process. The per-item text usually comprises book titles, tags like categories or genre labels, which can be coarse (e.g., “thriller”) or fine-grained (e.g., “Scandinavian crime noir”), and a short description of the book contents. The per-user text can comprise the titles and tags of her training-set books and the entirety of her review texts, which vary widely in length and informativeness, hence the need for smart text selection. In the following, we present the CUP training procedure in Subsection III-B, the test/deployment-time inference in Subsection III-C, the judicious selection of input text snippets in Subsection III-D, and the selection of negative training samples in Subsection III-E. B. Training Another optional mechanism is to process multiple chunks of bounded-length text, which could be of interest for long user reviews. In this case, the per-user representations are learned for each chunk, and there is an additional max-pooling step between the FFN and the dot product, to combine the per-chunk vectors. We use this option for comparison in experiments. As it incurs m times higher computational and energy costs, with m being the number of chunks, we do not favor it. A user-item pair is classified with score calculated by with <, > denoting dot product and σ for the sigmoid function. We use the Adam optimizer to minimize the binary cross entropy loss between predicted labels and the ground truth with sampled negatives. During training we update the topmost layer of BERT, which allows end-to-end training of all components. This is an important difference to earlier works with similar architectures, which use frozen BERT to encode user and item representations. C. Inference Prediction for Ranking. At test time, a prediction is made for user-item pairs. We encode the item description by running it through the trained network, and we compare it to the already learned user vector, which is based on the user’s training-time reviews (none for the given test item). The scores for different test items, computed by the final dot product, yield the ranking of the candidate items. This is a very efficient computation, following established practice in neural information retrieval [37]. Search-based Recommendation. In a deployed system (as opposed to lab experiments with test samples), a typical usage mode would be search-based re-ranking: a user provides context with a tag-based query or an example of a specific liked item, which can be thought of as query-by-example. The user’s expectation is to see a ranked list of recommended items that are similar to her positive sample (as opposed to recommendations from all kinds of categories). The system achieves this by first computing approximate matches to the query item (i.e., similarity-search neighbors), and then re-ranking a shortlist of say top-100 candidates. The CUP framework supports this mode, by using a light-weight BM25 retrieval model, querying all unlabeled items with the category and textual description of the positive point at hand, and keeping the top-100 highest scoring matches. D. Coping with Long and Noisy Texts For constructing user profiles from text, the simplest idea would be to concatenate all available reviews into a long token sequence. Two problems arise, though. User reviews are a noisy mix of descriptive elements (e.g., “the unusual murder weapon”), sentiment expressions (e.g., “it was fun to read”) and personal but irrelevant statements (e.g., “I read only on weekends”). Only the first aspect is helpful for content-based profiling (as the sentiment is already captured by the fact that the user likes the book). Second, the entirety of user-provided text can be too long to be fully digested by the Transformer. Even when it would fit into the token budget, the computational and energy cost is quadratic in the number of input tokens. Therefore, we tightly limit the tokens for each user’s text profile to 128, and devise a suite of light-weight techniques for judiciously selecting the most informative pieces. Variants of our methodology are to create concise profiles by generative language models upfront (e.g., using T5 or ChatGPT), or to digest multiple chunks of text with downstream aggregation by max-pooling or averaging. We include these techniques in our experimental comparisons, but emphasize that both of these alternatives have much higher computational cost and climate footprint. Our techniques for selecting the most informative parts of user reviews to construct concise profiles, are as follows: • Weighted Phrases: selected unigrams, bigrams or trigrams, ordered by descending tf-idf weights, where tf is the frequency of the phrase in all of the given user’s reviews, and idf is pre-computed on Google books n-grams to capture informativeness. • Weighted Sentences: selected sentences, ordered by descending idf weights, where a sentence’s total weight is the sum of the per-word idf weights normalized by sentence length. • Similar Sentences: selected sentences, ordered by descending similarity scores computed via Sentence-BERT [38] for comparing the user-review sentences against the sentences of the item description. To ensure that the selected set is not dominated by a single item, the sentences are picked from different items in a round-robin manner. • ChatGPT-generated Profiles: feeding all reviews of a user, in large chunks, into ChatGPT and instructing it to characterize the user’s book interests with a few short keyphrases. The output size is limited per chunk with a total budget of 128 tokens, to enforce concise profiles. • T5-generated Keywords: using a T5 model fine-tuned for keyword generation, to cast each user’s review text into a set of keywords, concatenated to create the profile. We give examples of selected profile constructions for a Goodreads and an Amazon user in Table VII. An online demo of the CUP framework is accessible on our website[2]. E. Coping with Unlabeled Data A challenge for training in the data-poor regime is how to handle the extreme skew between positive samples and unlabeled data points for sparse users. The crux in many recommender applications is that there are extremely few, if any, explicitly negative samples, such as books rated with low marks. This holds also for the datasets in this work. Therefore, we introduce and experiment with two different techniques to construct negative training samples from unlabeled data: • Uniform random samples. Under the closed world assumption (CWA), aka Selected Completely At Random, negative training points are sample uniformly from all unlabeled data. This is the baseline technique, widely used in prior works. • Weighted pos-neg samples. Prior works on PU learning [39], with positive and unlabeled data and without explicitly negative samples, is largely based on treating unlabeled points as pairs of samples, one positive and one negative with fractional weights. The weights can be based on (learned estimates of) class priors, but the extreme skew in our data renders these techniques ineffective. Instead we leverage the fact that relatedness measures between item pairs can be derived from tagging features or interaction data. We compute item-item relatedness via basic matrix factorization of the user-item matrix for the entire dataset. The relatedness of two items is set to the scalar product between their latent vectors, re-scaled for normalization between 0 and 1. Then, each originally unlabeled sample is cloned, with one instance positive with weight proportional to its average relatedness to the user’s explicitly positive points. The negative clone’s weight is set to the complement. [2] https://sirup.mpi-inf.mpg.de/ This paper is available on arxiv under CC 4.0 license. Authors: (1) Ghazaleh H. Torbati, Max Planck Institute for Informatics Saarbrucken, Germany & ghazaleh@mpi-inf.mpg.de; (2) Andrew Yates, University of Amsterdam Amsterdam, Netherlands & a.c.yates@uva.nl; (3) Anna Tigunova, Max Planck Institute for Informatics Saarbrucken, Germany & tigunova@mpi-inf.mpg.de; (4) Gerhard Weikum, Max Planck Institute for Informatics Saarbrucken, Germany & weikum@mpi-inf.mpg.de. This paper is available on arxiv under CC 4.0 license. Authors: (1) Ghazaleh H. Torbati, Max Planck Institute for Informatics Saarbrucken, Germany & ghazaleh@mpi-inf.mpg.de; (2) Andrew Yates, University of Amsterdam Amsterdam, Netherlands & a.c.yates@uva.nl; (3) Anna Tigunova, Max Planck Institute for Informatics Saarbrucken, Germany & tigunova@mpi-inf.mpg.de; (4) Gerhard Weikum, Max Planck Institute for Informatics Saarbrucken, Germany & weikum@mpi-inf.mpg.de. Table of Links Abstract and Introduction Related Work Methodology Experimental Design Experimental Results Conclusion Ethics Statement and References Abstract and Introduction Abstract and Introduction Related Work Related Work Methodology Methodology Experimental Design Experimental Design Experimental Results Experimental Results Conclusion Conclusion Ethics Statement and References Ethics Statement and References III. METHODOLOGY A. System Architecture A. System Architecture The CUP framework is based on a two-tower architecture for representation learning (one ”tower” for users, the other for items, following the prevalent architecture in neural information retrieval with query and document/passage encodings). The two towers are jointly trained, coupled by the shared loss function and ground truth. Figure 1 shows a pictorial overview. User reviews and items descriptions are fed into BERT followed by a feed-forward network to learn latent representations. Downstream, the vectors are simply compared by a dot product for scores that indicate whether the user likes an item or not. Importantly, unlike prior works, we allow the top-most layer of BERT itself to be fine-tuned as part of the end-to-end training process. The per-item text usually comprises book titles, tags like categories or genre labels, which can be coarse (e.g., “thriller”) or fine-grained (e.g., “Scandinavian crime noir”), and a short description of the book contents. The per-user text can comprise the titles and tags of her training-set books and the entirety of her review texts, which vary widely in length and informativeness, hence the need for smart text selection. In the following, we present the CUP training procedure in Subsection III-B, the test/deployment-time inference in Subsection III-C, the judicious selection of input text snippets in Subsection III-D, and the selection of negative training samples in Subsection III-E. B. Training B. Training Another optional mechanism is to process multiple chunks of bounded-length text, which could be of interest for long user reviews. In this case, the per-user representations are learned for each chunk, and there is an additional max-pooling step between the FFN and the dot product, to combine the per-chunk vectors. We use this option for comparison in experiments. As it incurs m times higher computational and energy costs, with m being the number of chunks, we do not favor it. A user-item pair is classified with score calculated by with <, > denoting dot product and σ for the sigmoid function. We use the Adam optimizer to minimize the binary cross entropy loss between predicted labels and the ground truth with sampled negatives. During training we update the topmost layer of BERT, which allows end-to-end training of all components. This is an important difference to earlier works with similar architectures, which use frozen BERT to encode user and item representations. C. Inference C. Inference Prediction for Ranking. At test time, a prediction is made for user-item pairs. We encode the item description by running it through the trained network, and we compare it to the already learned user vector, which is based on the user’s training-time reviews (none for the given test item). The scores for different test items, computed by the final dot product, yield the ranking of the candidate items. This is a very efficient computation, following established practice in neural information retrieval [37]. Prediction for Ranking. Search-based Recommendation. In a deployed system (as opposed to lab experiments with test samples), a typical usage mode would be search-based re-ranking: a user provides context with a tag-based query or an example of a specific liked item, which can be thought of as query-by-example. The user’s expectation is to see a ranked list of recommended items that are similar to her positive sample (as opposed to recommendations from all kinds of categories). The system achieves this by first computing approximate matches to the query item (i.e., similarity-search neighbors), and then re-ranking a shortlist of say top-100 candidates. The CUP framework supports this mode, by using a light-weight BM25 retrieval model, querying all unlabeled items with the category and textual description of the positive point at hand, and keeping the top-100 highest scoring matches. D. Coping with Long and Noisy Texts D. Coping with Long and Noisy Texts For constructing user profiles from text, the simplest idea would be to concatenate all available reviews into a long token sequence. Two problems arise, though. User reviews are a noisy mix of descriptive elements (e.g., “the unusual murder weapon”), sentiment expressions (e.g., “it was fun to read”) and personal but irrelevant statements (e.g., “I read only on weekends”). Only the first aspect is helpful for content-based profiling (as the sentiment is already captured by the fact that the user likes the book). Second, the entirety of user-provided text can be too long to be fully digested by the Transformer. Even when it would fit into the token budget, the computational and energy cost is quadratic in the number of input tokens. Therefore, we tightly limit the tokens for each user’s text profile to 128, and devise a suite of light-weight techniques for judiciously selecting the most informative pieces. Variants of our methodology are to create concise profiles by generative language models upfront (e.g., using T5 or ChatGPT), or to digest multiple chunks of text with downstream aggregation by max-pooling or averaging. We include these techniques in our experimental comparisons, but emphasize that both of these alternatives have much higher computational cost and climate footprint. Our techniques for selecting the most informative parts of user reviews to construct concise profiles, are as follows: • Weighted Phrases: selected unigrams, bigrams or trigrams, ordered by descending tf-idf weights, where tf is the frequency of the phrase in all of the given user’s reviews, and idf is pre-computed on Google books n-grams to capture informativeness. • Weighted Sentences: selected sentences, ordered by descending idf weights, where a sentence’s total weight is the sum of the per-word idf weights normalized by sentence length. • Similar Sentences: selected sentences, ordered by descending similarity scores computed via Sentence-BERT [38] for comparing the user-review sentences against the sentences of the item description. To ensure that the selected set is not dominated by a single item, the sentences are picked from different items in a round-robin manner. • ChatGPT-generated Profiles: feeding all reviews of a user, in large chunks, into ChatGPT and instructing it to characterize the user’s book interests with a few short keyphrases. The output size is limited per chunk with a total budget of 128 tokens, to enforce concise profiles. • T5-generated Keywords: using a T5 model fine-tuned for keyword generation, to cast each user’s review text into a set of keywords, concatenated to create the profile. We give examples of selected profile constructions for a Goodreads and an Amazon user in Table VII. An online demo of the CUP framework is accessible on our website[2]. E. Coping with Unlabeled Data E. Coping with Unlabeled Data A challenge for training in the data-poor regime is how to handle the extreme skew between positive samples and unlabeled data points for sparse users. The crux in many recommender applications is that there are extremely few, if any, explicitly negative samples, such as books rated with low marks. This holds also for the datasets in this work. Therefore, we introduce and experiment with two different techniques to construct negative training samples from unlabeled data: • Uniform random samples. Under the closed world assumption (CWA), aka Selected Completely At Random, negative training points are sample uniformly from all unlabeled data. This is the baseline technique, widely used in prior works. Uniform random samples. • Weighted pos-neg samples. Prior works on PU learning [39], with positive and unlabeled data and without explicitly negative samples, is largely based on treating unlabeled points as pairs of samples, one positive and one negative with fractional weights. The weights can be based on (learned estimates of) class priors, but the extreme skew in our data renders these techniques ineffective. Instead we leverage the fact that relatedness measures between item pairs can be derived from tagging features or interaction data. We compute item-item relatedness via basic matrix factorization of the user-item matrix for the entire dataset. The relatedness of two items is set to the scalar product between their latent vectors, re-scaled for normalization between 0 and 1. Then, each originally unlabeled sample is cloned, with one instance positive with weight proportional to its average relatedness to the user’s explicitly positive points. The negative clone’s weight is set to the complement. [2] https://sirup.mpi-inf.mpg.de/

Recommendations by Concise User Profiles from Review Text: Methodology

Loss of Amazon Rainforest Resilience: Materials and Methods

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
Authors:
(1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr;
(2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction
Organization and notation
Problem Setting and Preliminaries
Generalized Wiener algebra and Fréchet derivative
Characterization of coefficients
Coefficients from an ergodic process
Conclusion & References Conclusion In this mathematical study, we delve into the realm of statistical inference and introduce a novel approach to variational non-Bayesian inference. Most significantly, we propose a new method for uniquely determining the hidden PDF solely from a random sample while leveraging theory. Beyond merely approximating the shape of the unrevealed PDF, we yield results that provide practical assistance in inferring important moments, such as the mean and variance, from the estimated PDF. These days, methods utilizing artificial intelligence are widely employed, but they generally exhibit a drawback of depending on initial conditions, i.e., a prior distribution, and iterative computations, i.e., a backpropagation. The limitations of such approaches lie in their reliance on these conditions to achieve global optimization. We emphasize the significant contribution of the proposed method to predictive and classification models, even without any information on populations, highlighting its potential applications in various domains such as finance, economics, weather forecasting, and machine learning, all of which present unique challenges. The most well-known method for approximating hidden PDFs is the variational approach in a Bayesian context. In contrast, our study extends this problem by seeking to precisely determine unknown PDFs through a system of equations. Our approach includes proving the Fréchet differentiability of entropy to establish the uniqueness of the energy function space in the Wiener algebra. We then derive the unique determination of the energy function through the minimization of KL-divergence. Leveraging the Ergodic theorem, we elucidate that solutions to equations comprising polynomial function series are the coefficients of the energy function and numerically substantiate the convergence of partial sums of the energy function obtained from a finite number of equations. In summary, our mathematical exploration has unveiled the potential of variational Non-Bayesian inference in Wiener space. We anticipate that these mathematical ideas can offer an innovative framework for probability density estimation and predictive modeling. Ultimately, our research emphasizes its potential contribution to reshaping mathematical horizons and expanding the boundaries of knowledge in the field of statistical methodology. REFERENCES [1] K. Bharath A. Saha and S. Kurtek, A geometric variational approach to bayesian inference, Journal of the American Statistical Association 115 (2020), no. 530, 822–835. [2] P. Alquier and J. Ridgway, Concentration of tempered posteriors and of their variational approximations, The Annals of Statistics 48 (2020), no. 3, 1475–1497. [3] A. R. Barron and C. Sheu, Approximation of density functions by sequences of exponential families, The Annals of Statistics 19 (1991), no. 3, 1347–1369. [4] C. M. Bishop and M. E. Tipping, Probabilistic principal component analysis, J. R. Statist. Soc. B 61 (1999), no. 3, 611–622. [5] F. W. Olver, D. Lozier, R. F. Boisvert and C. W. Clark, NIST Handbook of Mathematical Functions, Cambridge University Press, 2010. [6] C Brouder and F Patras, One-particle irreducibility with initial correlations, (2011). [7] I. Csiszar, I-divergence geometry of probability distributions and minimization problems, Ann. Probab. 3 (1975), no. 1, 146–158. [8] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2 (1989), 303–314. [9] L. Grafakos, Classical and Modern Fourier Analysis, Prentice Hall, 2003. [10] D. E. Rumelhart, G. E. Hinton and R. Williams, Learning representations by back-propagating errors, Nature 323 (1986), 533–536. [11] G. E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, University of Toronto, 2010. [12] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), no. 2, 251–257. [13] Y. Katznelson, An introduction to harmonic analysis (third ed.), New York: Cambridge Mathematical Library, 2004. [14] D. M. Blei, A. Kucukelbir and J. D. McAuliffe, Variational inference: A review for statisticians, Jour. of the American Statistical Association 112 (2017), no. 518, 859–877. [15] S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statistics 22 (1951), no. 1, 79–86. [16] H. M. Möller and R. Tenberg, Multivariate polynomial system solving using intersections of eigenspaces, Journal of Symbolic Computation 32 (2001), no. 5, 513–531. [17] B. Mourrain, V. Y. Pan and O. Ruatta, Accelerated solution of multivariate polynomial systems of equations, SIAM Journal on Computing 32 (2003), no. 2, 435–454. [18] W. Rudin, Functional Analysis, McGraw-Hill, 1991. [19] D. W. Stroock, Probability theory, an analytic view, Cambridge University Press, 1993. [20] B. Sturmfels, Solving systems of polynomial equations (cbms regional conference series in mathematics), Amer. Math. Soc., 2002. [21] N. Wiener, Tauberian theorems, Annals of Mathematics 33 (1932), no. 1, 1–100. [22] F. Zhang and C. Gao, Convergence rates of variational posterior distributions, The Annals of Statistics 48 (2020), no. 4, 2180–2207. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction Organization and notation Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Coefficients from an ergodic process Conclusion & References Introduction Introduction Organization and notation Organization and notation Problem Setting and Preliminaries Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Characterization of coefficients Coefficients from an ergodic process Coefficients from an ergodic process Conclusion & References Conclusion & References Conclusion In this mathematical study, we delve into the realm of statistical inference and introduce a novel approach to variational non-Bayesian inference. Most significantly, we propose a new method for uniquely determining the hidden PDF solely from a random sample while leveraging theory. Beyond merely approximating the shape of the unrevealed PDF, we yield results that provide practical assistance in inferring important moments, such as the mean and variance, from the estimated PDF. These days, methods utilizing artificial intelligence are widely employed, but they generally exhibit a drawback of depending on initial conditions, i.e., a prior distribution, and iterative computations, i.e., a backpropagation. The limitations of such approaches lie in their reliance on these conditions to achieve global optimization. We emphasize the significant contribution of the proposed method to predictive and classification models, even without any information on populations, highlighting its potential applications in various domains such as finance, economics, weather forecasting, and machine learning, all of which present unique challenges. The most well-known method for approximating hidden PDFs is the variational approach in a Bayesian context. In contrast, our study extends this problem by seeking to precisely determine unknown PDFs through a system of equations. Our approach includes proving the Fréchet differentiability of entropy to establish the uniqueness of the energy function space in the Wiener algebra. We then derive the unique determination of the energy function through the minimization of KL-divergence. Leveraging the Ergodic theorem, we elucidate that solutions to equations comprising polynomial function series are the coefficients of the energy function and numerically substantiate the convergence of partial sums of the energy function obtained from a finite number of equations. In summary, our mathematical exploration has unveiled the potential of variational Non-Bayesian inference in Wiener space. We anticipate that these mathematical ideas can offer an innovative framework for probability density estimation and predictive modeling. Ultimately, our research emphasizes its potential contribution to reshaping mathematical horizons and expanding the boundaries of knowledge in the field of statistical methodology. REFERENCES [1] K. Bharath A. Saha and S. Kurtek, A geometric variational approach to bayesian inference, Journal of the American Statistical Association 115 (2020), no. 530, 822–835. [2] P. Alquier and J. Ridgway, Concentration of tempered posteriors and of their variational approximations, The Annals of Statistics 48 (2020), no. 3, 1475–1497. [3] A. R. Barron and C. Sheu, Approximation of density functions by sequences of exponential families, The Annals of Statistics 19 (1991), no. 3, 1347–1369. [4] C. M. Bishop and M. E. Tipping, Probabilistic principal component analysis, J. R. Statist. Soc. B 61 (1999), no. 3, 611–622. [5] F. W. Olver, D. Lozier, R. F. Boisvert and C. W. Clark, NIST Handbook of Mathematical Functions, Cambridge University Press, 2010. [6] C Brouder and F Patras, One-particle irreducibility with initial correlations, (2011). [7] I. Csiszar, I-divergence geometry of probability distributions and minimization problems, Ann. Probab. 3 (1975), no. 1, 146–158. [8] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2 (1989), 303–314. [9] L. Grafakos, Classical and Modern Fourier Analysis, Prentice Hall, 2003. [10] D. E. Rumelhart, G. E. Hinton and R. Williams, Learning representations by back-propagating errors, Nature 323 (1986), 533–536. [11] G. E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, University of Toronto, 2010. [12] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), no. 2, 251–257. [13] Y. Katznelson, An introduction to harmonic analysis (third ed.), New York: Cambridge Mathematical Library, 2004. [14] D. M. Blei, A. Kucukelbir and J. D. McAuliffe, Variational inference: A review for statisticians, Jour. of the American Statistical Association 112 (2017), no. 518, 859–877. [15] S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statistics 22 (1951), no. 1, 79–86. [16] H. M. Möller and R. Tenberg, Multivariate polynomial system solving using intersections of eigenspaces, Journal of Symbolic Computation 32 (2001), no. 5, 513–531. [17] B. Mourrain, V. Y. Pan and O. Ruatta, Accelerated solution of multivariate polynomial systems of equations, SIAM Journal on Computing 32 (2003), no. 2, 435–454. [18] W. Rudin, Functional Analysis, McGraw-Hill, 1991. [19] D. W. Stroock, Probability theory, an analytic view, Cambridge University Press, 1993. [20] B. Sturmfels, Solving systems of polynomial equations (cbms regional conference series in mathematics), Amer. Math. Soc., 2002. [21] N. Wiener, Tauberian theorems, Annals of Mathematics 33 (1932), no. 1, 1–100. [22] F. Zhang and C. Gao, Convergence rates of variational posterior distributions, The Annals of Statistics 48 (2020), no. 4, 2180–2207.

Variational Non-Bayesian Inference of the Probability Density Function: Conclusion & References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Experimental Datasets And Models MovieLens Dataset The MovieLens100K1 (Harper and Konstan 2015) dataset contains 100k ratings of N=1682 films from among M=943 users. The original user ratings run from 0 to 5. Following Geffner and Domke (2022), we binarise the ratings into just likes/dislikes, by taking user-ratings of {0, 1, 2, 3} as a binarised-rating of 0 (dislikes) and user-ratings of {4, 5} as a binarised-rating of 1 (likes). We assume binarised ratings of 0 for films which users have not previously rated. The probabilistic graphical model is given in Fig 3. We use n to index films, and m to index users. Each film has a known feature vector, xn, while each user has a latent weight-vector, zm, of the same length, describing whether or not they like any given feature. There are 18 features, indicating which genre tags the film has (Action, Adventure, Animation, Childrens,...). Each film may have more than one tag. The probability of the user liking a film is given by taking the dot-product of the film’s feature vector with the latent weight-vector, and applying a sigmoid (σ(·)) (final line) Additionally, we have latent vectors for the global mean, µ, and variance, ψ, of the weight vectors. We use a random subset of N = 20 films and M = 450 users for our experiment to ensure high levels of uncertainty. We use equally sized but disjoint subset of users held aside for calculation of the predictive log-likelihood. In Fig. 1c,d we consider the variance and MSE of estimators of zm. Bus Delay Datase In our second experiment, we model the length of delays to New York school bus journeys,2 working with a dataset supplied by the City of New York (DOE 2023). Our goal is to predict the length of a delay, based on the year, y, borough, b, bus company, c and journey type, j. Specifically, our data includes the years 2015 − 2022 inclusive and covers the five New York boroughs (Brooklyn, Manhatten, The Bronx, Queens, Staten Island) as well as some surrounding areas (Nassau County, New Jersey, Connecticut, Rockland County, Westchester). There are 57 bus companies, and 6 different journey types (e.g. pre-K/elementary school route, general education AM/PM route etc.) We take I = 60 delayed buses in each borough and year, and take Y = 3 years and B = 3 boroughs. We then split along the I dimension to get two equally sized train and test sets. Thus, each delay is uniquely identified by the year, y, the borough, b, and the index, i, giving delayybi The delays are recorded as an integer number of minutes and we discard any entries greater than 130 minutes. We have a hierarchical latent variable model describing the impact of each of the features (year, borough, bus company and journey type) on the length of delays. Specifically, the integer delay is modelled with a Negative Binomial distribution, with fixed total count of 130, as we discard entries with delays longer than this. The expected delay length is controlled by a logits latent variable, logitsybi, with one logits for each delayed bus. The logits is a sum of three terms: one for the borough and year jointly, one for the bus company and one for the journey type. Each of these three terms is themselves a latent variable that must be inferred. First, we have a term for the year and borough, YearBoroughWeightyb, which has a hierarchical prior. Specifically, we begin by sampling a global mean and variance, GlobalMean and GlobalVariance. Then for each year, we use GlobalMean and GlobalVariance to sample a mean for each year, YearMeany. Additionally, we sample a variance for each borough, BoroughVarianceb. Then we sample a YearBoroughWeightyb from a Gaussian distribution with a year-dependent mean, YearMeany, and a borough-dependent variance, BoroughVarianceb. Next, the weights for the bus company and journey type are very similar. Specifically, we have one latent weight for each bus company, CompanyWeightc, with c ∈ {1, . . . , 57}, and for each journey type, JourneyTypeWeightj, with j ∈ {1, . . . , 6}. We have a table identifying the bus company, bybi, and journey type, jybi, for each delayed bus journey (remember that a particular delayed bus journey is uniquely identified by the year, y, borough, b and index i). In logitsybi we use these tables to pick out the right company and journey type weight for that particular delayed bus journey, CompanyWeightcybi and JourneyTypeWeight cybi. The final generative model is, and the graphical model is given in Fig. 4 Experiment Details We ran all experiments on an 80GB NVIDIA A100 GPU and averaged the results over three different dataset splits generated using different random seeds. On each dataset split, the MovieLens experiments required a maximum 26.9 GB of memory at any one time. For the NYC Bus Breakdown experiments this value was 0.9 GB. For each experiment using a single dataset  split, we used global IS, MP IS, VI and HMC methods to obtain (for each value of K in the IS methods, or each iteration of VI/HMC) 1000 values of the ELBO, predictive log-likelihood (on a test set that was disjoint from the training set), and posterior mean estimates for each latent variable in the model. These 1000 values were averaged for the ELBO and predictive log-likelihood, whilst the 1000 mean estimates were used to calculate mean squared error or variance per variable, based on whether the observed data had been sampled from the model itself—in which case the true variable values were known—or not. We present error bars in the top rows of Fig. 1 and Fig. 2 representing the standard deviation over the three dataset splits. For VI, we optimize our approximate posterior using Adam with learning rates of 10−2, 10−3 and 10−4 (we found that learning rates faster than 10−2 were unstable), and ran 100 optimization steps for both for the MovieLens and Bus Breakdown models. For HMC, we ran the No U-Turn Sampler on our GPU using JAX (Bradbury et al. 2018) to obtain 10 consecutive samples after a single step with PyMC’s (Salvatier, Wiecki, and Fonnesbeck 2016) pre-tuning and with all other parameters set to their default values. We present results of the first 5 samples (not including the tuning sample) in order to align as best we can with the small time-scale of the IS methods.

Using Autodiff to Estimate Posterior Moments, Marginals and Samples: Experimental Datasets and Model

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ali Ghanbari, Dept. of Computer Science, Iowa State University; (2) Deepak-George Thomas, Dept. of Computer Science, Iowa State University; (3) Muhammad Arbab Arshad, Dept. of Computer Science, Iowa State University; (4) Hridesh Rajan, Dept. of Computer Science, Iowa State University. Table of Links Abstract & Introduction Background Motivating Example Proposed Approach Supported DNN Bugs Evaluation Threats to Validity Related Work Conclusion, Acknowledgments & References II. BACKGROUND A. Mutation Analysis Mutation analysis [18], is a program analysis method for assessing the quality of the test suite. It involves generating a pool of program variants, i.e., mutants, by systematically mutating program elements, e.g., replacing an arithmetic operator with another, and running the test suite against the mutants to check if the output of the mutated program is different from that of the original one; if different, the mutant is marked as killed, otherwise as survived. A mutant might survive because it is semantically equivalent to the original program, hence the name equivalent mutant. Test suite quality is assessed by computing a mutation score for the test suite, which is the ratio of killed mutants over the non-equivalent survived mutants. Mutation score indicates how good a test suite is in detecting real bugs [37]. In addition to its original use, mutation analysis has been used for many other purposes [38], such as fault localization [16], [17], automated program repair [39], [40], test generation [41], [42] and prioritization [43], program verification [44], [45], etc. B. Mutation-based Fault Localization Mutation-based fault localization (MBFL) uses mutation analysis to find bugs. In this section, we review two major approaches to MBFL, namely Metallaxis [30] and MUSE [17]. Both of these approaches are implemented in deepmufl. The reader is referred to the original papers [30], [17] for examples explicating the rationale behind each approach. : Metallaxis [30] posits that mutants generated by mutating the same program element are likely to exhibit similar behaviors and mutants generated by mutating different program elements are likely to exhibit different behaviors. Since a fault itself can also be viewed as a mutant, it is expected to behave similar to other mutants generated by mutating that same buggy program element and can be located by examining the mutants based on this heuristic. Metallaxis assumes that the mutants impacting the test outputs, or their error messages, e.g., stack trace, as impacting the tests. Thus, mutants impacting failing test cases might indicate that their corresponding code elements is the root cause of the test failures, while mutants impacting passing test cases might indicate that their corresponding code elements are correct. 1) Metallaxis Once the number of impacted passing and failing test cases are calculated, Metallaxis uses a fault localization formula to calculate suspiciousness values for each element. Metallaxis fault localization formula can be viewed as an extension to that of spectrum-based fault localization, by treating all mutants impacting the tests as covered elements while the others as uncovered elements. Specifically, the maximum suspiciousness value of the mutants of a corresponding code element is returned as the suspiciousness value of the code element. More concretely, assuming we are using SBI formula [46], suspiciousness value for a program element e, denoted s(e), is calculated as follows. where M(e) denotes the set of all mutants targeting program element e, Tf (m, e) is the set of failing test cases that are impacted by the mutant m, while Tp(m, e) denotes the set of passing test cases that are impacted by m. In this definition, and in the rest of the paper, the notation | · | represents the size of a set. Alternatively, had we used Ochiai [47], Metallaxis suspiciousness formula would be modified as follows. where Tf denotes the set of all failing tests cases. 2) MUSE: MUSE [17] is based on the assumption that mutating a faulty program element is likely to impact more failed test cases than passing test cases by “fixing” it, while mutating a correct program element is likely to impact more passing test cases than failing test cases by breaking it. The notion of “impacting test cases” in MUSE, unlike Metallaxis, is more rigid, in that it refers to making passing test cases fail, vice versa. Once the number of impacted failing and passing test cases are identified, suspiciousness values can be calculated using the following formula. C. Deep Neural Networks A neural network is intended to compute a function of the form R m −→ R n, where m, n are positive integers. A neural network is often represented as a weighted directed acyclic graph arranged in layers of three types, i.e., input layer, one or more hidden layers, and an output layer. Input and output layers output linear combination of their inputs, while hidden layers can be viewed as more complex computational units, e.g., a non-linear unit, convolutional unit, or a batch normalization unit. A non-linear unit is composed of neurons, functions applying a non-linear activation function, e.g., rectified linear unit (ReLU), tanh, or sigmoid, on the weighted sum of their inputs. A convolutional layer, calculates the convolution between the vector of the values obtained from the previous layer and a learned kernel matrix. Lastly, a batch normalization layer, normalizes the vector of values obtained from the previous layer via centering or re-scaling. A neural network with two or more hidden layers is referred to as a deep neural network (DNN).

Mutation-based Fault Localization of Deep Neural Networks: Background

Playing Checkers With Genetic Algorithms: Evolutionary Learning in Traditional Games

জেনেটিক অ্যালগরিদম দিয়ে চেকার খেলা: ঐতিহ্যগত গেমে বিবর্তনীয় শিক্ষা

Dame spielen mit genetischen Algorithmen: Evolutionäres Lernen in traditionellen Spielen

Jugar a las damas con algoritmos genéticos: aprendizaje evolutivo en juegos tradicionales

Jouer aux dames avec des algorithmes génétiques : apprentissage évolutif dans les jeux traditionnels

जेनेटिक एल्गोरिदम के साथ चेकर्स खेलना: पारंपरिक खेलों में विकासवादी शिक्षा

遺伝的アルゴリズムを使用したチェッカーのプレイ: 従来のゲームにおける進化的学習

유전 알고리즘으로 체커 게임하기: 전통 게임의 진화 학습

Jogando Damas com Algoritmos Genéticos: Aprendizagem Evolutiva em Jogos Tradicionais

Игра в шашки с использованием генетических алгоритмов: эволюционное обучение в традиционных играх

Genetik Algoritmalarla Dama Oynamak: Geleneksel Oyunlarda Evrimsel Öğrenme

Chơi cờ đam bằng thuật toán di truyền: Học tập tiến hóa trong các trò chơi truyền thống

用遗传算法玩西洋跳棋：传统游戏中的进化学习

How to Sample From Latent Space With Variational Autoencoder

ভেরিয়েশনাল অটোএনকোডার দিয়ে সুপ্ত স্থান থেকে কীভাবে নমুনা করবেন

So sampeln Sie mit Variational Autoencoder aus dem Latentraum

Cómo muestrear desde el espacio latente con el codificador automático variacional

Comment échantillonner à partir de l'espace latent avec l'encodeur automatique variationnel

वेरिएशनल ऑटोएनकोडर के साथ अव्यक्त स्थान से नमूना कैसे लें

変分オートエンコーダーを使用して潜在空間からサンプリングする方法

Variational Autoencoder를 사용하여 잠재 공간에서 샘플링하는 방법

Como fazer amostragem do espaço latente com codificador automático variacional

Как выполнить выборку из скрытого пространства с помощью вариационного автоэнкодера

Varyasyonel Otomatik Kodlayıcıyla Gizli Uzaydan Örnekleme Nasıl Yapılır?

Cách lấy mẫu từ không gian tiềm ẩn bằng bộ mã hóa tự động biến đổi

如何使用变分自动编码器从潜在空间进行采样

This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Flor Ortiz, University of Luxembour; (2) Juan A. Vasquez-Peralvo, University of Luxembour; (3) Jorge Querol, University of Luxembour; (4) Eva Lagunas, University of Luxembour; (5) Jorge L. Gonzalez Rios, University of Luxembour; (6) Marcele O. K. Mendonc¸a, University of Luxembour; (7) Luis Garces, University of Luxembour; (8) Victor Monzon Baeza, University of Luxembour; (9) Symeon Chatzinotas, University of Luxembou. Table of Links Abstract & Introduction System Model and Problem Statement Antenna Design and Training Data Generation Supervised Learning for Adaptive Beamforming Numerical Results Conclusions, Acknowledgment, and References IV. SUPERVISED LEARNING FOR ADAPTIVE BEAMFORMING The design of our beamforming matrix allows us to divide the 36-element matrix into four distinct 18-element sections for more detail see [8]. Consequently, our goal is reduced to predicting the 18 × 18 matrix, comprising a total of 324 elements. For this study. we present two approaches based on supervised learning: approach 1, based on Multi-Label Classification Neural Network, and approach 2 based on Clustering and Classification Neural Network. Approach 1 employs a multi-label classification neural network for beamforming, as shown in Algorithm 1. This method is designed to predict the activation state of individual elements within the beamforming matrix. The primary goal is determining which elements should be active (assigned a value of 1) and which should be inactive (assigned a value of 0) based on beam-related input features. The neural network takes eight beam-related input features, such as beam width in azimuth, beam width in elevation, minimum SLL in azimuth, minimum SLL in elevation, equivalent isotropic radiated power (EIRPb), azimuth, elevation, and number of elements. The output layer consists of 324 units, corresponding to the 324 elements of the 18times18 section of the beamforming matrix. Each unit in the output layer uses a sigmoid activation function, which produces values between 0 and 1, indicating the probability that each element is active. The loss function used is the binary cross-entropy, which calculates the error of each element individually and then aggregates them to evaluate the overall model performance. Model performance is evaluated by the multi-label accuracy or the f1score, which measures the percentage of correct element activations predicted for each input. The neural network architecture for Approach 1 is constructed accordingly, with input, hidden, and output layers. Once trained, this model can predict the activation state of the elements in the beamforming matrix based on the characteristics of the input beams. Approach 2 employs a two-step process involving clustering and classification to determine the appropriate beamforming matrix as explained in Algorithm 2. Initially, a K-meansbased clustering algorithm is used to group similar sets of input variables related to the beamforming matrix design. This clustering operation transforms the problem into a binary classification scenario, where each class corresponds to a specific predefined beamforming matrix. The neural network used for classification is designed to learn and solve this binary classification task by analyzing the characteristics of the input variables and assigning them to the appropriate pre-trained matrix. This approach aims to optimize the selection of beamforming matrices based on the input variables, improving the system’s overall performance.

Supervised Learning Based Real-Time Adaptive Beamforming On-board: Supervised Learning for Adaptive

Hate Speech Detection in Algerian Dialect Using Deep Learning: Our Methodology

Did you know that in the world of machine learning, the efficiency of Deep Learning (DL) models can be significantly boosted by a technique called quantization? Imagine reducing the computational burden of your neural network without sacrificing its performance. Just like compressing a large file without losing its essence, model quantization allows you to make your models smaller and faster. Let's dive into the fascinating concept of quantization and unveil the secrets of optimizing your neural networks for real-world deployment. Before we dig in, readers should be familiar with neural networks and the basic concept of quantization, including the terms scale (S) and zero point (ZP). For readers who would like a refresher, and explains the broad concept and types of quantization. this article this article In this guide I will briefly explain why quantization matters, and how to implement it using Pytorch. I will focus mainly on the type of quantization called “post training static quantization”, which results in 4x lesser memory footprint of the ML model, and makes the inference up to 4x faster. Concepts Why Does Quantization Matter? Neural Networks computations most commonly performed with 32 bit floating point numbers. A single 32 bit floating point number (FP32) requires 4 bytes of memory. In comparison, a single 8 bit integer number (INT8) only requires 1 byte of memory. Further, computers process integer arithmetic much faster than float operations. Right away, you can see that quantizing a ML model from FP32 to INT8 will result in 4x less memory. Further, it will also speed up inference by as much as 4x! With large models being all the rage right now, it is important for practitioners to be able to optimize trained models for memory and speed for real time inference. Key Terms Weights of the trained neural network. Weights- In terms of quantization, activations are not the activation functions like Sigmoid or ReLU. By activations, I mean the feature map outputs of the intermediate layers, which are inputs to the next layers. Activations- Post Training Static Quantization Post training static quantization means we do not need to train or finetune the model for quantization after training the original model. We also do not need to quantize intermediate layer inputs, called activations on the fly. In this mode of quantization, the weights are directly quantized by computing the scale and zero point for each layer. For activations however, as the input to the model changes, the activations will change as well. We do not know the range of each and every input that the model will encounter during inference. So how can we compute the scale and zero point for all activations of the network? We can do this by calibrating the model, using a good representative dataset. We then observe the range of values of activations for the calibration set, and then use those statistics to compute scale and zero point. This is done by inserting observers in the model, which collect data statistics during calibration. After preparing the model (inserting observers), we run the forward pass of the model on the calibration dataset. The observers use this calibration data to compute scale and zero point for activations. Now inference is only a matter of applying the linear transform to all layers with their respective scale and zero points. While the entire inference is done in INT8, the final model output is dequantized (from INT8 to FP32). Why do activations need to be quantized if the input and network weights are already quantized? This is an excellent question. While the network input and weights are indeed already INT8 values, the output of the layer is stored as INT32, to avoid overflow. To reduce complexity in processing the next layer the activations are quantized from INT32 to INT8. With the concepts clear, let’s dive into the code and see how it works! For this example, I will use a resnet18 model fine-tuned on the Flowers102 dataset, available directly in Pytorch. However, the code will work for any trained CNN, with the appropriate calibration dataset. Since this tutorial is focused on quantization, I will not cover the training and fine-tuning part. However, all code can be found . Lets dive in! here Quantization Code Let us import the libraries necessary for quantization and to load the fine-tuned model. import torch
import torchvision
import torchvision.transforms as transforms
from torchvision.models import resnet18
import torch.nn as nn

from torch.ao.quantization import get_default_qconfig
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx
from torch.ao.quantization import QConfigMapping

import warnings

warnings.filterwarnings('ignore') Next, let us define some parameters, define data transforms and data loaders, and load the fine-tuned model model_path = 'flowers_model.pth'
quantized_model_save_path = 'quantized_flowers_model.pth'
batch_size = 10
num_classes = 102

# Define data transforms
transform = transforms.Compose(
                                [transforms.Resize(256),
                                 transforms.CenterCrop(224),
                                 transforms.ToTensor(),
                                 transforms.Normalize(
                                            (0.485, 0.465, 0.406), 
                                            (0.229, 0.224, 0.225))]
)

# Define train data loader, for using as calibration set
trainset = torchvision.datasets.Flowers102(root='./data', split="train", 
                                           download=True, transform=transform)
trainLoader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, 
                                          shuffle=True, num_workers=2)

# Load the finetuned resnet model
model_to_quantize = resnet18(weights=None)
num_features = model_to_quantize.fc.in_features
model_to_quantize.fc = nn.Linear(num_features, num_classes)
model_to_quantize.load_state_dict(torch.load(model_path))
model_to_quantize.eval()
print('Loaded fine-tuned model') For this example, I will use some training samples as the calibration set. Now, let us define the configuration used for quantizing the model. # Define quantization parameters config for the correct platform, 
# "x86" for x86 devices or "qnnpack" for arm devices
qconfig = get_default_qconfig("x86")
qconfig_mapping = QConfigMapping().set_global(qconfig) In the above snippet, I have used the default config, but the class of Pytorch is used to describe how the model, or a part of the model should be quantized. We can do this by specifying the type of observer classes to be used to weights and activations. QConfig We are now ready to prepare the model for quantization # Fuse conv-> relu, conv -> bn -> relu layer blocks and insert observers
model_prep = prepare_fx(model=model_to_quantize, 
                        qconfig_mapping=qconfig_mapping, 
                        example_inputs=torch.randn((1,3,224,224))) The function inserts the observers into the model, and also fuses conv→relu and conv→bn→relu modules. This results in lesser operations and lower memory bandwidth due to not needing to store intermediate results of those modules. prepare_fx Calibrate the model by running forward pass on calibration data # Run calibration for 10 batches (100 random samples in total)
print('Running calibration')
with torch.no_grad():
    for i, data in enumerate(trainLoader):
        samples, labels = data

        _ = model_prep(samples)

        if i == 10: 
            break We don’t need to run calibration on the entire training set! In this example, I am using 100 random samples, but in practice, you should choose a dataset which is representative of what the model will see during deployment. Quantize the model and save the quantized weights! # Quantize calibrated model
quantized_model = convert_fx(model_prep)
print('Quantized model!')

# Save quantized
torch.save(quantized_model.state_dict(), quantized_model_save_path)
print('Saved quantized model weights to disk') And that’s it! Now let us see how to load a quantized model, and then compare the accuracy, speed and memory footprint of the original and quantized models. Load a quantized model A quantized model graph is not quite the same as the original model, even if both have the same layers. Printing the first layer ( ) of both models shows the difference. conv1 print('\nPrinting conv1 layer of fp32 and quantized model')
print(f'fp32 model: {model_to_quantize.conv1}')
print(f'quantized model: {quantized_model.conv1}') You will notice that along with the different class, the quantized model’s conv1 layer also contains the scale and zero point parameters. Thus, what we need to do it follow the quantization process (without calibration) to create the model graph, and then load the quantized weights. Of course, if we save the quantized model to onnx format, we can load it like any other onnx model, without running the quantization functions each time. In the meantime, let us define a function for loading the quantized model and save it to . inference_utils.py import torch

from torch.ao.quantization import get_default_qconfig
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx
from torch.ao.quantization import QConfigMapping

def load_quantized_model(model_to_quantize, weights_path):

    '''
    Model only needs to be calibrated for the first time.
    Next time onwards, to load the quantized model, 
    you still need to prepare and convert the model without calibrating it.
    After that, load the state dict as usual.
    '''
    model_to_quantize.eval()

    qconfig = get_default_qconfig("x86")
    qconfig_mapping = QConfigMapping().set_global(qconfig)
    model_prep = prepare_fx(model_to_quantize, qconfig_mapping, 
                            torch.randn((1,3,224,224)))
    quantized_model = convert_fx(model_prep)
    quantized_model.load_state_dict(torch.load(weights_path))

    return quantized_model Define functions for measuring accuracy and speed Measure accuracy import torch

def test_accuracy(model, testLoader):
    model.eval()
    running_acc = 0
    num_samples = 0
    with torch.no_grad():
        for i, data in enumerate(testLoader):
            samples, labels = data

            outputs = model(samples)
            preds = torch.argmax(outputs, 1)

            running_acc += torch.sum(preds == labels)
            num_samples += samples.size(0)
    
    return running_acc / num_samples This is a pretty straightforward Pytorch code. Measure inference speed in milliseconds (ms) import torch
from time import time

def test_speed(model):

    dummy_sample = torch.randn((1,3,224,224))

    # Average out inference speed over multiple iterations 
    # to get a true estimate
    num_iterations = 100
    start = time()
    for _ in range(num_iterations):
        _ = model(dummy_sample)

    end = time()
    return (end-start)/num_iterations * 1000 Add both these functions in . We are now ready to compare models. Let us go through the code. inference_utils.py Compare models for accuracy, speed, and size Let us first import necessary libraries, define parameters, data transforms, and the test dataloader. import os
import torch
import torch.nn as nn
import torchvision
from torchvision.models import resnet18
import torchvision.transforms as transforms
from inference_utils import test_accuracy, test_speed, load_quantized_model
import copy

import warnings
warnings.filterwarnings('ignore')

model_weights_path = 'flowers_model.pth'
quantized_model_weights_path = 'quantized_flowers_model.pth'
batch_size = 10
num_classes = 102

# Define data transforms
transform = transforms.Compose(
                                [transforms.Resize(256),
                                 transforms.CenterCrop(224),
                                 transforms.ToTensor(),
                                 transforms.Normalize(
                                            (0.485, 0.465, 0.406), 
                                            (0.229, 0.224, 0.225))]
)

testset = torchvision.datasets.Flowers102(root='./data', split="test", 
                                          download=True, transform=transform)
testLoader = torch.utils.data.DataLoader(testset, batch_size=batch_size, 
                                         shuffle=False, num_workers=2) Load the two models # Load the finetuned resnet model and the quantized model
model = resnet18(weights=None)
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, num_classes)
model.load_state_dict(torch.load(model_weights_path))
model.eval()

model_to_quantize = copy.deepcopy(model)
quantized_model = load_quantized_model(model_to_quantize, 
                                       quantized_model_weights_path) Compare models # Compare accuracy
fp32_accuracy = test_accuracy(model, testLoader)
accuracy = test_accuracy(quantized_model, testLoader)

print(f'Original model accuracy: {fp32_accuracy:.3f}')
print(f'Quantized model accuracy: {accuracy:.3f}\n')

# Compare speed
fp32_speed = test_speed(model)
quantized_speed = test_speed(quantized_model)
print(f'Inference time for original model: {fp32_speed:.3f} ms')
print(f'Inference time for quantized model: {quantized_speed:.3f} ms\n')

# Compare file size
fp32_size = os.path.getsize(model_weights_path)/10**6
quantized_size = os.path.getsize(quantized_model_weights_path)/10**6
print(f'Original model file size: {fp32_size:.3f} MB')
print(f'Quantized model file size: {quantized_size:.3f} MB') Results As you can see, the accuracy of the quantized model on the test data is almost as much as the accuracy of the original model! Inference with the quantized model is ~3.6x faster (!) and the quantized model requires ~4x lesser memory than the original model! Conclusion In this article, we understood the broad concept of ML model quantization, and a type of quantization called Post Training Static Quantization. We also looked at why quantization is important and a powerful tool in the time of large models. Finally, we went through example code to quantize a trained model using Pytorch, and reviewed the results. As the results showed, quantizing the original model did not impact performance, and at the same time decreased inference speed by ~3.6x and lowered the memory footprint by ~4x! A few points to note- Static quantization works well for CNNs, but dynamic quantization is the preferred method for sequence models. Additionally, if quantization drastically impacts model performance, the accuracy can be regained by a technique called Quantization Aware Training (QAT). How do Dynamic Quantization and QAT work? Those are posts for another time. I hope with this guide, you are provided the knowledge to perform static quantization on your own Pytorch models. References Basics of model quantization Tensor quantization Flowers 102 dataset Pytorch docs

Curious About Faster ML Models? Discover Model Quantization With PyTorch!

দ্রুত এমএল মডেল সম্পর্কে আগ্রহী? PyTorch দিয়ে মডেল কোয়ান্টাইজেশন আবিষ্কার করুন!

Neugierig auf schnellere ML-Modelle? Entdecken Sie die Modellquantisierung mit PyTorch!

¿Tiene curiosidad por los modelos de aprendizaje automático más rápidos? ¡Descubra la cuantización de modelos con PyTorch!

Curieux de connaître les modèles ML plus rapides ? Découvrez la quantification de modèles avec PyTorch !

तेज़ एमएल मॉडल के बारे में उत्सुक हैं? PyTorch के साथ मॉडल परिमाणीकरण की खोज करें!

より高速な ML モデルに興味がありますか? PyTorch でモデルの量子化を発見してください!

더 빠른 ML 모델이 궁금하십니까? PyTorch로 모델 양자화를 발견해보세요!

Curioso sobre modelos de ML mais rápidos? Descubra a quantização de modelos com PyTorch!

Хотите узнать о более быстрых моделях машинного обучения? Откройте для себя квантование модели с помощью PyTorch!

Daha Hızlı ML Modellerini Merak Ediyor musunuz? PyTorch ile Model Nicelemeyi Keşfedin!

Bạn tò mò về các mô hình ML nhanh hơn? Khám phá lượng tử hóa mô hình với PyTorch!

对更快的机器学习模型感到好奇吗？使用 PyTorch 探索模型量化！

Stories, News & Mentions about SIGMOID

Sigmoid’s Stories on HackerNoon