A team dedicated to making an elite business education more affordable and accessible

Section4

This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Pascal Baumann, Max Planck Institute for Software Systems (MPI-SWS), Germany; (2) Rupak Majumdar, Max Planck Institute for Software Systems (MPI-SWS), Germany; (3) Ramanathan S. Thinniyam, Max Planck Institute for Software Systems (MPI-SWS), Germany; (4) Georg Zetzsche, Max Planck Institute for Software Systems (MPI-SWS), Germany. Table of Links Introduction Dynamic Networks of Concurret Pushdown Systems (DCPS) Warm-up: Non-Termination Fair Non-Termination From Progressive Runs for VASSB to Reachability Starvation Conclusion, Acknowledgment & References A. Strengthening Fairness to Progressive Runs B. Proofs from Section 4.1 C. Proofs for Section 5 D. Proofs for Section 5.3 E. Proofs for Section 6 B PROOFS FROM SECTION 4.1 Intuitively, the edge in (1) spawns the initial thread, the edges in (2) remove a thread with csnumber 0 when it becomes active for the first time and the edges in (3) convert it to a balloon, the edges in (4) apply the spawns during the 푗th segment of a thread by transferring them from the balloon to the regular places, the edges in (5) and (6) finalize this application process with the edges in (6) handling the special case of the 퐾th segment of a thread, the edges in (7) burst balloons containing no more spawns, and the edges in (8) initialize the application of spawns for a thread with cs-number ≥ 1 that is already being represented by a balloon. Lemma B.1. The DCPS A has an infinite, progressive, K-context switch bounded run iff the VASSB V has an infinite progressive run. Proof. For the only if direction let p be an infinite, progressive, K-context switch bounded run of A. We want to decompose p into parts corresponding to a single thread each. To do this we look at the continuous segments of p where a thread is active, starting from the first configuration with an active thread and ending each segment at a configuration with no active thread, upon which a new segment begins at the very next configuration. We can now group segments together into executions of single threads: • Start with the earliest not yet grouped segment where the active thread has context switch number 0. Note the local configuration that gets added to the bag at the end of this segment. • Take the very next ungrouped segment in p where the active thread at the beginning matches the previously noted local configuration. Again, note the local configuration that gets added to the bag at the end of this segment. • Repeat the previous step until an empty stack occurs. The constructed run of V is also sound: For each segment of p it has a sequence of transitions that make the same state change as that segment, and also apply the same spawns. The latter is due to initially choosing the contents during the creation of a balloon correctly. A token representing the initial thread is also created in the beginning of the run. Since the transitions from one segment to the next require the same resumption rules Δr as in p (see edges (2) and (8)) these are also possible in V. For the if direction we do a very similar construction, but backwards. Starting with an infinite progressive run of V from c0, we decompose it into segments that end in a state in G. Afterwards we group those segments together by matching the local balloon configurations in the same way we did for local thread configurations before. Then by construction of Lt we know that there is a thread execution of type t that makes the same spawns in each segment as the group of one balloon does. Those executions will all terminate after K context switches, guaranteeing progressiveness, and we can interleave them in the same way as the balloon segments to form an infinite progressive run of A.

Context-Bounded Verification of Liveness: B Proofs from Section 4.1

Zoonomia, Vol. I Or, the Laws of Organic Life  by Erasmus Darwin is part of HackerNoon’s Book Blog Post series. You can jump to any chapter in this book here: [LINK TO TABLE OF LINK]. Section I: Laws of Animal Causation SECTION IV. LAWS OF ANIMAL CAUSATION. . The fibres, which constitute the muscles and organs of sense, possess a power of contraction. The circumstances attending the exertion of this power of CONTRACTION constitute the laws of animal motion, as the circumstances attending the exertion of the power of ATTRACTION constitute the laws of motion of inanimate matter. I . The spirit of animation is the immediate cause of the contraction of animal fibres, it resides in the brain and nerves, and is liable to general or partial diminution or accumulation. II . The stimulus of bodies external to the moving organ is the remote cause of the original contractions of animal fibres. III . A certain quantity of stimulus produces irritation, which is an exertion of the spirit of animation exciting the fibres into contraction. IV . A certain quantity of contraction of animal fibres, if it be perceived at all, produces pleasure; a greater or less quantity of contraction, if it be perceived at all, produces pain; these constitute sensation. V . A certain quantity of sensation produces desire or aversion; these constitute volition. VI . All animal motions which have occurred at the same time, or in immediate succession, become so connected, that when one of them is reproduced, the other has a tendency to accompany or succeed it. When fibrous contractions succeed or accompany other fibrous contractions, the connection is termed association; when fibrous contractions succeed sensorial motions, the connexion is termed causation; when fibrous and sensorial motions reciprocally introduce each other, it is termed catenation of animal motions. All these connections are said to be produced by habit, that is, by frequent repetition. These laws of animal causation will be evinced by numerous facts, which occur in our daily exertions; and will afterwards be employed to explain the more recondite phænomena of the production, growth, diseases, and decay of the animal system. VII About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Darwin, Erasmus, 2005. Zoonomia, Vol. Or, the Laws of Organic Life. Urbana, Illinois: Project Gutenberg. Retrieved May 2022 from https://www.gutenberg.org/files/15707/15707-h/15707-h.htm#sect_IV This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org, located at https://www.gutenberg.org/policy/license.html.

Section 4 - Laws of Animal Causation

Performance Analysis of Diverse Hateful Meme Detection Datasets

Analyzing Audience Responses to Ukrainian Memetic Warfare Tactics

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
Authors:
(1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr;
(2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction
Organization and notation
Problem Setting and Preliminaries
Generalized Wiener algebra and Fréchet derivative
Characterization of coefficients
Coefficients from an ergodic process
Conclusion & References 2. Organization and notation. This article is organized as follows: In the next section, we formulate the problem by defining the energy and partition function for the hidden PDF in a generalized Wiener algebra. We observe that the nonlinearity of energy functions poses challenges in maximizing likelihood functions. To overcome this issue, we adopt the KLdivergence between two distribution functions. In Section 4, we embed energy functions into a generalized Wiener algebra, thereby transforming the nonlinearity of the energy function into a system of equations in the function space and simultaneously converting the min-max problem into a problem of solving equations. We showthe Fréchet differentiability of the KL-divergence with respect to coefficients in the generalized Wiener algebra. From this property, we then uniquely obtain the existence such that its energy function yields a KL-divergence of 0. We establish a characterization of the coefficients of the energy function by expressing them in relation to the Fourier coefficients found within the (classical) Wiener algebra, as demonstrated in Section 5. This serves as an example of the generalized Wiener algebra. In the concluding Section 6, when dealing with a random sample that consists of realizations of a stationary ergodic process, we formulate a system of polynomial series with the coefficients of the energy function as variables, which consists of infinitely many equations in infinitely many variables. From the convergence property of the partial sums of an energy function in Section 4, the truncated system, consisting of a finite number of equations with partial sums, offers an avenue for coefficient approximations. The estimated coefficients effectively serve as estimators for the coefficients. More precisely, the hidden PDF can be approximated using these estimators. Furthermore, we provide a numerical example that emphasizes the approximation of the PDF, as well as the calculation of the mean and variance from the PDF using a random sample. This sample is generated from a bivariate normal distribution. In this paper, we consistently employ the following general notations (Table 1). Some notations will be elaborated upon in more detail when they are introduced. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: Authors: (1) U Jin Choi, Department of mathematical science, Korea Advanced Institute of Science and Technology & ujchoi@kaist.ac.kr; (2) Kyung Soo Rim, Department of mathematics, Sogang University & ksrim@sogang.ac.kr. Table of Links Introduction Organization and notation Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Coefficients from an ergodic process Conclusion & References Introduction Introduction Organization and notation Organization and notation Problem Setting and Preliminaries Problem Setting and Preliminaries Generalized Wiener algebra and Fréchet derivative Generalized Wiener algebra and Fréchet derivative Characterization of coefficients Characterization of coefficients Coefficients from an ergodic process Coefficients from an ergodic process Conclusion & References Conclusion & References 2. Organization and notation. This article is organized as follows: In the next section, we formulate the problem by defining the energy and partition function for the hidden PDF in a generalized Wiener algebra. We observe that the nonlinearity of energy functions poses challenges in maximizing likelihood functions. To overcome this issue, we adopt the KLdivergence between two distribution functions. In Section 4, we embed energy functions into a generalized Wiener algebra, thereby transforming the nonlinearity of the energy function into a system of equations in the function space and simultaneously converting the min-max problem into a problem of solving equations. We showthe Fréchet differentiability of the KL-divergence with respect to coefficients in the generalized Wiener algebra. From this property, we then uniquely obtain the existence such that its energy function yields a KL-divergence of 0. We establish a characterization of the coefficients of the energy function by expressing them in relation to the Fourier coefficients found within the (classical) Wiener algebra, as demonstrated in Section 5. This serves as an example of the generalized Wiener algebra. In the concluding Section 6, when dealing with a random sample that consists of realizations of a stationary ergodic process, we formulate a system of polynomial series with the coefficients of the energy function as variables, which consists of infinitely many equations in infinitely many variables. From the convergence property of the partial sums of an energy function in Section 4, the truncated system, consisting of a finite number of equations with partial sums, offers an avenue for coefficient approximations. The estimated coefficients effectively serve as estimators for the coefficients. More precisely, the hidden PDF can be approximated using these estimators. Furthermore, we provide a numerical example that emphasizes the approximation of the PDF, as well as the calculation of the mean and variance from the PDF using a random sample. This sample is generated from a bivariate normal distribution. In this paper, we consistently employ the following general notations (Table 1). Some notations will be elaborated upon in more detail when they are introduced.

Variational Non-Bayesian Inference of the Probability Density Function: Organization and Notation

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures ABSTRACT In this work we propose a HyperTransformer, a transformer-based model for fewshot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable. Finally, we extend our approach to a semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance. 1 INTRODUCTION In few-shot learning, a conventional machine learning paradigm of fitting a parametric model to training data is taken to a limit of extreme data scarcity where entire categories are introduced with just one or few examples. A generic approach to solving this problem uses training data to identify parameters φ of a learner aφ that given a small batch of examples for a particular task (called a support set) can solve this task on unseen data (called a query set). One broad family of few-shot image classification methods frequently referred to as metric-based learning, relies on pretraining an embedding eφ(·) and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples. These methods proved effective on numerous benchmarks (see Tian et al. (2020) for review and references), however the capabilities of the learner are limited by the capacity of the architecture itself, as these methods try to build a universal embedding function. On the other hand, optimization-based methods such as seminal MAML algorithm (Finn et al., 2017) can fine-tune the embedding eφ by performing additional SGD updates on all parameters φ of the model producing it. This partially addresses the constraints of metric-based methods by learning a new embedding for each new task. However, in many of these methods, all the knowledge extracted during training on different tasks and describing the learner aφ still has to “fit” into the same number of parameters as the model itself. Such limitation becomes more severe as the target models get smaller, while the richness of the task set increases. In this paper we propose a new few-shot learning approach that allows us to decouple the complexity of the task space from the complexity of individual tasks. The main idea is to use the transformer model (Vaswani et al., 2017) that given a few-shot task episode generates an entire inference model by producing all model weights in a single pass. This allows us to encode the intricacies of the available training data inside the transformer model, while still producing specialized tiny models that can solve individual tasks. Reducing the size of the generated model and moving the computational overhead to the transformer-based weight generator, we can lower the cost of the inference on new images. This can reduce the overall computation cost in cases where the tasks change infrequently and hence the weight generator is only used sporadically. We start by observing that the self-attention mechanism is well suited to be an underlying mechanism for a few-shot CNN weight generator. In contrast with earlier CNN- (Zhao et al., 2020) or BiLSTM-based approaches (Ravi & Larochelle, 2017), the vanilla[1] transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category. Furthermore, we demonstrate that a single-layer self-attention model can replicate a simplified gradient-descent-based learning algorithm. Using a transformer model to generate the logits layer on top of a conventionally learned embedding, we achieve competitive results on several common few-shot learning benchmarks. Varying transformer parameters we demonstrate that this high performance can be attributed to additional capacity of the transformer model that decouples its complexity from that of the generated CNN. We then extend our method to support unlabeled samples by using a special input token that we concatenate to all unlabeled examples encoding the fact that their classes are unknown. In our experiments outlined in Section 4.3, we observe that adding unlabeled samples can significantly improve model performance. Interestingly, the full benefit of using additional data is only realized if the transformers use two or more layers. This result is consistent with the basic mechanism described in Section 3.2, where we show that a transformer model with at least two layers can encode the nearest-neighbor style algorithm that associates unlabeled samples with similar labeled examples. In essence, by training the weight generator to produce CNN models with best possible performance on a query set, we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives. Our approach could be further generalized to treat partially known sample labels (when the true label is known to belong to some set of classes), but this will be the subject of the future work. Finally, we explore the capability of our approach to generate all weights of the CNN model, adjusting both the logits layer and all intermediate layers producing the sample embedding. We show that by generating all layers we can improve both the training and test accuracies[2] of CNN models below a certain size. But, interestingly, generation of the logits layer alone appears to be sufficient above a certain model size threshold (see Figure 3). This threshold is expected to depend on the variability and the complexity of the training tasks. In addition to being able to decouple the complexity of the task distribution from the complexity of individual tasks, another important advantage of our method is that it allows to do learning end to end without relying on complex nested gradients optimization and other meta-learning approaches, where the number of unrolls steps is large. In contrast with these methods, our optimization is done in a single loop of updates to the transformer (and feature extractor) parameters. The paper is structured as follows. In Section 2, we discuss the few-shot learning problem setup and highlight related work. Section 3 introduces our approach, discusses the motivation for choosing an attention-based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms. In Section 4, we discuss our experimental results. Finally, in Section 5, we provide concluding remarks. [1] without attention masking or positional encodings [2] As discussed in Section 4.2, HT with a high training accuracy can be a practical approach to model personalization under the assumption that real tasks come from the distribution seen at the training time. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures ABSTRACT In this work we propose a HyperTransformer, a transformer-based model for fewshot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable. Finally, we extend our approach to a semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance. 1 INTRODUCTION In few-shot learning, a conventional machine learning paradigm of fitting a parametric model to training data is taken to a limit of extreme data scarcity where entire categories are introduced with just one or few examples. A generic approach to solving this problem uses training data to identify parameters φ of a learner aφ that given a small batch of examples for a particular task (called a support set) can solve this task on unseen data (called a query set). One broad family of few-shot image classification methods frequently referred to as metric-based learning, relies on pretraining an embedding eφ(·) and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples. These methods proved effective on numerous benchmarks (see Tian et al. (2020) for review and references), however the capabilities of the learner are limited by the capacity of the architecture itself, as these methods try to build a universal embedding function. On the other hand, optimization-based methods such as seminal MAML algorithm (Finn et al., 2017) can fine-tune the embedding eφ by performing additional SGD updates on all parameters φ of the model producing it. This partially addresses the constraints of metric-based methods by learning a new embedding for each new task. However, in many of these methods, all the knowledge extracted during training on different tasks and describing the learner aφ still has to “fit” into the same number of parameters as the model itself. Such limitation becomes more severe as the target models get smaller, while the richness of the task set increases. In this paper we propose a new few-shot learning approach that allows us to decouple the complexity of the task space from the complexity of individual tasks. The main idea is to use the transformer model (Vaswani et al., 2017) that given a few-shot task episode generates an entire inference model by producing all model weights in a single pass. This allows us to encode the intricacies of the available training data inside the transformer model, while still producing specialized tiny models that can solve individual tasks. Reducing the size of the generated model and moving the computational overhead to the transformer-based weight generator, we can lower the cost of the inference on new images. This can reduce the overall computation cost in cases where the tasks change infrequently and hence the weight generator is only used sporadically. We start by observing that the self-attention mechanism is well suited to be an underlying mechanism for a few-shot CNN weight generator. In contrast with earlier CNN- (Zhao et al., 2020) or BiLSTM-based approaches (Ravi & Larochelle, 2017), the vanilla[1] transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category. Furthermore, we demonstrate that a single-layer self-attention model can replicate a simplified gradient-descent-based learning algorithm. Using a transformer model to generate the logits layer on top of a conventionally learned embedding, we achieve competitive results on several common few-shot learning benchmarks. Varying transformer parameters we demonstrate that this high performance can be attributed to additional capacity of the transformer model that decouples its complexity from that of the generated CNN. We then extend our method to support unlabeled samples by using a special input token that we concatenate to all unlabeled examples encoding the fact that their classes are unknown. In our experiments outlined in Section 4.3, we observe that adding unlabeled samples can significantly improve model performance. Interestingly, the full benefit of using additional data is only realized if the transformers use two or more layers. This result is consistent with the basic mechanism described in Section 3.2, where we show that a transformer model with at least two layers can encode the nearest-neighbor style algorithm that associates unlabeled samples with similar labeled examples. In essence, by training the weight generator to produce CNN models with best possible performance on a query set, we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives. Our approach could be further generalized to treat partially known sample labels (when the true label is known to belong to some set of classes), but this will be the subject of the future work. Finally, we explore the capability of our approach to generate all weights of the CNN model, adjusting both the logits layer and all intermediate layers producing the sample embedding. We show that by generating all layers we can improve both the training and test accuracies[2] of CNN models below a certain size. But, interestingly, generation of the logits layer alone appears to be sufficient above a certain model size threshold (see Figure 3). This threshold is expected to depend on the variability and the complexity of the training tasks. In addition to being able to decouple the complexity of the task distribution from the complexity of individual tasks, another important advantage of our method is that it allows to do learning end to end without relying on complex nested gradients optimization and other meta-learning approaches, where the number of unrolls steps is large. In contrast with these methods, our optimization is done in a single loop of updates to the transformer (and feature extractor) parameters. The paper is structured as follows. In Section 2, we discuss the few-shot learning problem setup and highlight related work. Section 3 introduces our approach, discusses the motivation for choosing an attention-based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms. In Section 4, we discuss our experimental results. Finally, in Section 5, we provide concluding remarks. [1] without attention masking or positional encodings [2] As discussed in Section 4.2, HT with a high training accuracy can be a practical approach to model personalization under the assumption that real tasks come from the distribution seen at the training time.

HyperTransformer: Abstract and Introduction

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Zheyu Oliver Wang, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and olivrw@mit.edu;
(2)  Ricardo Baptista, Computing + Mathematical Sciences, California Institute of Technology, Pasadena, CA and rsb@caltech.edu;
(3) Youssef Marzouk, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and ymarz@mit.edu;
(4) Lars Ruthotto, Department of Mathematics, Emory University, Atlanta, GA and lruthotto@emory.edu;
(5) Deepanshu Verma, Department of Mathematics, Emory University, Atlanta, GA and deepanshu.verma@emory.edu. Table of Links Abstract & Introduction
Background and Related Work
Partially Convex Potential Maps (PCP-Map) for Conditional OT
Conditional OT flow (COT-Flow)
Implementation and Experimental Setup
Numerical Experiments
Discussion and Reference 3. Partially Convex Potential Maps (PCP-Map) for Conditional OT. Our first approach, PCP-Map, approximately solves the static COT problem through maximum likelihood training over a set of partially monotone maps. Motivated by the structure of transport maps that are optimal with respect to the quadratic cost function, we parameterize these maps as the gradients of scalar-valued neural networks that are strictly convex in x for any choice of the weights. Training problem. Given samples from the joint distribution, our algorithm seeks a conditional generator g by solving the maximum likelihood problem The objective JNLL is the expected negative log-likelihood functional which agrees up to an additive constant with the negative logarithm of (2.1), and M is the set of maps that are monotonically increasing in their first argument, i.e., We note that JNLL also agrees (up to an additive constant) with the Kullback-Leibler (KL) divergence from the push forward of ρz through the generator g to the conditional distribution in expectation over the conditioning variable. Since the learning problem in (3.2) only involves the inverse generator g −1, we seek this map directly in the same space of monotone functions M in (3.3). This avoids inverting the generator during training. The drawback is that sampling the target conditional requires inverting the learned map. In this work, we will find the conditional generator g −1 (·, y) that pushes forward the conditional distribution π(x|y) to the reference distribution for each y. Limiting the search to monotone maps is motivated by the celebrated Brenier’s theorem, which ensures there exists a unique monotone map g −1 such that g −1 (·, y)♯π(x|y) = g(·, y) ♯π(x|y) = ρz among all maps written as the gradient of a convex potential; see [8] for the original result and [10] for conditional transport maps. Theorem 2.3 in [10] also shows that g −1 is optimal in the sense that among all maps that match the distributions, it minimizes the integrated L2 transport costs Neural Network Representation. In this work, we leverage the structural form of the conditional Brenier map by using partially input convex neural networks (PICNNs) [2] to express the inverse generator directly. In particular, we parameterize g −1 as the gradient of a PICNN G˜ θ : R n×R m → R that depends on weights θ. A PICNN is a feed-forward neural network that is specifically designed to ensure convexity in some of its inputs; see the original work that introduced this neural network architecture in [2] and its use for generative modeling in [20]. To the best of our knowledge, investigating if PICNNs are universal approximators of partially input convex functions is still an open issue, also see [23, Section IV], that is beyond the scope of our paper; a perhaps related result for fully input convex neural networks is given in [20, Appendix C]. To ensure the monotonicity of g −1, we construct G˜ to be strictly convex as a linear combination of a PICNN and a positive definite quadratic term. That is, where γ1, γ2, and γ3 are scalar parameters that are re-parameterized via the soft-plus function ψ(x) = log(1 + exp(x)) and the ReLU function σReLU(x) = max{0, x} to ensure strict convexity of G˜ θ. Here, wK is the output of an K-layer PICNN and is computed through forward propagation through the layers k = 0, . . . , K − 1 starting with the inputs v0 = y and w0 = x Using properties for the composition of convex functions [18], it can be verified that the forward propagation in (3.6) defines a function that is convex in x, but not necessarily in y (which is not needed), as long as σ (w) is convex and non-decreasing. To compute the log determinant of g −1, we use vectorized automatic differentiation to obtain the Hessian of G˜ θ with respect to its first input and then compute its eigenvalues. This is feasible when the Hessian is moderate; e.g., in our experiments, it is less than one hundred. We use efficient implementations of these methods that parallelize the computations over all the samples in a batch. Our algorithm enforces the non-negativity constraint by projecting the parameters into the non-negative orthant after each optimization step using ReLU. Thereby, we alleviate the need for re- parameterization, for example, using the softplus function in [2]. Another novelty introduced in PICNN is that we utilize trainable affine layer parameters L (v) k and a context feature width u as a hyperparameter to increase the expressiveness of the conditioning variables, which are pivotal to characterizing conditional distributions; existing works such as [20] set L (v) 1:K−2 = I. Sample generation. Due to our neural network parameterization, there is generally no closedform relation between g and g −1 = G˜ θ. As in [20], we approximate the inverse of G˜ θ during sampling as the Legendre-Fenchel dual. That is, we solve the convex optimization problem Due to the strict convexity of G˜ θ in its first argument, the first-order optimality conditions gives Hyperparameters. In our numerical experiments, we vary only three hyperparameters to adjust the complexity of the architecture. As described in section 5 we randomly sample the depth, K, the feature width, w, and the context width, u, from the values in Table 2. This paper is available on arxiv under CC 4.0 license. Authors: (1) Zheyu Oliver Wang, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and olivrw@mit.edu; (2)  Ricardo Baptista, Computing + Mathematical Sciences, California Institute of Technology, Pasadena, CA and rsb@caltech.edu; (3) Youssef Marzouk, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and ymarz@mit.edu; (4) Lars Ruthotto, Department of Mathematics, Emory University, Atlanta, GA and lruthotto@emory.edu; (5) Deepanshu Verma, Department of Mathematics, Emory University, Atlanta, GA and deepanshu.verma@emory.edu. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Zheyu Oliver Wang, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and olivrw@mit.edu; (2)  Ricardo Baptista, Computing + Mathematical Sciences, California Institute of Technology, Pasadena, CA and rsb@caltech.edu; (3) Youssef Marzouk, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA and ymarz@mit.edu; (4) Lars Ruthotto, Department of Mathematics, Emory University, Atlanta, GA and lruthotto@emory.edu; (5) Deepanshu Verma, Department of Mathematics, Emory University, Atlanta, GA and deepanshu.verma@emory.edu. Table of Links Abstract & Introduction Background and Related Work Partially Convex Potential Maps (PCP-Map) for Conditional OT Conditional OT flow (COT-Flow) Implementation and Experimental Setup Numerical Experiments Discussion and Reference Abstract & Introduction Abstract & Introduction Background and Related Work Background and Related Work Partially Convex Potential Maps (PCP-Map) for Conditional OT Partially Convex Potential Maps (PCP-Map) for Conditional OT Conditional OT flow (COT-Flow) Conditional OT flow (COT-Flow) Implementation and Experimental Setup Implementation and Experimental Setup Numerical Experiments Numerical Experiments Discussion and Reference Discussion and Reference 3. Partially Convex Potential Maps (PCP-Map) for Conditional OT. Our first approach, PCP-Map, approximately solves the static COT problem through maximum likelihood training over a set of partially monotone maps. Motivated by the structure of transport maps that are optimal with respect to the quadratic cost function, we parameterize these maps as the gradients of scalar-valued neural networks that are strictly convex in x for any choice of the weights. Training problem. Given samples from the joint distribution, our algorithm seeks a conditional generator g by solving the maximum likelihood problem The objective JNLL is the expected negative log-likelihood functional which agrees up to an additive constant with the negative logarithm of (2.1), and M is the set of maps that are monotonically increasing in their first argument, i.e., We note that JNLL also agrees (up to an additive constant) with the Kullback-Leibler (KL) divergence from the push forward of ρz through the generator g to the conditional distribution in expectation over the conditioning variable. Since the learning problem in (3.2) only involves the inverse generator g −1, we seek this map directly in the same space of monotone functions M in (3.3). This avoids inverting the generator during training. The drawback is that sampling the target conditional requires inverting the learned map. In this work, we will find the conditional generator g −1 (·, y) that pushes forward the conditional distribution π(x|y) to the reference distribution for each y. Limiting the search to monotone maps is motivated by the celebrated Brenier’s theorem, which ensures there exists a unique monotone map g −1 such that g −1 (·, y)♯π(x|y) = g(·, y) ♯π(x|y) = ρz among all maps written as the gradient of a convex potential; see [8] for the original result and [10] for conditional transport maps. Theorem 2.3 in [10] also shows that g −1 is optimal in the sense that among all maps that match the distributions, it minimizes the integrated L2 transport costs Neural Network Representation. In this work, we leverage the structural form of the conditional Brenier map by using partially input convex neural networks (PICNNs) [2] to express the inverse generator directly. In particular, we parameterize g −1 as the gradient of a PICNN G˜ θ : R n×R m → R that depends on weights θ. A PICNN is a feed-forward neural network that is specifically designed to ensure convexity in some of its inputs; see the original work that introduced this neural network architecture in [2] and its use for generative modeling in [20]. To the best of our knowledge, investigating if PICNNs are universal approximators of partially input convex functions is still an open issue, also see [23, Section IV], that is beyond the scope of our paper; a perhaps related result for fully input convex neural networks is given in [20, Appendix C]. To ensure the monotonicity of g −1, we construct G˜ to be strictly convex as a linear combination of a PICNN and a positive definite quadratic term. That is, where γ1, γ2, and γ3 are scalar parameters that are re-parameterized via the soft-plus function ψ(x) = log(1 + exp(x)) and the ReLU function σReLU(x) = max{0, x} to ensure strict convexity of G˜ θ. Here, wK is the output of an K-layer PICNN and is computed through forward propagation through the layers k = 0, . . . , K − 1 starting with the inputs v0 = y and w0 = x v0 = y w0 = x Using properties for the composition of convex functions [18], it can be verified that the forward propagation in (3.6) defines a function that is convex in x, but not necessarily in y (which is not needed), as long as σ (w) is convex and non-decreasing. To compute the log determinant of g −1, we use vectorized automatic differentiation to obtain the Hessian of G˜ θ with respect to its first input and then compute its eigenvalues. This is feasible when the Hessian is moderate; e.g., in our experiments, it is less than one hundred. We use efficient implementations of these methods that parallelize the computations over all the samples in a batch. Our algorithm enforces the non-negativity constraint by projecting the parameters into the non-negative orthant after each optimization step using ReLU. Thereby, we alleviate the need for re- parameterization, for example, using the softplus function in [2]. Another novelty introduced in PICNN is that we utilize trainable affine layer parameters L (v) k and a context feature width u as a hyperparameter to increase the expressiveness of the conditioning variables, which are pivotal to characterizing conditional distributions; existing works such as [20] set L (v) 1:K−2 = I. Sample generation. Due to our neural network parameterization, there is generally no closedform relation between g and g −1 = G˜ θ. As in [20], we approximate the inverse of G˜ θ during sampling as the Legendre-Fenchel dual. That is, we solve the convex optimization problem Due to the strict convexity of G˜ θ in its first argument, the first-order optimality conditions gives Hyperparameters. In our numerical experiments, we vary only three hyperparameters to adjust the complexity of the architecture. As described in section 5 we randomly sample the depth, K, the feature width, w, and the context width, u, from the values in Table 2.

SECTION4

section4.com

-- employees

Since n.d.

COMPANY RANKING

SECTION4

EVERGREEN INDEX #9965

Representing this company? Click here to claim and customize this page!

Section4 WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SECTION4

section4.com

-- employees

Since n.d.

COMPANY RANKING

SECTION4

EVERGREEN INDEX #9965

RELATED COMPANIESRANK

STORIES ABOUT SECTION4

Representing this company? Click here to claim and customize this page!

Section4 WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!