WeGive

wegive

Authors:
(1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch);
(2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch);
(3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.2 Anchor Regression 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation B Experiments Abstract We propose a novel algorithm for data augmentation in nonlinear over-parametrized regression. Our data augmentation algorithm borrows from the literature on causality and extends the recently proposed Anchor regression (AR) method for data augmentation, which is in contrast to the current state-of-the-art domain-agnostic solutions that rely on the Mixup literature. Our Anchor Data Augmentation (ADA) uses several replicas of the modified samples in AR to provide more training examples, leading to more robust regression predictions. We apply ADA to linear and nonlinear regression problems using neural networks. ADA is competitive with state-of-the-art C-Mixup solutions. [1] 1 Introduction Data augmentation is one of the key ingredients of any successful application of a machine learning classifier. The first example that typically comes to mind is the in-depth description of the data augmentation in the now-famous Alexnet paper [26]. Data augmentation algorithms come in different flavors, and they mostly rely on the expectation that small perturbations, invariances, or symmetries applied to the input will not change the class label. That way, we can present ‘fresh new’ samples as alterations of the available examples for training. These transformations modify the input distribution to make the algorithm more robust for cases where the distribution of the test set may differ from that of the training set. We refer the reader to the related work section (Section 2.1) for an overview and description of different data augmentation strategies. The literature for data augmentation in regression is slim. The paper on Mixup augmentation [51] proposes a simple and general scheme for data augmentation using convex combinations of samples. The authors only apply their data augmentation proposal to classification problems. They conjecture in the discussion that the application to regression is straightforward, however, this is not the case in practice. Mixup is theoretically analyzed in [5, 52] as a regularization technique for classification and regression problems. However, it is only illustrated in classification problems. The Mixup algorithm has been extended to regression problems in [18, 49], in which the authors explain that Mixup cannot be blindly applied to regression problems. To our knowledge, these are the only two papers in which data augmentation for regression is proposed. RegMix [18] relies on a hard-to-train prior neural network controller before augmenting the data using a Mixup strategy. C-Mixup [49], a method proposed more recently, solves some of the issues limiting the standard Mixup algorithm for regression problems. The authors propose to mix only closeby samples in the output space (i.e., samples which have close enough labels). This strategy is only valid when the target variables are monotonic with the input and is applied in a transformed space. The authors present comprehensive results in data augmentation for in-distribution generalization, task generalization and out-of-distribution robustness. In this paper, we rely on the causality literature to provide a different avenue for augmenting data in regression problems. Causal discovery finds the causes of a response variable among a given set of observations or helps to recognize the causal relations between a set of variables [39]. These causes allow us to understand how these relations will change if we were to intervene in a subset of the (input) variables or what would be the effect on the output. So, in general, the regression model will be robust to perturbations in the input variables making the prediction less sensitive to changes in the distribution of the test set. For example, the authors in [40] use the invariance property for prediction to perform causal inference. In turn, Anchor Regression (AR) builds upon the causality literature to obtain robust regression solutions when the input variables have been perturbed [42]. The procedure relies on anchor variables capturing the heterogeneity within a dataset and a parameter γ that measures the deviation with respect to the least square solution. Once the values of the anchors are known, AR modifies the data and obtains the least square solution, as detailed in Section 2.2. In this paper, we propose Anchor Data Augmentation (ADA) to augment the training dataset with several replicas of the available data. We use a simple clustering of the data to encode a homogeneous group of observations and use different values of γ to robustify the solution to different strengths of potential distribution shifts. In every minibatch, we sample γ from a predetermined range around γ = 1. As AR was developed for linear regression, the data augmentation strategy needs to be modified for nonlinear regression accordingly. We validate ADA for in-distribution generalization and out-of-distribution robustness under the same conditions proposed in C-Mixup [49], as well as some illustrative linear and nonlinear regression examples. In the replicated experiments, ADA is competitive or superior to other augmentation strategies such as C-Mixup, although on some datasets the performance gain is marginal. The rest of the paper is organized as follows: First, we provide background information in Section 2. We give a brief overview of related work on data augmentation in Section 2.1 and summarize the key concepts on Anchor Regression in Section 2.2. Second, Section 3 shows how we extend Anchor Regression and introduces ADA. Section 4 reports empirical evidence that our approach can improve predictions, especially in over-parameterized settings. We conclude the paper in Section 5. This paper is available on arxiv under CC0 1.0 DEED license. [1] Our Python implementation of ADA is available at: https://github.com/noraschneider/ anchordataaugmentation/ Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Authors: Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.1 Data Augmentation 2.2 Anchor Regression 2.2 Anchor Regression 3 Anchor Data Augmentation 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation A Additional information for Anchor Data Augmentation B Experiments B Experiments Abstract We propose a novel algorithm for data augmentation in nonlinear over-parametrized regression. Our data augmentation algorithm borrows from the literature on causality and extends the recently proposed Anchor regression (AR) method for data augmentation, which is in contrast to the current state-of-the-art domain-agnostic solutions that rely on the Mixup literature. Our Anchor Data Augmentation (ADA) uses several replicas of the modified samples in AR to provide more training examples, leading to more robust regression predictions. We apply ADA to linear and nonlinear regression problems using neural networks. ADA is competitive with state-of-the-art C-Mixup solutions. [1] 1 Introduction Data augmentation is one of the key ingredients of any successful application of a machine learning classifier. The first example that typically comes to mind is the in-depth description of the data augmentation in the now-famous Alexnet paper [26]. Data augmentation algorithms come in different flavors, and they mostly rely on the expectation that small perturbations, invariances, or symmetries applied to the input will not change the class label. That way, we can present ‘fresh new’ samples as alterations of the available examples for training. These transformations modify the input distribution to make the algorithm more robust for cases where the distribution of the test set may differ from that of the training set. We refer the reader to the related work section (Section 2.1) for an overview and description of different data augmentation strategies. The literature for data augmentation in regression is slim. The paper on Mixup augmentation [51] proposes a simple and general scheme for data augmentation using convex combinations of samples. The authors only apply their data augmentation proposal to classification problems. They conjecture in the discussion that the application to regression is straightforward, however, this is not the case in practice. Mixup is theoretically analyzed in [5, 52] as a regularization technique for classification and regression problems. However, it is only illustrated in classification problems. The Mixup algorithm has been extended to regression problems in [18, 49], in which the authors explain that Mixup cannot be blindly applied to regression problems. To our knowledge, these are the only two papers in which data augmentation for regression is proposed. RegMix [18] relies on a hard-to-train prior neural network controller before augmenting the data using a Mixup strategy. C-Mixup [49], a method proposed more recently, solves some of the issues limiting the standard Mixup algorithm for regression problems. The authors propose to mix only closeby samples in the output space (i.e., samples which have close enough labels). This strategy is only valid when the target variables are monotonic with the input and is applied in a transformed space. The authors present comprehensive results in data augmentation for in-distribution generalization, task generalization and out-of-distribution robustness. In this paper, we rely on the causality literature to provide a different avenue for augmenting data in regression problems. Causal discovery finds the causes of a response variable among a given set of observations or helps to recognize the causal relations between a set of variables [39]. These causes allow us to understand how these relations will change if we were to intervene in a subset of the (input) variables or what would be the effect on the output. So, in general, the regression model will be robust to perturbations in the input variables making the prediction less sensitive to changes in the distribution of the test set. For example, the authors in [40] use the invariance property for prediction to perform causal inference. In turn, Anchor Regression (AR) builds upon the causality literature to obtain robust regression solutions when the input variables have been perturbed [42]. The procedure relies on anchor variables capturing the heterogeneity within a dataset and a parameter γ that measures the deviation with respect to the least square solution. Once the values of the anchors are known, AR modifies the data and obtains the least square solution, as detailed in Section 2.2. In this paper, we propose Anchor Data Augmentation (ADA) to augment the training dataset with several replicas of the available data. We use a simple clustering of the data to encode a homogeneous group of observations and use different values of γ to robustify the solution to different strengths of potential distribution shifts. In every minibatch, we sample γ from a predetermined range around γ = 1. As AR was developed for linear regression, the data augmentation strategy needs to be modified for nonlinear regression accordingly. We validate ADA for in-distribution generalization and out-of-distribution robustness under the same conditions proposed in C-Mixup [49], as well as some illustrative linear and nonlinear regression examples. In the replicated experiments, ADA is competitive or superior to other augmentation strategies such as C-Mixup, although on some datasets the performance gain is marginal. Anchor Data Augmentation (ADA) The rest of the paper is organized as follows: First, we provide background information in Section 2. We give a brief overview of related work on data augmentation in Section 2.1 and summarize the key concepts on Anchor Regression in Section 2.2. Second, Section 3 shows how we extend Anchor Regression and introduces ADA. Section 4 reports empirical evidence that our approach can improve predictions, especially in over-parameterized settings. We conclude the paper in Section 5. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv [1] Our Python implementation of ADA is available at: https://github.com/noraschneider/ anchordataaugmentation/

A Novel Anchor Data Augmentation Algorithm for Nonlinear Models

GRAND CAYMAN, Cayman Islands, November 13th, 2024/Chainwire/--With a 70-year legacy of offering investment solutions across diverse industries and asset classes, privately held VanEck now provides millions of customers access to SUI Investment firm VanEck has introduced a product representing SUI. VanEck, known for its financial products that provide access to Bitcoin, Ethereum, and Solana, now offers exposure to SUI, the token behind the groundbreaking Layer 1 blockchain recognized for its industry-leading performance and infinite horizontal scalability. This fully-collateralized financial instrument provides millions of investors access to SUI. With the new offering from VanEck, millions can now access SUI without needing to hold SUI in a crypto wallet. VanEck holds the referenced SUI tokens in a regulated custodian environment covered by each note, ensuring that the product's value tracks the underlying tokens' value. Like other financial products, the new VanEck offering enables users to easily invest and divest. The launch of VanEck's SUI-backed product is the latest demonstration of institutional confidence in Sui, following notable on-chain developments. These include partnerships with services like Copper, Zero Hash and Fordefi and the integration of stablecoins such as USDC, AUSD, and the stablecoin-like USDY. Over the past year, Sui has experienced remarkable growth, with total value locked (TVL) increasing by 430% and DeFi volume surging by 692%. In August, 2024, in another show of support, Grayscale, one of the world’s largest crypto asset managers, launched its Grayscale® Sui Trust. “We are excited to introduce this product as part of our growing portfolio of digital asset investment products,” said Menno Martens, Crypto Specialist and Product Manager at VanEck. “Sui’s proven track record and reputation as an industry leader, as well as its innovative and technological advancements, made this the next logical product to house in the portfolio. We believe this development will give investors a unique opportunity to access one of the most advanced blockchain networks available today.” “The creation of this SUI-based product by VanEck is a pivotal moment for the Sui ecosystem and a significant vote of confidence in Sui’s technology and long-term vision,” said Gap Kim, Head of Marketing for the Sui Foundation. “VanEck is a trusted leader in the investment industry, and the decision to offer SUI as part of their product suite underscores the growing institutional interest in Sui.” To learn more, visit https://blog.sui.io/vaneck-investor-product/ or https://www.vaneck.com/. About Sui Sui is a first-of-its-kind Layer 1 blockchain and smart contract platform designed from the ground up to make digital asset ownership fast, private, secure, and accessible to everyone. Its object-centric model, based on the Move programming language, enables parallel execution, sub-second finality, and rich on-chain assets. With horizontally scalable processing and storage, Sui supports a wide range of applications with unrivaled speed at low cost. Sui is a step-function advancement in blockchain and a platform on which creators and developers can build amazing user-friendly experiences. For more information about Sui, users can visit https://sui.io. About VanEck VanEck has a history of looking beyond the financial markets to identify trends that are likely to create impactful investment opportunities. VanEck was one of the first U.S. asset managers to offer investors access to international markets. This set the tone for the firm’s drive to identify asset classes and trends—including gold investing in 1968, emerging markets in 1993, and Exchange Traded Funds in 2006—that subsequently shaped the investment management industry. Today, VanEck offers active and passive strategies with compelling exposures supported by well-designed investment processes. The firm’s capabilities range from core investment opportunities to more specialized exposures to enhance portfolio didiversification. VanEck's actively managed strategies are fueled by in-depth, bottom-up research and security selection from portfolio managers with direct experience in the sectors and regions in which they invest. Investability, liquidity, diversity, and transparency are key to the experienced decision-making around market and index selection underlying VanEck’s passive strategies. Since VanEck's founding in 1955, putting its clients’ interests first, in all market environments, has been at the heart of the firm’s mission. Contact Sui Foundation media@sui.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here GRAND CAYMAN, Cayman Islands, November 13th, 2024/Chainwire/--With a 70-year legacy of offering investment solutions across diverse industries and asset classes, privately held VanEck now provides millions of customers access to SUI Investment firm VanEck has introduced a product representing SUI. VanEck, known for its financial products that provide access to Bitcoin, Ethereum, and Solana, now offers exposure to SUI, the token behind the groundbreaking Layer 1 blockchain recognized for its industry-leading performance and infinite horizontal scalability. This fully-collateralized financial instrument provides millions of investors access to SUI. With the new offering from VanEck, millions can now access SUI without needing to hold SUI in a crypto wallet. VanEck holds the referenced SUI tokens in a regulated custodian environment covered by each note, ensuring that the product's value tracks the underlying tokens' value. Like other financial products, the new VanEck offering enables users to easily invest and divest. The launch of VanEck's SUI-backed product is the latest demonstration of institutional confidence in Sui, following notable on-chain developments. These include partnerships with services like Copper , Zero Hash and Fordefi and the integration of stablecoins such as USDC , AUSD , and the stablecoin-like USDY . Copper Copper Zero Hash Zero Hash Fordefi Fordefi USDC USDC AUSD AUSD USDY USDY Over the past year, Sui has experienced remarkable growth, with total value locked (TVL) increasing by 430% and DeFi volume surging by 692%. In August, 2024, in another show of support, Grayscale, one of the world’s largest crypto asset managers, launched its Grayscale® Sui Trust. “We are excited to introduce this product as part of our growing portfolio of digital asset investment products,” said Menno Martens, Crypto Specialist and Product Manager at VanEck. “We are excited to introduce this product as part of our growing portfolio of digital asset investment products,” said Menno Martens, Crypto Specialist and Product Manager at VanEck. “Sui’s proven track record and reputation as an industry leader, as well as its innovative and technological advancements, made this the next logical product to house in the portfolio. We believe this development will give investors a unique opportunity to access one of the most advanced blockchain networks available today.” “Sui’s proven track record and reputation as an industry leader, as well as its innovative and technological advancements, made this the next logical product to house in the portfolio. We believe this development will give investors a unique opportunity to access one of the most advanced blockchain networks available today.” “The creation of this SUI-based product by VanEck is a pivotal moment for the Sui ecosystem and a significant vote of confidence in Sui’s technology and long-term vision,” said Gap Kim, Head of Marketing for the Sui Foundation. “The creation of this SUI-based product by VanEck is a pivotal moment for the Sui ecosystem and a significant vote of confidence in Sui’s technology and long-term vision,” said Gap Kim, Head of Marketing for the Sui Foundation. “VanEck is a trusted leader in the investment industry, and the decision to offer SUI as part of their product suite underscores the growing institutional interest in Sui.” “VanEck is a trusted leader in the investment industry, and the decision to offer SUI as part of their product suite underscores the growing institutional interest in Sui.” To learn more, visit https://blog.sui.io/vaneck-investor-product/ or https://www.vaneck.com/ . https://blog.sui.io/vaneck-investor-product/ https://blog.sui.io/vaneck-investor-product/ https://www.vaneck.com/ https://www.vaneck.com/ About Sui Sui is a first-of-its-kind Layer 1 blockchain and smart contract platform designed from the ground up to make digital asset ownership fast, private, secure, and accessible to everyone. Its object-centric model, based on the Move programming language, enables parallel execution, sub-second finality, and rich on-chain assets. With horizontally scalable processing and storage, Sui supports a wide range of applications with unrivaled speed at low cost. Sui is a step-function advancement in blockchain and a platform on which creators and developers can build amazing user-friendly experiences. For more information about Sui, users can visit https://sui.io . https://sui.io https://sui.io About VanEck VanEck has a history of looking beyond the financial markets to identify trends that are likely to create impactful investment opportunities. VanEck was one of the first U.S. asset managers to offer investors access to international markets. This set the tone for the firm’s drive to identify asset classes and trends—including gold investing in 1968, emerging markets in 1993, and Exchange Traded Funds in 2006—that subsequently shaped the investment management industry. Today, VanEck offers active and passive strategies with compelling exposures supported by well-designed investment processes. The firm’s capabilities range from core investment opportunities to more specialized exposures to enhance portfolio didiversification. VanEck's actively managed strategies are fueled by in-depth, bottom-up research and security selection from portfolio managers with direct experience in the sectors and regions in which they invest. Investability, liquidity, diversity, and transparency are key to the experienced decision-making around market and index selection underlying VanEck’s passive strategies. Since VanEck's founding in 1955, putting its clients’ interests first, in all market environments, has been at the heart of the firm’s mission. Contact Sui Foundation media@sui.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here here here

Investment Giant VanEck Launches SUI-Based Financial Product

የኢንቨስትመንት ጃይንት ቫንኢክ በሱአይ ላይ የተመሰረተ የፋይናንሺያል ምርትን ጀመረ

Investiční gigant VanEck uvádí na trh finanční produkt založený na SUI

El gigante de las inversiones VanEck lanza un producto financiero basado en SUI

Envestisman Giant VanEck lanse pwodwi finansye ki baze sou SUI

Raksasa Investasi VanEck Luncurkan Produk Keuangan Berbasis SUI

Il gigante degli investimenti VanEck lancia un prodotto finanziario basato su SUI

投資大手ヴァンエックがSUIベースの金融商品を発売

ការវិនិយោគ Giant VanEck បើកដំណើរការផលិតផលហិរញ្ញវត្ថុផ្អែកលើ SUI

Investment Giant VanEck ले SUI-आधारित वित्तीय उत्पादन सुरु गर्‍यो

Maalgashiga Giant VanEck wuxuu bilaabay SUI-ku salaysan wax soo saarka maaliyadeed

முதலீட்டு நிறுவனமான VanEck SUI-அடிப்படையிலான நிதித் தயாரிப்பை அறிமுகப்படுத்துகிறது

Yatırım Devi VanEck, SUI Tabanlı Finansal Ürünü Piyasaya Sürüyor

Authors:
(1) Shih-Tang Su, University of Michigan, Ann Arbor (shihtang@umich.edu);
(2) Vijay G. Subramanian, University of Michigan, Ann Arbor and (vgsubram@umich.edu);
(3) Grant Schoenebeck, University of Michigan, Ann Arbor (schoeneb@umich.edu). Table of Links Abstract and 1. Introduction 2. Problem Formulation 2.1 Model of Binary-Outcome Experiments in Two-Phase Trials 3 Binary-outcome Experiments in Two-phase Trials and 3.1 Experiments with screenings 3.2 Assumptions and induced strategies 3.3 Constraints given by phase-II experiments 3.4 Persuasion ratio and the optimal signaling structure 3.5 Comparison with classical Bayesian persuasion strategies 4 Binary-outcome Experiments in Multi-phase trials and 4.1 Model of binary-outcome experiments in multi-phase trials 4.2 Determined versus sender-designed experiments 4.3 Multi-phase model and classical Bayesian persuasion and References 3.4 Persuasion ratio and the optimal signaling structure Careful readers may notice that if we multiply the persuasion ratio with the prior, the value will be the (maximum) expected utility the sender can achieve from the given set of strategies. Since the sender’s expected utility is monotone increasing in the prior p regardless of which set of strategies the sender adopts, the persuasion ratio under a given prior can be viewed as the relative utility gain this set of strategies can offer to the sender. Hence, given a specific prior, if a set of strategies has a higher persuasion ratio with respect to another set of strategies, the sender should use a strategy in the former instead of the latter. According to this discussion, we can draw a persuasion ratio curve for each set of strategies as the prior is varied in [0, 1]. Abusing notation, we represent the persuasion ratio curve by P R(S). It may appear that an optimization needs to be carried out for each value of the prior. However, structural insights presented in the following two lemmas considerably simplify the analysis. Properties presented in Lemma 3 narrow down the space where the sender needs to search for the optimal signaling strategy. This allows us to depict persuasion ratio curves P R(S) for some basic strategies. On top of that, Lemma 4 provides a systematic approach to derive persuasion ratio curves for all types of strategies. Lemma 3. Given a type of strategy S and a prior p, there exists a (sender’s) optimal strategy S ∈ S which satisfies one of the following two conditions: At least one IC requirement of the constituent induced strategies is tight;


There is a signal that will be sent with probability 1 under S. Before discussing the simplifications that Lemma 3 yields in terms of the key properties for solving the problem, we give an intuitive outline of the proof of Lemma 3. When the IC requirements of the two induced strategies are not tight and both signals are sent with non-zero probability in a strategy S, the sender can increase her expected utility by slightly raising the probability of the signal with a higher persuasion ratio (and adjust the probability of the other signal to respect the prior) to form a strategy S+. The sender can keep doing this ‘slight’ modification of her strategies until either one of the IC conditions is satisfied or the signal is sent with probability one. Lemma 5. The persuasion ratio curve of the optimal signaling strategy P R∗ (p) is the upper envelope of the different types of strategies’ persuasion curves. Further, the optimal signaling strategy (under a given prior) is the strategy that reaches the frontier of the persuasion ratio curve (at that prior). For a two-phase trial or to solve the last two phases of a trial with more than two phases studied in Section 4, the following corollary can further simplify the sender’s optimization procedure. With Corollary 1, the Equation (3) in Lemma 4 reduces to a linear equation. Hence, the comparison in Lemma 5 and the computation in Lemma 4 can be reduced to a comparison of the (unique) corresponding IC strategies (if one exists) under the interim belief listed in Corollary1 for different types of strategies. This paper is available on arxiv under CC 4.0 license. Authors: (1) Shih-Tang Su, University of Michigan, Ann Arbor (shihtang@umich.edu); (2) Vijay G. Subramanian, University of Michigan, Ann Arbor and (vgsubram@umich.edu); (3) Grant Schoenebeck, University of Michigan, Ann Arbor (schoeneb@umich.edu). Authors: Authors: (1) Shih-Tang Su, University of Michigan, Ann Arbor (shihtang@umich.edu); (2) Vijay G. Subramanian, University of Michigan, Ann Arbor and (vgsubram@umich.edu); (3) Grant Schoenebeck, University of Michigan, Ann Arbor (schoeneb@umich.edu). Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Problem Formulation 2. Problem Formulation 2.1 Model of Binary-Outcome Experiments in Two-Phase Trials 2.1 Model of Binary-Outcome Experiments in Two-Phase Trials 3 Binary-outcome Experiments in Two-phase Trials and 3.1 Experiments with screenings 3 Binary-outcome Experiments in Two-phase Trials and 3.1 Experiments with screenings 3.2 Assumptions and induced strategies 3.2 Assumptions and induced strategies 3.3 Constraints given by phase-II experiments 3.3 Constraints given by phase-II experiments 3.4 Persuasion ratio and the optimal signaling structure 3.4 Persuasion ratio and the optimal signaling structure 3.5 Comparison with classical Bayesian persuasion strategies 3.5 Comparison with classical Bayesian persuasion strategies 4 Binary-outcome Experiments in Multi-phase trials and 4.1 Model of binary-outcome experiments in multi-phase trials 4 Binary-outcome Experiments in Multi-phase trials and 4.1 Model of binary-outcome experiments in multi-phase trials 4.2 Determined versus sender-designed experiments 4.2 Determined versus sender-designed experiments 4.3 Multi-phase model and classical Bayesian persuasion and References 4.3 Multi-phase model and classical Bayesian persuasion and References 3.4 Persuasion ratio and the optimal signaling structure Careful readers may notice that if we multiply the persuasion ratio with the prior, the value will be the (maximum) expected utility the sender can achieve from the given set of strategies. Since the sender’s expected utility is monotone increasing in the prior p regardless of which set of strategies the sender adopts, the persuasion ratio under a given prior can be viewed as the relative utility gain this set of strategies can offer to the sender. Hence, given a specific prior, if a set of strategies has a higher persuasion ratio with respect to another set of strategies, the sender should use a strategy in the former instead of the latter. According to this discussion, we can draw a persuasion ratio curve for each set of strategies as the prior is varied in [0, 1]. Abusing notation, we represent the persuasion ratio curve by P R(S). It may appear that an optimization needs to be carried out for each value of the prior. However, structural insights presented in the following two lemmas considerably simplify the analysis. Properties presented in Lemma 3 narrow down the space where the sender needs to search for the optimal signaling strategy. This allows us to depict persuasion ratio curves P R(S) for some basic strategies. On top of that, Lemma 4 provides a systematic approach to derive persuasion ratio curves for all types of strategies. Lemma 3. Given a type of strategy S and a prior p, there exists a (sender’s) optimal strategy S ∈ S which satisfies one of the following two conditions: Lemma 3. Given a type of strategy S and a prior p, there exists a (sender’s) optimal strategy S ∈ S which satisfies one of the following two conditions: At least one IC requirement of the constituent induced strategies is tight; There is a signal that will be sent with probability 1 under S. At least one IC requirement of the constituent induced strategies is tight; At least one IC requirement of the constituent induced strategies is tight; At least one IC requirement of the constituent induced strategies is tight; There is a signal that will be sent with probability 1 under S. There is a signal that will be sent with probability 1 under S. There is a signal that will be sent with probability 1 under S. Before discussing the simplifications that Lemma 3 yields in terms of the key properties for solving the problem, we give an intuitive outline of the proof of Lemma 3. When the IC requirements of the two induced strategies are not tight and both signals are sent with non-zero probability in a strategy S, the sender can increase her expected utility by slightly raising the probability of the signal with a higher persuasion ratio (and adjust the probability of the other signal to respect the prior) to form a strategy S+. The sender can keep doing this ‘slight’ modification of her strategies until either one of the IC conditions is satisfied or the signal is sent with probability one. Lemma 5. The persuasion ratio curve of the optimal signaling strategy P R∗ (p) is the upper envelope of the different types of strategies’ persuasion curves. Further, the optimal signaling strategy (under a given prior) is the strategy that reaches the frontier of the persuasion ratio curve (at that prior). Lemma 5. The persuasion ratio curve of the optimal signaling strategy P R∗ (p) is the upper envelope of the different types of strategies’ persuasion curves. Further, the optimal signaling strategy (under a given prior) is the strategy that reaches the frontier of the persuasion ratio curve (at that prior). For a two-phase trial or to solve the last two phases of a trial with more than two phases studied in Section 4, the following corollary can further simplify the sender’s optimization procedure. With Corollary 1, the Equation (3) in Lemma 4 reduces to a linear equation. Hence, the comparison in Lemma 5 and the computation in Lemma 4 can be reduced to a comparison of the (unique) corresponding IC strategies (if one exists) under the interim belief listed in Corollary1 for different types of strategies. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

How to Maximize Persuasion Ratios in Two-Phase Trials

I've been picking away at a new board game for a few years now. I made some progress early on but got stalled with other priorities. I'm now going to push to get the game done. I think it might be a fun experiment to share my design progress online — and create a better game with live feedback from you — where you can influence the design. If people choose to follow and post feedback in the comments, I will take your direction with the design of the game, continue to post my progress with gameplay refinements and design sketches, and continue to ask for your feedback. About The Game This will be an educational board game to help players learn personal finance — but more broadly about life, money, and happiness. I know what you are thinking — The Game of Life! Wikipedia says this game was created in 1860 with the more modern version in 1960. The game has an iconic three-dimensional board with a colorful spinner in the middle. This game is exciting to look at and nicely tactile. It's a vivid childhood memory for many of us. But, to be honest, in IMO, it's not a great game to play. The gameplay is dated. I have no intention of duplicating any part of this game. Our game will have a fresh start (as I try to do with all of my games). I have already designed board games and software games and simulations for personal finance, so I have a good foundation for our new game. I want this game to be like a simulation that recreates realistic decisions people make in the real world. Player & Customer Profile The primary target customers will be schools. But if we can attract home users that will be great too. Designing a product for education first, instead of entertainment, is particularly challenging. It's even more challenging to design a game that crosses over from schools to homes. But we'll give it a shot! (I've written about this here — Using Entertainment Games for Education). Knowing your customer is super important when designing a product. To refine the player profile further, the target age range is likely to be 13 to adult. This encompasses middle schools and high schools. The game may also play well for 11 and 12-year-olds, but here's an interesting fact — we can save thousands of dollars in production costs by setting the minimum age to 13. Why? Because toys and games for ages 12 and under require comprehensive materials testing for child safety. Of course, we always source and use safe materials for our games, but when costly double-testing is not necessary, then we should consider our options. Goals The first step — choosing the goals of the game. This is an interesting challenge with personal finance — because there are no clear right and wrong answers — everyone has their own path in life. Some may choose to pursue financial wealth over everything else, while others focus on personal lifestyle or relationships with friends and family. Some may try to have it all. Our game goals must reflect this reality of personal choice. We must refrain from applying our own biases in an educational product. With this in mind, here is what I'm currently thinking — The overall goal should be Happiness. I feel like this is a good term that can be applied on a personal level regardless of individual life goals. The player with the most Happiness Points (HP) wins the game — although "win" may not be the right term here as we still have to decide if this will be a cooperative or competitive game or something in between. Reminder — the educational objective of the game is to help you gain the personal finance knowledge and skills to achieve your definition of happiness. Next, we have to define Happiness — my current thinking is that it should be made up of three metrics — Financial Wealth + Lifestyle + Friends & Family. Each player may put more or less emphasis on any combination of these three metrics. BTW, our brains like to work in threes. Fours become too complicated. Two is even better but may be too restrictive for the gameplay we want to design. You will see the word "Relationship" in my graphic above because I don't like that "Friends & Family" uses two words, so I'm thinking about alternatives. Where is Health? I carefully considered whether health should be one of the three metrics since it is perhaps the single most important influence on happiness. But I decided not to include in the three metrics because: Everyone wants good health (even if they may not make healthy choices).
Health influences all of our life goals.
Adding a fourth metric breaks our 3-metric rule. So, the best way to include health is to have it applied throughout the game as various choices and events, such as personal care, nutrition, fitness, routine checkups, health insurance, etc. There you have it ... that's where we are so far. I have more to share in the next post. My direction will be influenced by the feedback you provide. Share Your Thoughts In the Comments Am I on track? Am I missing something? Do you have any suggestions? Should I keep posting about this game? PS: Please only share ideas if you are willing to allow me or anyone following this discussion to use them for free without any obligation. If you contribute an idea that has a significant influence on the design of my game, I will be pleased to gift you a copy of the game. I've been picking away at a new board game for a few years now. I made some progress early on but got stalled with other priorities. I'm now going to push to get the game done. I think it might be a fun experiment to share my design progress online — and create a better game with live feedback from you — where you can influence the design. If people choose to follow and post feedback in the comments, I will take your direction with the design of the game, continue to post my progress with gameplay refinements and design sketches, and continue to ask for your feedback. About The Game This will be an educational board game to help players learn personal finance — but more broadly about life, money, and happiness. I know what you are thinking — The Game of Life! Wikipedia says this game was created in 1860 with the more modern version in 1960. The game has an iconic three-dimensional board with a colorful spinner in the middle. This game is exciting to look at and nicely tactile. It's a vivid childhood memory for many of us. But, to be honest, in IMO, it's not a great game to play. The gameplay is dated. I have no intention of duplicating any part of this game. Our game will have a fresh start (as I try to do with all of my games). I have already designed board games and software games and simulations for personal finance, so I have a good foundation for our new game. I want this game to be like a simulation that recreates realistic decisions people make in the real world. board games software games and simulations Player & Customer Profile The primary target customers will be schools. But if we can attract home users that will be great too. Designing a product for education first, instead of entertainment, is particularly challenging. It's even more challenging to design a game that crosses over from schools to homes. But we'll give it a shot! (I've written about this here — Using Entertainment Games for Education ). Using Entertainment Games for Education Knowing your customer is super important when designing a product. To refine the player profile further, the target age range is likely to be 13 to adult. This encompasses middle schools and high schools. The game may also play well for 11 and 12-year-olds, but here's an interesting fact — we can save thousands of dollars in production costs by setting the minimum age to 13. Why? Because toys and games for ages 12 and under require comprehensive materials testing for child safety. Of course, we always source and use safe materials for our games, but when costly double-testing is not necessary, then we should consider our options. Goals The first step — choosing the goals of the game. This is an interesting challenge with personal finance — because there are no clear right and wrong answers — everyone has their own path in life. Some may choose to pursue financial wealth over everything else, while others focus on personal lifestyle or relationships with friends and family. Some may try to have it all. Our game goals must reflect this reality of personal choice. We must refrain from applying our own biases in an educational product. With this in mind, here is what I'm currently thinking — The overall goal should be Happiness. I feel like this is a good term that can be applied on a personal level regardless of individual life goals. The player with the most Happiness Points (HP) wins the game — although "win" may not be the right term here as we still have to decide if this will be a cooperative or competitive game or something in between. Reminder — the educational objective of the game is to help you gain the personal finance knowledge and skills to achieve your definition of happiness. Next, we have to define Happiness — my current thinking is that it should be made up of three metrics — Financial Wealth + Lifestyle + Friends & Family. Each player may put more or less emphasis on any combination of these three metrics. BTW, our brains like to work in threes. Fours become too complicated. Two is even better but may be too restrictive for the gameplay we want to design. You will see the word "Relationship" in my graphic above because I don't like that "Friends & Family" uses two words, so I'm thinking about alternatives. Where is Health? I carefully considered whether health should be one of the three metrics since it is perhaps the single most important influence on happiness. But I decided not to include in the three metrics because: Everyone wants good health (even if they may not make healthy choices). Health influences all of our life goals. Adding a fourth metric breaks our 3-metric rule. Everyone wants good health (even if they may not make healthy choices). Health influences all of our life goals. Adding a fourth metric breaks our 3-metric rule. So, the best way to include health is to have it applied throughout the game as various choices and events, such as personal care, nutrition, fitness, routine checkups, health insurance, etc. There you have it ... that's where we are so far. I have more to share in the next post. My direction will be influenced by the feedback you provide. Share Your Thoughts In the Comments Am I on track? Am I missing something? Do you have any suggestions? Should I keep posting about this game? PS: Please only share ideas if you are willing to allow me or anyone following this discussion to use them for free without any obligation. If you contribute an idea that has a significant influence on the design of my game, I will be pleased to gift you a copy of the game.

Assist Me in Designing a Board Game - Part 1

Authors:
(1) Suriya Gunasekar, Microsoft Research;
(2) Yi Zhang, Microsoft Research;
(3) Jyoti Aneja, Microsoft Research;
(4) Caio C´esar Teodoro Mendes, Microsoft Research;
(5) Allie Del Giorno, Microsoft Research;
(6) Sivakanth Gopi, Microsoft Research;
(7) Mojan Javaheripi, Microsoft Research;
(8) Piero Kauffmann, Microsoft Research;
(9) Gustavo de Rosa, Microsoft Research;
(10) Olli Saarikivi, Microsoft Research;
(11) Adil Salim, Microsoft Research;
(12) Shital Shah, Microsoft Research;
(13) Harkirat Singh Behl, Microsoft Research;
(14) Xin Wang, Microsoft Research;
(15) S´ebastien Bubeck, Microsoft Research;
(16) Ronen Eldan, Microsoft Research;
(17) Adam Tauman Kalai, Microsoft Research;
(18) Yin Tat Lee, Microsoft Research;
(19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction
2 Training details and the importance of high-quality data
2.1 Filtering of existing code datasets using a transformer-based classifier
2.2 Creation of synthetic textbook-quality datasets
2.3 Model architecture and training
3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries
4 Evaluation on unconventional problems with LLM grading
5 Data pruning for unbiased performance evaluation
5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis
6 Conclusion and References
A Additional examples for Section 3
B Limitation of phi-1
C Examples for Section 5 Abstract We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of “textbook quality” data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval. 1 Introduction The art of training large artificial neural networks has made extraordinary progress in the last decade, especially after the discovery of the Transformer architecture [VSP+ 17], yet the science behind this success remains limited. Amidst a vast and confusing array of results, a semblance of order emerged around the same time as Transformers were introduced, namely that performance improves somewhat predictably as one scales up either the amount of compute or the size of the network [HNA+ 17], a phenomenon which is now referred to as scaling laws [KMH+ 20]. The subsequent exploration of scale in deep learning was guided by these scaling laws [BMR+ 20], and discoveries of variants of these laws led to rapid jump in performances [HBM+ 22]. In this work, following the footsteps of Eldan and Li [EL23], we explore the improvement that can be obtained along a different axis: the quality of the data. It has long been known that higher quality data leads to better results, e.g., data cleaning is an important part of modern dataset creation [RSR+ 20], and it can yield other side benefits such as somewhat smaller datasets [LYR+ 23, YGK+ 23] or allowing for more passes on the data [MRB+ 23]. The recent work of Eldan and Li on TinyStories (a high quality dataset synthetically generated to teach English to neural networks) showed that in fact the effect of high quality data extends well past this: improving data quality can dramatically change the shape of the scaling laws, potentially allowing to match the performance of large-scale models with much leaner training/models. In this work we go beyond the initial foray of Eldan and Li to show that high quality data can even improve the SOTA of large language models (LLMs), while dramatically reducing the dataset size and training compute. Importantly, smaller models requiring less training can significantly reduce the environmental cost of LLMs [BGMMS21]. We focus our attention on LLMs trained for code, and specifically writing simple Python functions from their docstrings as in [CTJ+ 21]. The evaluation benchmark proposed in the latter work, HumanEval, has been widely adopted for comparing LLMs’ performance on code. We demonstrate the power of high 1 quality data in breaking existing scaling laws by training a 1.3B-parameter model, which we call phi-1, for roughly 8 passes over 7B tokens (slightly over 50B total tokens seen) followed by finetuning on less than 200M tokens. Roughly speaking we pretrain on “textbook quality” data, both synthetically generated (with GPT-3.5) and filtered from web sources, and we finetune on “textbook-exercise-like” data. Despite being several orders of magnitude smaller than competing models, both in terms of dataset and model size (see Table 1), we attain 50.6% pass@1 accuracy on HumanEval and 55.5% pass@1 accuracy on MBPP (Mostly Basic Python Programs), which are one of the best self-reported numbers using only one LLM generation. In Section 2, we give some details of our training process, and we discuss evidence for the importance of our data selection process in achieving this result. Moreover, despite being trained on much fewer tokens compared to existing models, phi-1 still displays emergent properties. In Section 3 we discuss these emergent properties, and in particular we confirm the hypothesis that the number of parameters plays a key role in emergence (see e.g., [WTB+ 22]), by comparing the outputs of phi-1 with those of phi-1-small, a model trained with the same pipeline but with only 350M parameters. The methodology used in this section is reminiscent of the Sparks of AGI paper [BCE+ 23] that argued for moving away from static benchmarks to test LLMs’ performance. Finally in Section 4 we discuss alternative benchmarks to evaluate the model and in Section 5 we study possible contamination of our training data with respect to HumanEval. We release the model for usage and evaluation by the broader community, but omit some details of the synthetic data generation, for proprietary reasons. More related works Our work is part of the recent program of using LLMs for program synthesis, see [CTJ+ 21, NPH+ 22] for more references on this. Our approach is also part of the emerging trend of using existing LLMs to synthesize data for the training of new generations of LLMs, [WKM+ 22, TGZ+ 23, MMJ+ 23, LGK+ 23, JWJ+ 23]. There is an ongoing debate about whether such “recursive training” might lead to narrower scope for the resulting LLM [SSZ+ 23, GWS+ 23], see [MMJ+ 23] for a counterviewpoint. Note that in this paper we focus on a narrow task, similarly to [JWJ+ 23], in which case it seems plausible to attain better performance than the teacher LLM on that specific task (as is argued in the latter paper). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Authors: Authors: (1) Suriya Gunasekar, Microsoft Research; (2) Yi Zhang, Microsoft Research; (3) Jyoti Aneja, Microsoft Research; (4) Caio C´esar Teodoro Mendes, Microsoft Research; (5) Allie Del Giorno, Microsoft Research; (6) Sivakanth Gopi, Microsoft Research; (7) Mojan Javaheripi, Microsoft Research; (8) Piero Kauffmann, Microsoft Research; (9) Gustavo de Rosa, Microsoft Research; (10) Olli Saarikivi, Microsoft Research; (11) Adil Salim, Microsoft Research; (12) Shital Shah, Microsoft Research; (13) Harkirat Singh Behl, Microsoft Research; (14) Xin Wang, Microsoft Research; (15) S´ebastien Bubeck, Microsoft Research; (16) Ronen Eldan, Microsoft Research; (17) Adam Tauman Kalai, Microsoft Research; (18) Yin Tat Lee, Microsoft Research; (19) Yuanzhi Li, Microsoft Research. Table of Links Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References A Additional examples for Section 3 B Limitation of phi-1 C Examples for Section 5 Abstract and 1. Introduction Abstract and 1. Introduction 2 Training details and the importance of high-quality data 2 Training details and the importance of high-quality data 2.1 Filtering of existing code datasets using a transformer-based classifier 2.1 Filtering of existing code datasets using a transformer-based classifier 2.2 Creation of synthetic textbook-quality datasets 2.2 Creation of synthetic textbook-quality datasets 2.3 Model architecture and training 2.3 Model architecture and training 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 3 Spikes of model capability after finetuning on CodeExercises, 3.1 Finetuning improves the model’s understanding, and 3.2 Finetuning improves the model’s ability to use external libraries 4 Evaluation on unconventional problems with LLM grading 4 Evaluation on unconventional problems with LLM grading 5 Data pruning for unbiased performance evaluation 5 Data pruning for unbiased performance evaluation 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 5.1 N-gram overlap and 5.2 Embedding and syntax-based similarity analysis 6 Conclusion and References 6 Conclusion and References A Additional examples for Section 3 A Additional examples for Section 3 B Limitation of phi-1 B Limitation of phi-1 C Examples for Section 5 C Examples for Section 5 Abstract We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of “textbook quality” data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval. 1 Introduction The art of training large artificial neural networks has made extraordinary progress in the last decade, especially after the discovery of the Transformer architecture [VSP+ 17], yet the science behind this success remains limited. Amidst a vast and confusing array of results, a semblance of order emerged around the same time as Transformers were introduced, namely that performance improves somewhat predictably as one scales up either the amount of compute or the size of the network [HNA+ 17], a phenomenon which is now referred to as scaling laws [KMH+ 20]. The subsequent exploration of scale in deep learning was guided by these scaling laws [BMR+ 20], and discoveries of variants of these laws led to rapid jump in performances [HBM+ 22]. In this work, following the footsteps of Eldan and Li [EL23], we explore the improvement that can be obtained along a different axis: the quality of the data. It has long been known that higher quality data leads to better results, e.g., data cleaning is an important part of modern dataset creation [RSR+ 20], and it can yield other side benefits such as somewhat smaller datasets [LYR+ 23, YGK+ 23] or allowing for more passes on the data [MRB+ 23]. The recent work of Eldan and Li on TinyStories (a high quality dataset synthetically generated to teach English to neural networks) showed that in fact the effect of high quality data extends well past this: improving data quality can dramatically change the shape of the scaling laws, potentially allowing to match the performance of large-scale models with much leaner training/models. In this work we go beyond the initial foray of Eldan and Li to show that high quality data can even improve the SOTA of large language models (LLMs), while dramatically reducing the dataset size and training compute. Importantly, smaller models requiring less training can significantly reduce the environmental cost of LLMs [BGMMS21]. We focus our attention on LLMs trained for code, and specifically writing simple Python functions from their docstrings as in [CTJ+ 21]. The evaluation benchmark proposed in the latter work, HumanEval, has been widely adopted for comparing LLMs’ performance on code. We demonstrate the power of high 1 quality data in breaking existing scaling laws by training a 1.3B-parameter model, which we call phi-1, for roughly 8 passes over 7B tokens (slightly over 50B total tokens seen) followed by finetuning on less than 200M tokens. Roughly speaking we pretrain on “textbook quality” data, both synthetically generated (with GPT-3.5) and filtered from web sources, and we finetune on “textbook-exercise-like” data. Despite being several orders of magnitude smaller than competing models, both in terms of dataset and model size (see Table 1), we attain 50.6% pass@1 accuracy on HumanEval and 55.5% pass@1 accuracy on MBPP (Mostly Basic Python Programs), which are one of the best self-reported numbers using only one LLM generation. In Section 2, we give some details of our training process, and we discuss evidence for the importance of our data selection process in achieving this result. Moreover, despite being trained on much fewer tokens compared to existing models, phi-1 still displays emergent properties. In Section 3 we discuss these emergent properties, and in particular we confirm the hypothesis that the number of parameters plays a key role in emergence (see e.g., [WTB+ 22]), by comparing the outputs of phi-1 with those of phi-1-small, a model trained with the same pipeline but with only 350M parameters. The methodology used in this section is reminiscent of the Sparks of AGI paper [BCE+ 23] that argued for moving away from static benchmarks to test LLMs’ performance. Finally in Section 4 we discuss alternative benchmarks to evaluate the model and in Section 5 we study possible contamination of our training data with respect to HumanEval. We release the model for usage and evaluation by the broader community, but omit some details of the synthetic data generation, for proprietary reasons. More related works Our work is part of the recent program of using LLMs for program synthesis, see [CTJ+ 21, NPH+ 22] for more references on this. Our approach is also part of the emerging trend of using existing LLMs to synthesize data for the training of new generations of LLMs, [WKM+ 22, TGZ+ 23, MMJ+ 23, LGK+ 23, JWJ+ 23]. There is an ongoing debate about whether such “recursive training” might lead to narrower scope for the resulting LLM [SSZ+ 23, GWS+ 23], see [MMJ+ 23] for a counterviewpoint. Note that in this paper we focus on a narrow task, similarly to [JWJ+ 23], in which case it seems plausible to attain better performance than the teacher LLM on that specific task (as is argued in the latter paper). More related works This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Textbooks Are All You Need: Abstract and Introduction

Welcome to HackerNoon’s Writing Prompts! Would you like to take a stab at answering some of these questions? The link for the template is HERE. 1. What is your company in 2–5 words? Cassie is a Consent and Preference Management Platform 2. Why is now the time for your company to exist? When I founded Cassie back in 2000, we were one of the earliest pioneers of consent and preference management (CMP). In the nearly 25 years since then, the company has tailored solutions for some of the world’s top brands, helping them stay ahead of constantly changing data privacy laws and customer expectations.With data privacy becoming more complex, businesses face increasing pressure to comply with new regulations, especially in the U.S. with new laws emerging nationwide. Cassie doesn’t just help businesses keep up—we stay ahead with reliable, always-on compliance solutions. Having a trusted CMP partner like Cassie is crucial for maintaining consumer trust and ensuring growth built on a solid compliance foundation. 3. What do you love about your team, and why are you the ones to solve this problem? I love that the Cassie team is all about collaboration, with a strong focus on customer success and listening. We value ideas from every team member, as everyone shares a passion for innovation and delivering top-notch solutions to the brands we work with daily.We’re not just reactive - we’re proactive in shaping solutions that anticipate our customers' needs and finding creative ways to solve their challenges. It’s our talent, drive, and dedication that truly sets us apart in the CMP landscape, enabling us to make a positive impact for our customers. 4. If you weren’t building your startup, what would you be doing? My husband and I live in the hills of North Wales with our chickens, sheep, and bees. If I weren’t at Cassie, I’d spend all my time with our animals and tending to our garden! 5. At the moment, how do you measure success? What are your metrics? At Cassie, we measure success through client satisfaction, platform adoption, and compliance effectiveness. We look at metrics like client retention rates, user engagement, and compliance audit outcomes. This helps us measure our ability to meet customer needs and demonstrate our impact in ensuring regulatory compliance across industries. 6. In a few sentences, what do you offer to whom? Cassie caters to any organization seeking a dynamic solution to help them meet compliance standards, with our platform centrally managing over 1.2 billion customer records for large companies. Our clients range across industries, including healthcare, higher education, and retail to name a few. 7. What’s most exciting about your traction to date? The most exciting part of our traction has been seeing our team’s flexibility and innovation create tangible results for customers. Earlier this year, we rolled out our Cross-Domain Consent feature in response to a client's request for a seamless way to manage consent across different domains. We were the first to market with this solution, and it’s been a game-changer, simplifying user experiences by reducing repetitive consent requests. This shows how we’re committed to staying ahead in offering solutions that meet our client’s needs - it’s all about evolving to meet these opportunities head-on and bring our company to new levels. 8. Where do you think your growth will be next year? I think we’ll be able to see a lot of growth in response to evolving U.S. data privacy standards. Businesses are looking for ways to stay compliant and keep up with state-by-state regulations without all the headaches.With proposed data privacy standards like the American Privacy Rights Act bouncing around Congress, people are demanding more control over how their information is handled. There's a growing need for solutions that give individuals more say in their data. That's where we see a big opportunity to expand, helping companies navigate these regulatory changes smoothly. 9. Tell us about your first paying customer and revenue expectations over the next year. Cassie continues to grow substantially each year. We are privately owned without debt, so don’t have to make decisions that are not advantageous for our product or our clients. Our client and employee retention are one of our proudest metrics! Weexpect mid-double figure percentage growth again this year. 10. What’s your biggest threat? Our biggest threat comes from the break-speed at which data privacy laws and customer expectations are evolving – both of which indicate there is a heightened need to ensure Cassie not only meets but exceeds compliance standards andkeeps earning consumer trust. Recent Cassie research found that 93% of consumers are concerned about the security of their personal information online, with 59% also reporting that their concerns about data privacy have increased over the last 12 months. This is where we step in as a resource to help companies ease the worries of the public when it comes to data and consent. Especially in the age of AI, with vast amounts of user data being collected and used for training, many consumers aren’t fully aware of just how much information is being collected about them online. We’re all about being proactive and looking ahead, which helps us address any changes or challenges and implement top-notch solutions for our global customers. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Welcome to HackerNoon’s Writing Prompts! Would you like to take a stab at answering some of these questions? The link for the template is HERE. Welcome to HackerNoon ’s Writing Prompts! Would you like to take a stab at answering some of these questions? The link for the template is HERE. HackerNoon The link for the template is HERE. 1. What is your company in 2–5 words? Cassie is a Consent and Preference Management Platform 2. Why is now the time for your company to exist? When I founded Cassie back in 2000, we were one of the earliest pioneers of consent and preference management (CMP). In the nearly 25 years since then, the company has tailored solutions for some of the world’s top brands, helping them stay ahead of constantly changing data privacy laws and customer expectations. With data privacy becoming more complex, businesses face increasing pressure to comply with new regulations, especially in the U.S. with new laws emerging nationwide. Cassie doesn’t just help businesses keep up—we stay ahead with reliable, always-on compliance solutions. Having a trusted CMP partner like Cassie is crucial for maintaining consumer trust and ensuring growth built on a solid compliance foundation. 3. What do you love about your team, and why are you the ones to solve this problem? I love that the Cassie team is all about collaboration, with a strong focus on customer success and listening. We value ideas from every team member, as everyone shares a passion for innovation and delivering top-notch solutions to the brands we work with daily. We’re not just reactive - we’re proactive in shaping solutions that anticipate our customers' needs and finding creative ways to solve their challenges. It’s our talent, drive, and dedication that truly sets us apart in the CMP landscape, enabling us to make a positive impact for our customers. 4. If you weren’t building your startup, what would you be doing? My husband and I live in the hills of North Wales with our chickens, sheep, and bees. If I weren’t at Cassie, I’d spend all my time with our animals and tending to our garden! 5. At the moment, how do you measure success? What are your metrics? At Cassie, we measure success through client satisfaction, platform adoption, and compliance effectiveness. We look at metrics like client retention rates, user engagement, and compliance audit outcomes. This helps us measure our ability to meet customer needs and demonstrate our impact in ensuring regulatory compliance across industries. 6. In a few sentences, what do you offer to whom? Cassie caters to any organization seeking a dynamic solution to help them meet compliance standards, with our platform centrally managing over 1.2 billion customer records for large companies. Our clients range across industries, including healthcare, higher education, and retail to name a few. 7. What’s most exciting about your traction to date? The most exciting part of our traction has been seeing our team’s flexibility and innovation create tangible results for customers. Earlier this year, we rolled out our Cross-Domain Consent feature in response to a client's request for a seamless way to manage consent across different domains. We were the first to market with this solution, and it’s been a game-changer, simplifying user experiences by reducing repetitive consent requests. This shows how we’re committed to staying ahead in offering solutions that meet our client’s needs - it’s all about evolving to meet these opportunities head-on and bring our company to new levels. 8. Where do you think your growth will be next year? I think we’ll be able to see a lot of growth in response to evolving U.S. data privacy standards. Businesses are looking for ways to stay compliant and keep up with state-by-state regulations without all the headaches. With proposed data privacy standards like the American Privacy Rights Act bouncing around Congress, people are demanding more control over how their information is handled. There's a growing need for solutions that give individuals more say in their data. That's where we see a big opportunity to expand, helping companies navigate these regulatory changes smoothly. 9. Tell us about your first paying customer and revenue expectations over the next year. Cassie continues to grow substantially each year. We are privately owned without debt, so don’t have to make decisions that are not advantageous for our product or our clients. Our client and employee retention are one of our proudest metrics! We expect mid-double figure percentage growth again this year. 10. What’s your biggest threat? Our biggest threat comes from the break-speed at which data privacy laws and customer expectations are evolving – both of which indicate there is a heightened need to ensure Cassie not only meets but exceeds compliance standards and keeps earning consumer trust. Recent Cassie research found that 93% of consumers are concerned about the security of their personal information online, with 59% also reporting that their concerns about data privacy have increased over the last 12 months. This is where we step in as a resource to help companies ease the worries of the public when it comes to data and consent. Especially in the age of AI, with vast amounts of user data being collected and used for training, many consumers aren’t fully aware of just how much information is being collected about them online. We’re all about being proactive and looking ahead, which helps us address any changes or challenges and implement top-notch solutions for our global customers. Would you like to take a stab at answering some of these questions? The link for the template is HERE. Would you like to take a stab at answering some of these questions? The link for the template is HERE. The link for the template is HERE.

Cassie Founder, Nicky Watson, Answers: 10 Questions for Every Startup Founder

Authors:
(1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier;
(2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier;
(3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier;
(4) Stefano Ermon, CZ Biohub;
(5) Christopher D. Manning, Stanford University;
(6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 6 Experiments 7 Discussion, Acknowledgements, and References Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details 5 Theoretical Analysis of DPO In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO [37]). 5.1 Your Language Model Is Secretly a Reward Model Definition 1. We say that two reward functions r(x, y) and r ′ (x, y) are equivalent iff r(x, y) − r ′ (x, y) = f(x) for some function f. It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas: Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution. Lemma 2.** Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. The proofs are straightforward and we defer them to Appendix A.5. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models [30]. Due to this under-specification, we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [4]. The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6: 5.2 Instability of Actor-Critic Algorithms This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Authors: Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Direct Preference Optimization 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 5 Theoretical Analysis of DPO 6 Experiments 6 Experiments 7 Discussion, Acknowledgements, and References 7 Discussion, Acknowledgements, and References Author Contributions Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details D.3 Human study details 5 Theoretical Analysis of DPO In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO [37]). 5.1 Your Language Model Is Secretly a Reward Model Definition 1. We say that two reward functions r(x, y) and r ′ (x, y) are equivalent iff r(x, y) − r ′ (x, y) = f(x) for some function f. Definition 1. We say that two reward functions r(x, y) and r ′ (x, y) are equivalent iff r(x, y) − r ′ (x, y) = f(x) for some function f. It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas: Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution. Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution. Lemma 2. ** Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. Lemma 2 . Lemma 2 Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. The proofs are straightforward and we defer them to Appendix A.5. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models [30]. Due to this under-specification, we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [4]. The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6: 5.2 Instability of Actor-Critic Algorithms This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Theoretical Analysis of Direct Preference Optimization

Let's start with: Are CSS positions that aren't other properties enough to make you faint? Why CSS positions? Say you want to create a navigation bar that stays on the page when you scroll, or the chat with us icon/button that stays on the bottom  right corner always. These things are placed outside of the document's (i.e., the webpage) workflow. For example, you see below the chat button or the navigation bar is still visible even after scrolling. To build elements that have similar behavior, we use CSS position. The CSS position property is used to define the position of an element on a webpage. The position property has the following five values: static (default value)


relative


absolute


fixed


sticky We will look into each of them. Before we begin, let’s first understand the basic context; we have a very basic div which contains 9 more boxes each with a different color and also has a number on it. CSS Static Position (Default Value) The position: staticproperty allows elements to be positioned in the normal flow of the document. Note: This is the default value. The below boxes has position: static;. Eg codesandbox link CSS Relative Position The position: relative; property positions the element relative to its original position. The top, right, bottom, left properties are used to move the element accordingly. Here, we give position: relative; to box one and make it top: 35px and right: 5px;. So, the box one moves to the top and right to its original position. Note: The space is still preserved in the original position of the element. CSS Absolute Position The position: absolute; property removes the element completely from the normal flow of the document. It is positioned relative to the nearest parent element having a position other than static. If there is no parent with a position other than static, then it is positioned relative to the document itself. Below, both the boxes has top: 50px; and right: 50px; the only difference is the first eg is relative to the document (webpage), and the second eg is relative to the border container. Here, box one is no longer part of the border container. Note: An absolutely positioned element loses original space in the document flow. CSS Fixed Position The position: fixed; property positions an element to remain fixed in the same position, even when the page is scrolled. It is similar to the absolute value, but it remains relative to the viewport at all times. Here, the box one is positioned 50px from top and right; it will remain in the same position even when the page is scrolled. CSS Sticky Position The position: fixed; positions the element in the combination of fixed and relative values. This property allows the element to stick to a specific position in the viewport as you scroll, but only within its containing element. When you first scroll, the element behaves like it has relative positioning. It moves with the flow of the document. Once the element reaches the specified position (defined by top, right, bottom, or left), it "sticks" to that position. It will stay in that position as you continue to scroll. The element will stop sticking and resume normal document flow once the containing element (the nearest scrollable ancestor) is out of view. Conclusion Thank you for reading!! If you found this helpful, drop your reactions, and share this piece with others. Let me know in the comments, "Why  you suck at CSS?" You can also stay connected with me by following me here and on X, GitHub, and LinkedIn. Let's start with: Are CSS positions that aren't other properties enough to make you faint? Why CSS positions? Say you want to create a navigation bar that stays on the page when you scroll, or the chat with us icon/button that stays on the bottom  right corner always. These things are placed outside of the document's (i.e., the webpage) workflow. For example, you see below the chat button or the navigation bar is still visible even after scrolling. To build elements that have similar behavior, we use CSS position . CSS position The CSS position property is used to define the position of an element on a webpage. CSS position The position property has the following five values: static (default value) relative absolute fixed sticky static (default value) static (default value) relative relative absolute absolute fixed fixed sticky sticky We will look into each of them. Before we begin, let’s first understand the basic context; we have a very basic div which contains 9 more boxes each with a different color and also has a number on it. div CSS Static Position (Default Value) The position: static property allows elements to be positioned in the normal flow of the document. position: static Note: This is the default value. Note: This is the default value. The below boxes has position: static; . position: static; Eg codesandbox link Eg codesandbox link CSS Relative Position The position: relative; property positions the element relative to its original position. The top, right, bottom, left properties are used to move the element accordingly. position: relative; top, right, bottom, left Here, we give position: relative; to box one and make it top: 35px and right: 5px; . So, the box one moves to the top and right to its original position. position: relative; top: 35px right: 5px; Note: The space is still preserved in the original position of the element. Note: The space is still preserved in the original position of the element. CSS Absolute Position The position: absolute; property removes the element completely from the normal flow of the document. position: absolute; It is positioned relative to the nearest parent element having a position other than static . relative static If there is no parent with a position other than static , then it is positioned relative to the document itself. static Below, both the boxes has top: 50px; and right: 50px; the only difference is the first eg is relative to the document (webpage), and the second eg is relative to the border container. top: 50px; and right: 50px; Here, box one is no longer part of the border container. Note: An absolutely positioned element loses original space in the document flow. Note: An absolutely positioned element loses original space in the document flow. CSS Fixed Position The position: fixed; property positions an element to remain fixed in the same position, even when the page is scrolled. It is similar to the absolute value, but it remains relative to the viewport at all times. position: fixed; Here, the box one is positioned 50px from top and right ; it will remain in the same position even when the page is scrolled. 50px from top and right CSS Sticky Position The position: fixed; positions the element in the combination of fixed and relative values. position: fixed; fixed relative This property allows the element to stick to a specific position in the viewport as you scroll, but only within its containing element. When you first scroll, the element behaves like it has relative positioning. It moves with the flow of the document. Once the element reaches the specified position (defined by top, right, bottom, or left), it "sticks" to that position. It will stay in that position as you continue to scroll. The element will stop sticking and resume normal document flow once the containing element (the nearest scrollable ancestor) is out of view. Conclusion Thank you for reading!! If you found this helpful, drop your reactions, and share this piece with others. Let me know in the comments, "Why  you suck at CSS?" You can also stay connected with me by following me here and on X , GitHub , and LinkedIn . X GitHub LinkedIn

CSS Positions: Real Examples to Help You Learn

My Testing Journey With Jasmine and Mocha: What I've Learned

Key Management System: Everything You Need to Know

কী ম্যানেজমেন্ট সিস্টেম: আপনার যা জানা দরকার

Schlüsselverwaltungssystem: Alles, was Sie wissen müssen

Sistema de gestión de claves: todo lo que necesita saber

Système de gestion des clés : tout ce que vous devez savoir

कुंजी प्रबंधन प्रणाली: वह सब कुछ जो आपको जानना आवश्यक है

鍵管理システム: 知っておくべきことすべて

키 관리 시스템: 알아야 할 모든 것

Sistema de gerenciamento de chaves: tudo o que você precisa saber

Система управления ключами: все, что вам нужно знать

Anahtar Yönetim Sistemi: Bilmeniz Gereken Her Şey

Hệ thống quản lý khóa: Mọi thứ bạn cần biết

密钥管理系统：您需要了解的一切

Stories, News & Mentions about WEGIVE

Wegive’s Stories on HackerNoon