We help businesses of all sizes build intelligent applications

Azumo

Not all crypto products are born equal. Chainalysis’ 2020 Global Adoption Index recently illustrated just how much the adoption of cryptocurrencies is continuing to grow worldwide year on year. Before Zumo arrived in the app store, choosing a place to buy, sell or store your cryptocurrency was a bit like playing that old favourite Minesweeper; the landscape was littered with an array of well-orchestrated scams and schemes ready to rid you of your hard-earned cash. Cryptocurrency scams netted over $4 billion in 2019. Amongst the targets were popular trading exchanges, world-leading social media platforms, and thousands of private email accounts. Blockchain analytics provider Whale Alert earlier this year indicating that $24 million of cryptocurrency was stolen in the first 6 months of 2020 alone. released figures From fake Bitcoin giveaways hosted by Elon Musk (not the real one) to multi-tier Ponzi schemes, scamming  big business in crypto. Here are 5 ways that you could have lost your crypto in 2020: 1. Falling for the biggest Twitter scam of the year Spend 10 minutes scrolling results of a tag search on Twitter. #cryptocurrency You will find an online community full of tech investors, thought-leaders, top tier computer programmers, revolutionaries, and professional meme artists (yes that is a real job now) that live and breathe everything crypto. But what also becomes all too clear whilst clambering through the depths of what is often referred to as “ ” (or CT for short) is that there are plenty of ways to get fleeced of your cash. Crypto Twitter A popular type of scam on Twitter and one that made world news headlines earlier this year is the “fake giveaway”. On July 15th, . The hackers used their profiles to promote a fake cryptocurrency giveaway that promised users $2,000 in Bitcoin in return for $1000. verified Twitter accounts of high profile figures and companies were hacked Accounts hacked included Apple, President Barack Obama, former Vice President Joe Biden, Elon Musk, Kanye West, Bill Gates, and Uber. Hackers collected over $100k of stolen funds from the attack, and accessed Twitter’s “God Mode” through a phishing attack that tricked a Twitter employee into handing over login credentials (we cover the joys of phishing in #5). If you want a safe giveaway, stick with Zumo’s refer and share programme: guaranteed 0.02 Eth for you and for every person you refer, every time. 2. Leaving funds on a centralised exchange (that gets hacked) Centralised exchanges act as a middleman for anyone looking to buy and sell cryptocurrencies. Offering a variety of currencies to trade in a somewhat regulated environment, centralised exchanges are a popular choice for many newcomers to crypto. But there’s an issue. Once deposited on a centralised exchange, the fate of your funds lie entirely in the hands of the exchange owners. Sounds risky? It can be. On September 25th 2020, from KuCoin, a popular cryptocurrency exchange platform. hackers stole more than $275 million worth of cryptocurrency This incident was one of the largest ever exchange hacks and, as the hackers used various methods including mixers, diversifying into a handful of currencies on decentralised exchanges in an attempt to wash their ill-gotten gains, it has been difficult to trace. It is imperative that you take custody of your own funds when not actively trading them. Holding them in a that only you have access to is the only way that you can ensure safety from large scale hacks. secure personal wallet So write it down, make it your morning mantra, heck get it tattooed in your forehead if you must, but always remember: "Not your keys, not your crypto". Zumo hands you the keys and is by far the safest way to interact with your crypto on the blockchain. 3. Getting SIM swapped SIM swapping is on the rise in 2020. SIM swappers trick your mobile service provider into porting your phone number over to a new SIM card embedded in a device that they control. Scammers can gain access to everything on your phone including email accounts, bank accounts and trading exchange accounts. by professors and Ph.D. students at Harvard University highlighted the increased vulnerability to SIM swap attacks in 2020, especially as more people are working remotely than ever before. Research published earlier this year What is scarier is that whilst conducting the study, one of the authors of the research paper himself was SIM swapped. Go figure. To give you an idea of just how much damage can be done through SIM swapping, hold your breath and of how one man lost $24 million of cryptocurrency in a coordinated attack. read this unfortunate tale But don’t worry, if you’re a Zumo app user, even if your SIM were to be swapped, they still wouldn’t be able to access your crypto without your keys. Always keep your key codes separate to your phone! 4. Being tricked by phishing websites 2020 may go down in the history books as the . “year of the phish” Socially engineered cyber attacks might well be this year’s flavour when it comes to scams, and have been prevalent in the cryptocurrency space. These intricately designed attacks often direct people via email to fake websites, ultimately tricking them into unknowingly handing over their private details. Just earlier this month over by scammers that managed to trick users into visiting replicated versions of the popular hardware wallet website Ledger. $280k of the cryptocurrency Ripple (XRP) was stolen This year also saw accused of stealing nearly $17 million worth of cryptocurrencies, in a spate of phishing attacks during 2017 and 2018 that tricked people into visiting replica websites of popular cryptocurrency exchanges. US authorities jail two Russian men Our advice? Stay well aware from anything phishy by only using the Zumo app to exchange cryptocurrencies. 5. Investing in Ponzi schemes Ponzi schemes have been around for a long time and they don’t seem to be going anywhere anytime soon, especially when it comes to cryptocurrency. Ponzis work by convincing you to invest in something with the promise of sizable returns on your investment and little or no risk. Your returns are made with new money when fresh investors are lured into the scheme, creating a pyramid of promised returns and deceit. Most of the time there is, in reality, little or even no business development going on in the background and ultimately these schemes collapse when the founders disappear with all of your funds. Promising a 30% return to investors, the has collected over to date. most recent high profile Ponzi scheme $4 million Conclusion When it comes to cryptocurrency, it’s still a jungle out there. With the total amount of money stolen in crypto scams to date well into the billions of dollars, the utmost care needs to be taken when dipping your toes into the world of cryptocurrency. It isn’t all doom and gloom though; pioneering tech companies around the world are working hard to address these issues head on, and with great effect. Leading the charge is , a company that offers people a way to get involved in cryptocurrencies without the risks, with a secure and easy to use non-custodial cryptocurrency wallet. Zumo and all encryption performed locally, the Zumo app offers users an airtight and easy to use solution for buying, selling, storing, sending and spending both cryptocurrencies and traditional money. With brute force attack protection Discover how Zumo are leading the way into the future of smart, safe and secure money at https://zumo.money.

Top 5 Ways You Lost Your Crypto in 2020

Decentralized Betting Protocol receives Investment from Institutions Azuro, a decentralized betting protocol, raised $3.5 million in a seed round from Gnosis, Flow Ventures, , Ethereal Ventures, , Arrington Capital, and Meta Cartel Ventures. The investment comes in the light of the increase in trading of prediction instruments such as derivatives on , Binance and FTX. Azuro is a decentralized betting protocol that uses smart contracts to provide an innovative solution for liquidity supply and distribution, which uses pooled liquidity to scale prediction markets, similar to how pooled liquidity grew lending protocols like Aave or Compound and DEXs like Uniswap. Consequently, as betting becomes more transparent and trustworthy, the breadth of betting events, markets, and user experience remains unrivalled. Polymorphic Capital Delhi Digital ByBit The DAO will oversee the development of the betting protocol, its ecosystem economics, treasury administration, and dispute resolution, among other things. Decentralizing the betting sector and its administration can result in a more equitable atmosphere, removing the well-deserved stigma that betting holds in countries and communities. Understanding Peer-to-Peer Betting Earlier, the peer-to-peer architecture of prediction markets prevents effective liquidity allocation. Liquidity providers must manually launch markets, set odds, and seed liquidity for each market they create on prediction markets. The liquidity is thus market-specific, implying that the LP bears betting risk associated with that betting market. All of this is due to the enormous lift that LPs are required to provide, and as a consequence, there is little liquidity and hence little betting. Furthermore, prediction markets are inefficient for bets with three or more outcomes. Most people are stuck with YES/NO marketplaces, which are woefully inadequate, particularly in sports. Another example of a lack of product depth is an “accumulator,” one of the most common bets (and the most popular among casual players). Bettors who put numerous bets on the same slip will get a substantially higher potential payout if all of their bets are successful. Instead of employing various pools that are event and market-specific, core liquidity is provided via a single pool. Liquidity providers are not needed to build markets manually and are not subject to risk particular to betting markets. Instead, the risk is dispersed throughout all betting markets on the protocol, significantly reducing risk for the LPs. For LPs, the procedure is standard DeFi – no human effort is required, and no special knowledge is necessary to supply liquidity. The yield on the protocol for LPs, on the other hand, is completely unrelated to the status of DeFi or financial markets in general. Yield is provided by the Liquidity Pool’s profitability, which is earned by betting, allowing liquidity providers and farmers in DeFi to become comparable to the “house” in the classic bookmaking model. The Rise of Online Sports Betting Industry & Crypto In 1996, roughly 15 companies offered sports betting, which was the start of online sports betting. It is now one of the most popular pastimes on the planet, and with the advancement of technology, it has never been simpler. Despite more localized control, the sector has expanded dramatically and is worth more than $100 billion each year. Like the United States, large economies have long forbidden sports betting, but in 2018, the US chose to reconsider its position. This resulted in a massive increase in sports betting income, with the United States alone taking in $1.5 billion in 2020 — mid-pandemic, and with just a few states participating. Many states have enacted sports betting laws as of 2022. Despite strict regulation and a high barrier to entry, the number of online bookies is increasing year after year. By 2028, the betting industry is expected to be worth $140 billion. However, significant innovations have only been related to broader technological leaps in its more than 20 years online — the introduction of smartphones led to mobile betting, which fueled dramatic growth; a few new forms of betting emerged, such as eSports and crypto-based gambling, which has been growing rapidly with innovations, such as bet cashouts and bonuses paid in crypto to expand the industry. Don’t forget to like this story!

Delphi Digital, Arrington Capital, Gnosis and Flow Ventures Invest $3.5 million in Azuro

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures G ADDITIONAL TABLES AND FIGURES This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures G ADDITIONAL TABLES AND FIGURES

HyperTransformer: G Additional Tables and Figures

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures F VISUALIZATION OF THE GENERATED CNN WEIGHTS. Figures 9 and 10 show the examples of the CNN kernels that are generated by a single-head, 1- layer transformer for a simple 2-layer CNN model with 9 × 9 stride-4 kernels. Different figures correspond to different approaches to re-assembling the weights from the generated slices: using “output” allocation or “spatial” allocation (see Section 3.1 in the main text for more information). Notice that “spatial” weight allocation produces more homogeneous kernels for the first layer when compared to the “output” allocation. In both figures we show the difference of the final generated kernels for 3 variants: model with both layers generated, one generated and one trained and both trained. Trained layers are always fixed for the inference for all the episodes, but the generated layers vary, albeit not significantly. In Figures 11 and 12 we show the generated kernels for two different episodes and, on the right, the difference between them. It appears that the generated convolutional kernel change withing 10 − 15% form episode to episode. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures F VISUALIZATION OF THE GENERATED CNN WEIGHTS. Figures 9 and 10 show the examples of the CNN kernels that are generated by a single-head, 1- layer transformer for a simple 2-layer CNN model with 9 × 9 stride-4 kernels. Different figures correspond to different approaches to re-assembling the weights from the generated slices: using “output” allocation or “spatial” allocation (see Section 3.1 in the main text for more information). Notice that “spatial” weight allocation produces more homogeneous kernels for the first layer when compared to the “output” allocation. In both figures we show the difference of the final generated kernels for 3 variants: model with both layers generated, one generated and one trained and both trained. Trained layers are always fixed for the inference for all the episodes, but the generated layers vary, albeit not significantly. In Figures 11 and 12 we show the generated kernels for two different episodes and, on the right, the difference between them. It appears that the generated convolutional kernel change withing 10 − 15% form episode to episode.

HyperTransformer: F Visualization of The Generated CNN Weights

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures C ADDITIONAL SUPERVISED EXPERIMENTS While the advantage of decoupling parameters of the weight generator and the generated CNN model is expected to vanish with the growing CNN model size, we compared our approach to two other methods, LGM-Net (Li et al., 2019b) and LEO (Rusu et al., 2019), to verify that our approach can match their performance on sufficiently large models. For our comparison with the LGM-Net method, we used the same image augmentation technique that was used in Li et al. (2019b) where it was applied both at the training and the evaluation stages (Ma, 2019). We also used the same CNN architecture with 4 learned 64-channel convolutional layers followed by two generated convolutional layers and the final logits layer. In our weight generator, we used 2-layer transformers with local feature extractors that relied on 48-channel convolutional layers and did not use any global features. We trained our model in the end-to-end fashion on the MINIIMAGENET 1-shot-5-way task and obtained a test accuracy of 69.3% ± 0.3% almost identical to the 69.1% accuracy reported in Li et al. (2019b). We also carried out a comparison with LEO by using our method to generate a fully-connected layer on top of the TIEREDIMAGENET embeddings pre-computed with a WideResNet-28 model employed by Rusu et al. (2019). For our experiments, we used a simpler 1-layer transformer model with 2 heads that did not have the final fully-connected layer and nonlinearity. We also used L2 regularization of the generated fully-connected weights setting the regularization weight to 10−3. As a result of training this model, we obtained 66.2% ± 0.2% and 81.6% ± 0.2% test accuracies on the 1-shot-5-way and 5-shot-5-way TIEREDIMAGENET tasks correspondingly. These results are almost identical to 66.3% and 81.4% accuracies reported in Rusu et al. (2019). This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures C ADDITIONAL SUPERVISED EXPERIMENTS While the advantage of decoupling parameters of the weight generator and the generated CNN model is expected to vanish with the growing CNN model size, we compared our approach to two other methods, LGM-Net (Li et al., 2019b) and LEO (Rusu et al., 2019), to verify that our approach can match their performance on sufficiently large models. For our comparison with the LGM-Net method, we used the same image augmentation technique that was used in Li et al. (2019b) where it was applied both at the training and the evaluation stages (Ma, 2019). We also used the same CNN architecture with 4 learned 64-channel convolutional layers followed by two generated convolutional layers and the final logits layer. In our weight generator, we used 2-layer transformers with local feature extractors that relied on 48-channel convolutional layers and did not use any global features. We trained our model in the end-to-end fashion on the MINIIMAGENET 1-shot-5-way task and obtained a test accuracy of 69.3% ± 0.3% almost identical to the 69.1% accuracy reported in Li et al. (2019b). We also carried out a comparison with LEO by using our method to generate a fully-connected layer on top of the TIEREDIMAGENET embeddings pre-computed with a WideResNet-28 model employed by Rusu et al. (2019). For our experiments, we used a simpler 1-layer transformer model with 2 heads that did not have the final fully-connected layer and nonlinearity. We also used L2 regularization of the generated fully-connected weights setting the regularization weight to 10−3. As a result of training this model, we obtained 66.2% ± 0.2% and 81.6% ± 0.2% test accuracies on the 1-shot-5-way and 5-shot-5-way TIEREDIMAGENET tasks correspondingly. These results are almost identical to 66.3% and 81.4% accuracies reported in Rusu et al. (2019).

HyperTransformer: C Additional Supervised Experiments

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures D DEPENDENCE ON PARAMETERS AND ABLATION STUDIES Most of our parameter explorations were conducted for OMNIGLOT dataset. We chose a 16-channel model trained on a 1-shot-20-way OMNIGLOT task as an example of a model, for which just the logits layer generation was sufficient. We also chose a 4-channel model trained on a 5-shot-20-way OMNIGLOT task for the role of a model, for which generation of all convolutional layers proved to be beneficial. Figures 5 and 6 show comparison of training and test accuracies on OMNIGLOT for different parameter values for these two models. Here we only used two independent runs for each parameter value, which did not allow us to sufficiently reduce the statistical error. Despite of this, in the following, we try to highlight a few notable parameter dependencies. Note here that in some experiments with particularly large feature or model sizes, training progressed beyond the target number of steps and there could also be overfitting for very large models. Number of transformer layers. Increasing the number of transformer layers is seen to be particularly important in the 4-channel model. The 16-channel model also demonstrates the benefit of using 1 vs 2 transformer layers, but the performance appears to degrade when we use 3 transformer layers. Local feature dimension. Particularly low local feature dimension can be seen to hurt the performance in both models, while using higher local feature dimension appears to be advantageous cases except for the 32-dimensional local feature in the 4-channel model. Embedding dimension. Particularly low embedding dimension of 16 can be seen to hurt the performance of both models. Number of transformer heads. Increasing the number of transformer heads leads to performance degradation in the 16-channel model, but does not have a pronounced effect in the 4-channel model. Shared feature dimensions. Removing the shared feature, or using an 8-dimensional feature can be seen to hurt the performance in both cases of the 4- and 16-channel models. Transformer architecture. While the majority of our experiments were conducted with a sequence of transformer encoder layers, we also experimented with an alternative weight generation approach, where both encoder and decoder transformer layers were employed (see Fig. 4). Our experiments with both architectures suggest that the role of the decoder is pronounced, but very different in two models: in the 16-channel model, the presence of the decoder increases the model performance, while in the 4-channel model, it leads to accuracy degradation. Inner transformer embedding sizes. Varying the ν parameter for different components of the transformer model (key/query pair, value and inner fully-connect layer size), we quantify their importance on the model performance. Using very low ν for the value dimension hurts performance of both models. The effect of key/query and inner dimensions can be distinctly seen only in the 4-channel model, where using ν = 1 or ν = 1.5 appears to produce the best results. Weight allocation approach. Our experiments with the “spatial” weight allocation in 4- and 16-channel models showed slightly inferior performance (both accuracies dropping by about 0.2% to 0.4% in both experiments) compared to that obtained with the “output” weight allocation method. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures D DEPENDENCE ON PARAMETERS AND ABLATION STUDIES Most of our parameter explorations were conducted for OMNIGLOT dataset. We chose a 16-channel model trained on a 1-shot-20-way OMNIGLOT task as an example of a model, for which just the logits layer generation was sufficient. We also chose a 4-channel model trained on a 5-shot-20-way OMNIGLOT task for the role of a model, for which generation of all convolutional layers proved to be beneficial. Figures 5 and 6 show comparison of training and test accuracies on OMNIGLOT for different parameter values for these two models. Here we only used two independent runs for each parameter value, which did not allow us to sufficiently reduce the statistical error. Despite of this, in the following, we try to highlight a few notable parameter dependencies. Note here that in some experiments with particularly large feature or model sizes, training progressed beyond the target number of steps and there could also be overfitting for very large models. Number of transformer layers. Increasing the number of transformer layers is seen to be particularly important in the 4-channel model. The 16-channel model also demonstrates the benefit of using 1 vs 2 transformer layers, but the performance appears to degrade when we use 3 transformer layers. Number of transformer layers. Local feature dimension. Particularly low local feature dimension can be seen to hurt the performance in both models, while using higher local feature dimension appears to be advantageous cases except for the 32-dimensional local feature in the 4-channel model. Local feature dimension. Embedding dimension. Particularly low embedding dimension of 16 can be seen to hurt the performance of both models. Embedding dimension. Number of transformer heads. Increasing the number of transformer heads leads to performance degradation in the 16-channel model, but does not have a pronounced effect in the 4-channel model. Number of transformer heads. Shared feature dimensions. Removing the shared feature, or using an 8-dimensional feature can be seen to hurt the performance in both cases of the 4- and 16-channel models. Shared feature dimensions. Transformer architecture. While the majority of our experiments were conducted with a sequence of transformer encoder layers, we also experimented with an alternative weight generation approach, where both encoder and decoder transformer layers were employed (see Fig. 4). Our experiments with both architectures suggest that the role of the decoder is pronounced, but very different in two models: in the 16-channel model, the presence of the decoder increases the model performance, while in the 4-channel model, it leads to accuracy degradation. Transformer architecture. Inner transformer embedding sizes. Varying the ν parameter for different components of the transformer model (key/query pair, value and inner fully-connect layer size), we quantify their importance on the model performance. Using very low ν for the value dimension hurts performance of both models. The effect of key/query and inner dimensions can be distinctly seen only in the 4-channel model, where using ν = 1 or ν = 1.5 appears to produce the best results. Inner transformer embedding sizes. Weight allocation approach. Our experiments with the “spatial” weight allocation in 4- and 16-channel models showed slightly inferior performance (both accuracies dropping by about 0.2% to 0.4% in both experiments) compared to that obtained with the “output” weight allocation method. Weight allocation approach.

HyperTransformer: D Dependence On Parameters and Ablation Studies

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com;
(2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com;
(3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction
Problem Setup and Related Work
HyperTransformer
Experiments
Conclusion and References
A Example of a Self-Attention Mechanism For Supervised Learning
B Model Parameters
C Additional Supervised Experiments
D Dependence On Parameters and Ablation Studies
E Attention Maps of Learned Transformer Models
F Visualization of The Generated CNN Weights
G Additional Tables and Figures A EXAMPLE OF A SELF-ATTENTION MECHANISM FOR SUPERVISED LEARNING Self-attention in its rudimentary form can implement a cosine-similarity-based sample weighting, which can also be viewed as a simple 1-step MAML-like learning algorithm. This can be seen by considering a simple classification model [5] here we use only local features hφl (ei) of the sample embedding vectors ei This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. This paper is available on arxiv under CC 4.0 license. Authors: (1) Andrey Zhmoginov, Google Research & {azhmogin,sandler,mxv}@google.com; (2) Mark Sandler, Google Research & {azhmogin,sandler,mxv}@google.com; (3) Max Vladymyrov, Google Research & {azhmogin,sandler,mxv}@google.com. Table of Links Abstract and Introduction Problem Setup and Related Work HyperTransformer Experiments Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights G Additional Tables and Figures Abstract and Introduction Abstract and Introduction Problem Setup and Related Work Problem Setup and Related Work HyperTransformer HyperTransformer Experiments Experiments Conclusion and References Conclusion and References A Example of a Self-Attention Mechanism For Supervised Learning A Example of a Self-Attention Mechanism For Supervised Learning B Model Parameters B Model Parameters C Additional Supervised Experiments C Additional Supervised Experiments D Dependence On Parameters and Ablation Studies D Dependence On Parameters and Ablation Studies E Attention Maps of Learned Transformer Models E Attention Maps of Learned Transformer Models F Visualization of The Generated CNN Weights F Visualization of The Generated CNN Weights G Additional Tables and Figures G Additional Tables and Figures A EXAMPLE OF A SELF-ATTENTION MECHANISM FOR SUPERVISED LEARNING Self-attention in its rudimentary form can implement a cosine-similarity-based sample weighting, which can also be viewed as a simple 1-step MAML-like learning algorithm. This can be seen by considering a simple classification model [5] here we use only local features hφl (ei) of the sample embedding vectors ei

AZUMO

azumo.com

-- employees

Since n.d.

COMPANY RANKING

AZUMO

EVERGREEN INDEX #2313

Representing this company? Click here to claim and customize this page!

Azumo WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AZUMO

azumo.com

-- employees

Since n.d.

COMPANY RANKING

AZUMO

EVERGREEN INDEX #2313

RELATED COMPANIESRANK

STORIES ABOUT AZUMO

Representing this company? Click here to claim and customize this page!

Azumo WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!