Get the most recent info and news about Duelit on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Duelit is a social mobile gaming company that allows users to win money while playing games. The Duelit app allows users to challenge friends or participate in tournaments to win one of the grand cash prizes.

Duelit

duelit

Bad broadband is more than just a mild inconvenience. We know how important it is to have a strong and stable connection, good customer service when things go wrong, and the reassurance that you're getting value for money. That's why we survey thousands of ...

Best and worst broadband providers 2023

Please add a valid postcode or town. store Delivery FREE FAST delivery on orders over £40 Checking Stock... store collection Free collection from store Checking Stock...

DUALIT Lite 72015 Jug Kettle - White

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details A DUALITY BETWEEN DOWNWEIGHTED RESIDUALS AND RESTRICTING UPDATES IN LINEAR LAYERS In Sec. 4.1, we motivated our reparameterisation of the value and projection parameters, Eq. (6), through a duality between downweighted residuals branches and restricting parameter updates (materialised through smaller learning rates) in linear layers. This is a relatively simple argument, found elsewhere in the literature e.g. Ding et al. (2023), which we outline here for completeness. We suppose we have a (differentiable) loss function L(W), which is a function of some parameter matrix W. We consider taking a gradient step to minimise L, with learning rate ηW from initialisation W0. This would give new parameters W1: This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details A DUALITY BETWEEN DOWNWEIGHTED RESIDUALS AND RESTRICTING UPDATES IN LINEAR LAYERS In Sec. 4.1, we motivated our reparameterisation of the value and projection parameters, Eq. (6), through a duality between downweighted residuals branches and restricting parameter updates (materialised through smaller learning rates) in linear layers. This is a relatively simple argument, found elsewhere in the literature e.g. Ding et al. (2023), which we outline here for completeness. We suppose we have a (differentiable) loss function L(W), which is a function of some parameter matrix W. We consider taking a gradient step to minimise L, with learning rate ηW from initialisation W0. This would give new parameters W1: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

A Duality Between Downweighted Residual and Restricting Updates In Linear Layers

, a leading name in the global crypto sportsbook and casino gaming arena, is thrilled to unveil a groundbreaking collaboration with one of football's most illustrious talents, . WILLEMSTAD, Curaçao, November 29th, 2023/Chainwire/-- Duelbits Luis Suárez This partnership celebrates Suarez's remarkable legacy, including his victories in the Champions League, Copa America, and securing the Golden Boot on two occasions. Luis Suárez, affectionately known as El Pistolero, has left an indelible mark on European football, with storied tenures at Barcelona, Liverpool, Atletico Madrid, and Ajax, and is presently dazzling fans with Gremio in Brazil. A talisman for Uruguay's national team, Suarez has netted over 500 goals, earning him a place among the pantheon of football's greatest strikers. This collaboration signifies a major milestone for Duelbits, as it aligns with a sports figure who embodies the same values of excellence and innovation that have driven Duelbits to the forefront of the industry. As Duelbits continues to foster prestigious partnerships, including with the English Premier League's and , bringing Luis Suárez on board marks another strategic move in expanding its global presence. Aston Villa Argentina's national team Throughout this partnership, Suárez will champion Duelbits' campaigns, lending his image to a brand synonymous with superior quality and unmatched rewards. His professional ethos mirrors Duelbits' commitment to offering a secure, trustworthy, and leading-edge crypto sportsbook and casino experience, further solidifying the brand's identity and its resonance with a discerning global audience. Duelbits is set to amplify this alliance with an enticing welcome offer, providing new members a $100 credit to kickstart their betting journey on any desired outcome upon signing up, thereby inviting fans to not just witness, but also partake in the thrill of the game. In a statement, Luis Suárez shared his excitement about the collaboration, "Joining forces with Duelbits is a proud moment for me. It's more than a partnership; it's a shared vision of excellence and trust. I look forward to what we will achieve together." Marco Pinnisi, CEO of Duelbits, echoed Suarez's sentiments, "We are ecstatic to welcome Luis Suárez to the Duelbits family. His unparalleled achievements and our platform's promise of an exceptional betting experience are a perfect match. We're paving a new path for the future of crypto gambling together." Through this collaboration, Duelbits is poised to continue offering enthusiasts an elevated gaming experience, where innovation and top-tier entertainment converge seamlessly. About Duelbits Duelbits is a leading name in the crypto gaming industry, offering a premium platform tailored for bold and passionate cryptocurrency enthusiasts. With an exclusive library of original games, Duelbits redefines the gaming experience through innovation, personalisation, and unparalleled rewards. The platform prioritises security and trustworthiness, recognising and rewarding every player's unique journey. Our unwavering commitment is to provide the most enriching iGaming experience in the market, defined by fast payments and seamless accessibility. At Duelbits, we pride ourselves on offering the best odds, monumental bonuses, including up to a whopping 50% in Rakeback bonuses, and a uniquely personalised user experience. Positioned at the forefront of embracing cutting-edge innovations, we hold a steadfast belief in advanced technologies, AI, and cryptocurrencies. Each click is a purposeful stride towards an extraordinary adventure. | | | | | Website X Instagram Telegram YouTube d Discor Contact Head of Marketing Caroline Attard caroline.attard@duelbits.com This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program . here

Duelbits Announces Landmark Collaboration With Football Icon Luis Suárez

https://youtu.be/1GCnH8wMzWI?si=hQLp7z_5R1qmKZME&embedable=true “Man is not truly one, but truly two.” is one of my favorite books. The idea that each of us has two people inside, constantly fighting for dominance, wasn’t new, even when Robert Louis. Strange Case of Dr. Jekyll and Mr. Hyde Stevenson penned the novella in 1886. It goes much further back than that. Descartes, in the 1600s, and even Plato two millennia earlier, wrote extensively about dualism, though their philosophies were much more complicated than what I’ll examine today. I was reminded of the phrase “duality of man” when recently talking to Steven Kotler, executive director of the Flow Research Collective. You know him well—author of , , Tomorrowland Bold , and a handful of other best-sellers. Abundance Steven was speaking about the team-building process and mentioned how every time you make a new hire, you’re actually adding two people: The person they are when everything is great The person they are when they’re terrified Without fully understanding this, you can quickly find yourself in a burning building with no fire escape. A sinking ship with no crew. The Misrepresentation of Potential Too often, we define “potential” only in the positive sense. Even if you type it into Google, you’ll find this: Potential: latent qualities or abilities that may be developed and lead to future success or usefulness. But that’s not really accurate. Potential is a knife that cuts both ways. There is potential for future success and usefulness, but also for fear and poor decision-making. Each of us has the capacity to add value to a company, but the necessary environment changes drastically from person to person. I might fare best in an asynchronous environment where I can work uninterrupted for hours at a time. Another teammate might need people around them to act as a sounding board. The real question, then, is how well each of us works when the environment perfect. isn’t Does Pressure Drive Performance? There is a lot of literature about performance under pressure and how . Daniela Kaufer of the University of California, Berkeley says, “Some amounts of stress are good to push you just to the level of optimal alertness, behavioral and cognitive performance.” some stress can be good for you However, this may only be true in effort-based tasks, not skill-based ones. other research suggests For the latter, people either “choke,” “clutch,” or, frustratingly, neither. It would be a lot easier if human performance was cut and dry. What has been established is that too much stress, especially in the workplace, almost always results in burnout, decreased motivation, and lower productivity. But that is on the aggregate and doesn’t look at specific situations. How can we know how an individual employee will react in a frightening situation? The Interview Trap I’ve been to some frightening interviews before. You are brought into a large conference room with six or seven people on the other side of the table, all rattling off questions before you can breathe. But a savvy job-seeker will be ready with interview prep and rehearsed answers. It’s stressful, sure, but it doesn’t really get you the answers you need as a manager. I want to know if this person will be creative and communicative—perform at a high level even when frightened of the outcome. Steven has a trick that he shared with me. When he is hiring someone, he’ll give them an impossible task. I don’t mean the infamous Google genius billboard from Silicon Valley circa 2004 that read: {first 10-digit prime found in consecutive digits of e}.com. That was just a test to weed out the unsuitable applicants. No, I mean a genuinely task. Something that no one would be able to complete on time or to the specifications given. impossible The idea isn’t to get them to try and fail. It’s to see who will return and admit they can’t do it. Who will stride into that interview room, afraid of rejection, and be honest with you? Only when they’re frightened of losing the opportunity will you get a small glimpse at that other person, the one they’ll be when things aren’t going well. If that person is still honest, hard-working, and willing to take direction—it sounds like a hire to me! Getting Mr. Hyde His Doctorate Remember that even though everyone has two sides, you can still work on bringing them closer together. Not every Hyde has to be a monster. There are methods to increase your peak performance and get yourself into a “flow state” more often, even in stressful situations. For a more detailed discussion of what flow is, check out my —he is a fascinating mind that I can’t wait to have on the podcast again. whole interview with Steven But for now, let’s go through three simple steps. Daily Gratitude Starting or ending your day with a daily gratitude practice can profoundly affect your mental health and, in turn, your performance at work. This practice should involve consciously thinking about or writing down what you are grateful for. It can include anything from small everyday pleasures to meaningful personal relationships or significant events. This simple act to focus on positive aspects rather than engaging in negative thought patterns. rewires your brain Practicing gratitude regularly can decrease stress levels, improve resilience, and create a more adaptable mindset in the face of challenges. Regular Exercise I know, “eat right and work out” is tired advice. But it works! I’ll keep shouting it from the rooftops. Adopting a regular exercise routine is beneficial not just for your physical health but for your mental health as well. Physical activity , the brain’s natural mood boosters. increases the production of endorphins Moreover, regular exercise helps sleep patterns, improves self-confidence, and increases relaxation. Regular physical exercise can make you better equipped to handle stress and remain calm during challenging situations, whether it’s a short walk, a demanding workout, or an activity like dancing, yoga, or swimming. Mindful Practices In a world that’s constantly demanding attention and speed, practicing mindfulness can help manage stress and induce a state of calm. Practices like meditation, taking slow, deep breaths, or simply taking a break from screens to observe your surroundings can be significantly beneficial. These allow you to stay in the present moment rather than worrying about future tasks or dwelling on past events. Mindfulness can enhance your focus, increase emotional resilience, and improve decision-making skills, which are invaluable in stressful or high-pressure situations. One thing I’ve started doing is spending an entire day completely removed from work. Don’t look at your email, don’t respond to that Slack message, or even enter your home office. It’s Saturday—get outside! By incorporating these practices into your daily routine and encouraging your team to do the same, you can foster a more stress-resilient, mindful, and efficient work environment. Learning to manage your dualities effectively can improve individual performance and the team’s overall productivity. The Fear Caveat I don’t want anyone to get the wrong idea here. I’m not saying you should go scare the living daylights out of your team as some sort of twisted motivational exercise. Quite the opposite, actually. Confronting fear head-on can be a powerful growth experience, but if you’re constantly knee-deep in stress, it tends to backfire pretty quickly. We’re all humans, and a steady diet of dread can be traumatizing. So, while it’s important to understand your team members’ (and your own) two-sided nature, everything needs balance. Foster a work environment that’s encouraging, supportive, and safe. When the tough times inevitably come, your team will be more ready to face them head-on, with good coping mechanisms at their disposal. Final Thoughts In sports, a prospect’s potential is usually listed in two ways: ceiling and floor. Ceiling means the very best performance they could eventually get to. The floor is what is guaranteed every day. Sometimes, it’s worth shooting for that high-ceiling player, but don’t underestimate the importance of a high floor. When the you-know-what hits the fan, you will want some stability on your team. If you enjoyed this article, I’d love to hear from you. Write me at or at me , and I’ll do my best to get back to everyone. scott@scottdclary.com tweet @ScottDClary Also published . here

When You Hire One Person, You're Actually Hiring Two: The Duality in Every Hire 

When you optimise your web app, your goal is to make the experience better for the user; this means usually ‘faster’ by transferring and parsing less data. But caution: the same web app can cause Cumulative Layout Shift (CLS) on slower connections but runs without CLS on faster connections. If you’d like a refresher about Core Web Vitals, I explained them with GIFs in . this post TL;DR: slower connections result in CLS when lazy loading components that you wouldn’t see on wifi connections. can Either don’t lazy load the component at all or wait for the js file to be loaded and mounted. The Duality We assume that a web app loads the same on slower connections, just . Unfortunately, that’s not always the case with lazy loadable components. slower With lazy loadable components we deal with two : asyncnesses async API responses (JSON) async lazy loading components (JS) What if the API responses (1) are faster than the dynamically loaded JS (2)? What if you lazy load a component that sits in the middle of your web content? The answer to these questions you see in the screen capture above: Google will punish you with CLS. CLS Measurement I’ve seen a lot of confusion about Core Web Vitals, especially CLS. Unlike other Core Web Vitals, CLS is continuously measured and added to the score. For a classic SPA web app this means that Google will keep the CLS score on a per-route basis. cumulatively CLS has the following characteristics: After each route change, the CLS resets to 0 After any user interaction, you get a grace period of 500ms where CLS is taken into account not : Chrome users send Core Web Vitals metrics to Google directly. It’s not a Googlebot that captures these metrics while crawling the site. Measurements by real users These real user measurements are collected as and flow into Google’s . Field Data CrUX report That means you need to take the real world into account: If you have a lot of traffic from India, but your servers are on the other end of the world, your LCP (Largest Contentful Paint) will probably suffer. If you have a lot of traffic from China, it’s likely that some services are blocked by Chinese ISPs and won’t load. This could cause unwanted CLS within your content. Solutions We need to be in full control of what to display to the user at what time. With duality in mind, we need to know the following: are API requests still loading? are async components still loading? A skeleton loader is an ideal way to wait until both, API requests and async components, are ready. Solution #1: Don’t lazy load components at all The quickest and least error prone solution would be to pass on lazy loading components. In most cases the saved kilobytes through lazy loading doesn’t justify the CLS that it might cause. If your performance budget allows it, go with this solution. Let’s assume we have a Vue web app with 10% logged-in users. function render() {
	// not all users require to download and render HugeComponent
	if (isLoggedIn) {
		return <HugeComponent />;
	}
} Without code splitting we’d send the JS of to 90% of the users that don’t need it. This can affect LCP and FID. <HugeComponent> With code splitting we’d pack the JS of into the additional-comps.js chunk and only send it over the wire when it’s needed. <HugeComponent> // without code splitting
import HugeComponent from '@/components/HugeComponent';

// with code splitting
const HugeComponent = () => (await import(
	/* webpackChunkName: "additional-comps" */ 
	'@/components/HugeComponent')
).default; There are two criteria that help you decide if you should lazy load a component or not: size of <HugeComponent> how often would be rendered for users <HugeComponent> If you come to the conclusion that your performance budget is tight and you need to lazy load the component, see solution #2. Solution #2: Await loading and mounting of component If your component isn’t used for the majority of users and it would increase the bundle by a lot, have a look at this solution. Wait for async components to be lazily loaded and mounted can be tricky. You need to render the component but the mounting happens later. Here’s a gist of how it could be done. function Parent({ isLoggedIn }) {
	const [isLoading, setLoading] = useState(false);
	// important: only hide the children with display: none.
	// if we used if-else, we'd never load the lazy loadable component;
	// and then we'd never change isLoading with its callback.
	const styles = { display: isLoading ? 'none' : 'block' };
	
	return (
		{/* HugeComponent is lazy loaded, because not all users will need it */}
		(isLoggedIn && <HugeComponent 
			style={ styles } 
			mounted={() => setLoading(true)}
		/>)

		{/* display skeleton element for time of loading */}
		(isLoading && <div className="skeleton-loader"></div>)
	);
}


function HugeComponent(props: { mounted: () => void }) {
	const [dataA, setData] = useState(null);

	useEffect(async () => {
		const result = await lib.request();
		setData(result);
		// tell the parent component that everything is ready to be fully rendered
		props.mounted();
	}, []);

	return (/* ... */);
} If we didn’t use for the loading state on the , we’d never trigger the loading of the async component. Thus, Line 28 would never be reached and the state would stay on forever. display: hidden <HugeComponent> isLoading false Conclusion When you lost green URLs in the due to CLS and you can’t reproduce it yourself, try debugging your web app with a slower connection. Google Search Console So if you’re using lazy loadable components, chances are high that might be a victim of the duality of CLS. If you found this post interesting please leave a ❤️ on and consider following my 🎢 journey about #webperf, #nonfiction books, #buildinpublic, and #frontend matters . this tweet on Twitter

Using Lazy Loadable Components Can Lead to CLS Duality and Slow Connections

In this thread, we chat to Nicole Nguyen, the co-founder, and CMO of Duelist King, an NFT card game that has just recently released its closed alpha. The closed alpha was made available to 2000, lucky whitelisted. We had the pleasure of chatting to Nicole about Duelist King, its closed alpha, and the future of the NFT game. slogging You can sign up to be next in line for the closed alpha here: https://forms.gle/ZrWKTwGYY6wRv38L9 Also make sure to check out their playbook, which was released alongside the alpha: https://blog.duelistking.com/dk-playbook-a-guide-to-getting-started-with-the-demo-31430 This Slogging thread by Jack Boreham, Nicole Nguyen, Mónica Freitas, SuperSaiyanProgramming, Zaeem Shoaib, Sara Pinto and Limarc Ambalina occurred in slogging's official #amas channel, and has been edited for readability. Jack Boreham Mar 9, 2022, 2:01 PM Hey @channel, please join me in welcoming our next AMA guest, Nicole Nguyen. Nicole is the CMO and one of the co-founders of Duelist King, an NFT card game that has recently released its closed alpha. The game is shaking up the NFT gaming world. She is also our first returning AMA guest, so please give her a warm returning welcome. Make sure to ask her Qs about the Duelist King alpha for a chance at winning $200 worth of NFTs. Please feel free to ask Nicole anything about: The Duelist King Alpha version The future ambitions of Duelist King What Duelist King has achieved over the last six months NFTs, gaming, and Duelist King in general Jack Boreham Mar 9, 2022, 2:07 PM Hi Nicole Nguyen, a pleasure to have you with us. To refresh our audience, can you tell us a bit about yourself and your role at Duelist King? Nicole Nguyen Mar 9, 2022, 2:12 PM Hi Jack Boreham and everyone thanks for having me. Great meeting you all and sharing about our NFT card game Duelist King with the closed alpha just fresh from oven B-) Nicole Nguyen Mar 9, 2022, 2:12 PM We have an awesome team of 35 people in Vietnam, India, the US working on a multi-chain NFT card game that supports Win2Earn narrative with first-class gameplay and design. Some of our recent milestones October 2021 - launch on Pancake Swap via an Initial Farm Offerings with 640M USD committed to the public sales Sept 2021 and Jan 2022 - 500,000 NFTs in total sold out. Also, we are voted the second-best blockchain game developer by Hackernoon readers 😉 March 2022 - 23,000 registrants to the closed alpha and 2,000 people are whitelisted for 1.0 version Jack Boreham Mar 9, 2022, 2:40 PM Nicole Nguyen, thanks for the great intro! Let's start with some of those milestones you mentioned. Since you were last here, Duelist King has sold around 500,000 worth of NFTs. How did DK manage to do this? Why do you think the cards sold so well with your community?😄 Nicole Nguyen Mar 9, 2022, 3:03 PM Our previous card sales are officially endorsed by Polygon and Fantom, two of the prominent chains that we are partnering with to realize our vision of a multi-chain NFT card game. Moreover, lots of our members also love the designs of the cards. It's not just the cards that they are getting bullish about - I believe the community is also betting on our community-driven approach - not just as gamers, they can actively participate in shaping up the gameplay and operations once the DAO governing model kicks in next year. We've been fortunate to gather early adopters/NFT holders for Duelist King and gathered 23,000 registrants for our Closed Alpha version 😄 Jack Boreham Mar 9, 2022, 3:32 PM Nicole Nguyen Wow! We definitely see that. 23,000 is a fantastic number. You mentioned, the release of your closed alpha. Let's dive into this. What is your closed alpha, and what are players experiencing when they get their hands on it? Nicole Nguyen Mar 10, 2022, 8:17 AM Jack Boreham our Closed Alpha is open for the first 2,000 most loyal and dedicated members out of 23,000 registrants. This is similar to a prototype of our game where you can dive in and trial the Player vs. Enemy combat style of the game. Players would be able to find out more about our gameplay and how to combine cards and dice to play in the game. You might need to check out our Playbook to get a head start and also sign up to stand next in line for our Closed Alpha followed by Public Alpha in April 😄 Playbook link Closed Alpha registration link: https://blog.duelistking.com/dk-playbook-a-guide-to-getting-started-with-the-demo-31430 https://forms.gle/ZrWKTwGYY6wRv38L9 Mónica Freitas Mar 9, 2022, 5:17 PM Hi Nicole Nguyen. Great to have you here! I'm still an NFT newbie. Could you explain how NFTs and Duelist Kingdom converge? Nicole Nguyen Mar 10, 2022, 8:20 AM Mónica Freitas Cards aka the most important and valuable assets in our game are issued as NFTs with different rarity level and each card is issued as a unique item which are transferrable and tradable between players and across different marketplaces. You can also authenticate the cards by checking on blockchain if they are issued by us. Blockchain plays a pivotal role in assuring the authenticity, liquidity, tradability and subsequently the value of the NFT cards in our game. SuperSaiyanProgramming Mar 9, 2022, 5:19 PM Hey Nicole happy to have you join us SuperSaiyanProgramming Mar 9, 2022, 5:21 PM I have a question about the future plans and position of the game on the market. Blockchain games are getting more popular and thus more saturated. There are big nft card games already kind of solidified in the market like gods unchained. How do you differentiate your game from those other card games to attract an audience? Nicole Nguyen Mar 10, 2022, 8:26 AM SuperSaiyanProgramming Thanks for the question and I assume you might be a heavy card gamer 😄 I think blockchain gaming has just started - it's not just about bringing mainstream games onto the blockchain. There's a whole different world of web3 and metaverse where you can build parallel or different dimensions of business model, designs, and structure of the game. Blockchain is not just about tokenization or NFTs - it does offer multitudes of value such as asset verification, incentivization, community ownership, proven fairness, and randomization. The first and foremost differentiator we are trying to make is to actually eradicate the barriers between blockchain games and mainstream games in terms of design and gameplay. We want to build a game of kickass design and intense gameplay to start with. Also, we have developed in-house infrastructure to make sure that the card distribution can be verified totally random on the blockchain, and eventually, the community will be the main owners of the game once the DAO model kicks in. Also, we do not follow the much-saturated path of Play2Earn - Duelist King aims to incentivize people through winnings and tournament prizes or Win2Earn i.e. a much more winding road yet a more sustainable direction for a game to survive and thrive on its own. Zaeem Shoaib Mar 9, 2022, 7:16 PM Hi Nicole, Great to have you here! I have 2 questions: What is your take on the Metaverse as envisioned by Meta (formerly Facebook), The Sandbox, and Decentraland? and What do you think is the future for interactive experiences such as Duelist King? There has been a lot of bad press in the mainstream media regarding NFTs in the gaming space and a lot of backlash from gamers. What is your perspective on that and what is Duelist King doing differently to endear itself to a primarily gaming audience? Nicole Nguyen Mar 10, 2022, 8:36 AM Really great questions and I think they are like industry-scope challenges 😄 Metaverse is not a novel concept and I believe each project has its own concept and vision about their metaverse. I would strongly agree with any vision that focuses on interoperability and composability of metaverse infrastructure since that's where the community, the underdogs, and smaller players can leverage and initialize their projects or ideas. Duelist King is speaking to some projects like Sandbox and competitors to Sandbox too (why not?) to see how we can bring better experiences for our gamers - at the end of the day, we think that's all that matters since we will be able to offer much more values and collaborative experiments for our gamers. NFTs and blockchain gaming are thriving - just in the last quarter alone, they account for 42% of the venture funding into blockchain projects according to research by The Block. Numerous projects have failed to prove their use cases but a lot of the projects are still focusing on building. Still, the barrier between blockchain games and games is relatively eminent when it comes to design and gameplay although Play2Earn has done a good job of enticing an influx of newbies to blockchain gaming. I think moving forward, not only us but other blockchain projects are focusing on refining gamer's experiments and finding a better and more sustainable incentives or earning models for the game to slowly transition from Play2Earn. Sara Pinto Mar 9, 2022, 10:54 PM Hi Nicole Nguyen, glad to have you here with us! How did the idea of Duelist King come up? How is the Duelist King Alpha version different from the original version? Nicole Nguyen Mar 10, 2022, 8:41 AM Sara Pinto thanks for the question and I assume you might have been able to see or lay your hands on our very first version of the game last year 🙂 Our co-founding team have got to know each other for 6 years before working on this project and we own complementary skill sets and networks in blockchain since 2016. I guess it's just about time when we convened last May and decided that we should do something about the thriving scene of NFT and gaming and of course have some fun working on games that we like. The Closed Alpha is quite different from our previous version actually. It's a huge reiteration from the previous Yu-Gi-Oh/GWENT-inspired gameplay and this time we are offering dual fun of using cards and rolling dice to decide how you can play, summon and use your decks. The random distribution of the cards and the dice component underline the fairness in the game - regardless of how much money you have, how much experiences you've got, still you will need to pray for luck to be able to win in the game. Sign up for our Closed Alpha and share with me how you feel about it 😄 https://forms.gle/ZrWKTwGYY6wRv38L9 Jack Boreham Mar 10, 2022, 10:38 AM Nicole Nguyen, that's a great playbook😂. So it seems the community is at the forefront of your game roadmap. How important would you say the community is for Duelist King? Also, how did you go about selecting members to partake? The most loyal ones, right? Nicole Nguyen Mar 11, 2022, 3:37 AM Jack Boreham the playbook is actually a collaborative experience with our community as well 😄 We invite the community to leave comment feedback, ideas, and even designs for Duelist King through one of our art contests. As mentioned, the DAO or Decentralized Autonomous Organization will be the key governing model of Duelist King next year so we are having the community familiarize themselves with having a say in how our gameplay and design play out. We are engaging with them on regular basis through Telegram, games, polls, and feedback loop for the Closed Alpha. The members whitelisted for the Closed Alpha version are the most loyal ones yes - since we actually keep score of their engagement with the community, the volume of constructive feedback contributes to the project. We also put aside a bounty pool as incentives for prospective contributors and also advocates/future owners of Duelist King as rewards for their contribution to the Duelist King community and project 😄 SuperSaiyanProgramming Mar 10, 2022, 3:36 PM Very interesting and thanks for the thorough answer! What is the barrier to entry? God's unchained starts players off with a free deck (that isn't minted/owned NFTs) but as the player wins games they can then earn cards as NFTs. In DK do you need to buy a deck to start playing or is there a free starter deck? Nicole Nguyen Mar 11, 2022, 3:43 AM SuperSaiyanProgramming we will offer the Free2Play alongside the Standard version. The F2P is for experimenting with the gameplay and comes with no rewards. The Standard version requires 22 cards at the minimum to start playing and that costs 22 USD if you buy directly from us. In our last 2 batches, we sold 500K NFTs at a flat price of 1 USD per card or 5 USD per 5 cards in one mystery box. The cool thing is you can get anywhere from a Common to a Legendary card with just 1 USD since we run on-chain random distribution of the cards and the community can verify the whole process on blockchain 😉 Jack Boreham Mar 11, 2022, 8:12 AM Nicole Nguyen, what has Duelist King learned from the closed alpha that you guys would take to new iterations of your game 😄 Nicole Nguyen Mar 11, 2022, 11:20 AM Jack Boreham we have gathered a number of feedbacks from the community - like 100 per day regarding anything from gameplay to game design. We will reflect this feedback in our upcoming  Public Alpha version to be released in April. There will be few reiterations in terms of card feature, in-game effects and gameplay so make sure you've signed up for our closed alpha and stay tuned for our public alpha B-) Mónica Freitas Mar 11, 2022, 9:06 AM Nicole Nguyen that's awesome! What's next for the Duelist Kingdom? What is the next milestone you want to achieve? Nicole Nguyen Mar 11, 2022, 11:24 AM Mónica Freitas the next milestone is definitely the Player vs. Enemy version coming in June. We are gathering momentum by engaging as much as we can with 23K closed alpha registrants, launching an incentive pool for feedback and contributions from the community, listing on centralized exchanges, and expanding our partnership with more AMAs, games, and engagements to gather around 50K registrants to Public Alpha which will turn to around 5-10K gamers to Duelist King. This is an intense genre of the game so we hope to start small yet bold with genuine enthusiasts and build a strong loyal team of Duelist King early adopters. 🔥 1 Jack Boreham Mar 11, 2022, 11:25 AM Nicole Nguyen, that's fantastic. What are you most proud about over the last six months since you were here that DK has accomplished? And what excites you most about the future of Duelist King over the next six? Nicole Nguyen Mar 14, 2022, 4:18 AM Jack Boreham There are too many things to pick from but let's say growing the team to 35 motivated and diverse people have been one of our biggest achievements so far in the last 8 months. Having a strong core team is a great foundation for us to achieve the following milestones of 500K NFTs sold out and the closed alpha launch. Over the next 6 months, the PvE beta version would be our main target so that we can convert DK enthusiasts and advocates into genuine gamers. Limarc Ambalina Mar 11, 2022, 4:17 PM Hey Nicole Nguyen as someone who is working with a blockchain game atm (Ethermore) I'm interested to hear your thoughts on how to keep communities engaged and what resources/outlets have worked best for you in terms of drumming up enthusiasm and growing the community? Secondly, when the game was being developed where did the team draw inspiration for the theme/art style? Also happy to have you here again! Nicole Nguyen Mar 14, 2022, 4:27 AM Limarc Ambalina thanks again for having us and I can't express how grateful we are for the support from you and Hackernoon team. To answer your first question, I think consistency and constant communication, and feedback loop are key to getting the community engaged. For Duelist King, we make sure that the community is well aware and informed of all information about the project and our regular engagement schemes. Also, the feedback and communication loop between us are encouraged and also incentivized so that the community can absorb the milestones and vision of Duelist King becoming a community-owned and driven project. Additionally, we were inspired by Yu-Gi-Oh and GWENT at the start of game development although we did have a number of reiterations with the game to make sure it's unique, fun, and just intense and hardcore enough for both newbies and crypto gamers. The art is inspired by a number of folklores and myths that you can observe across different cultures like vampires, werewolves, shamans, etc. Everyone can see a piece of their childhood heroes or characters in Duelist King and eventually we will allow the community to come up with the card design and also gameplay. Those who followed Duelist King might have already observed this through our art contest with Ludena Protocol back in November last year and we want to make sure this will take place more in the future so you can start to familiarize yourself as a decision-maker to Duelist King 🙂 Nicole Nguyen Mar 14, 2022, 4:29 AM Thanks a lot for having me here Jack Boreham and Limarc Ambalina, we hope to see more Duelist King supporters and advocates from Hackernoon and we welcome the HN community to partake in our activities and discussions to fine-tune Duelist King gameplay and design in the near future! Jack Boreham Mar 14, 2022, 7:57 AM Nicole Nguyen A pleasure to have you with us. To wrap up, how can people get involved with Duelist King and its future game releases? Nicole Nguyen Mar 14, 2022, 8:35 AM Jack Boreham Drop us a line, follow our Twitter and join our Telegram group to stay updated with our upcoming milestones 🙂 We hope to hear from you about anything like your favorite card, how you feel about our closed alpha, or how we can tweak our gameplay better for upcoming versions, also we have prizes to offer to constructive and/or lucky members so don't miss this out! ✅Website: ✅Twitter: ✅Telegram group: https://duelistking.com/ https://twitter.com/duelistkingnft https://t.me/DuelistKingNFT

Duelist King Releases its Closed Alpha   with Win2Earn Game Mechanics

Hey Hackers! I’m NICOLE Nguyen and I’m the Co-founder at Duelist King. First of all, a huge thank you to the HackerNoon community and staff for nominating us for a 2021 Noonies award! We’ve been nominated in the following categories please do check out these award pages and vote: Blockchain Game Developer of the Year As someone in the gaming industry, I believe that the most exciting technology of the present is blockchain because it has the potential to empower individuals and indie game studios versus big game distributors and democratize the gaming space like never before to give the community more decision making power. Learn more about my thoughts and opinions on blockchain in gaming and my journey in the tech industry via the interview below. 1. What do you do and why do you do it? (tell us your story) My name is Nicole and I have been in the blockchain space since 2016. My first 3 years with blockchain community building was filled with sweat but also joy as my former company Infinity Blockchain Labs was considered the earliest pioneer in promoting blockchain in Vietnam. I was the key organizer of the first mass-scale blockchain conference in Vietnam - Vietnam Blockchain Week for 2,500 people, the first blockchain hackathon in Vietnam - Blockathon, and 20+ meetups. My next job was to set up and run a regional blockchain media and VC called Asia Blockchain Review and Infinity Blockchain Ventures. The amazing people in the blockchain space who can come from any walk of life are the main inspiration for me to keep moving forward and go all-in with this transformative technology. 2. Tell us more about the things you create / write / manage / build! As the first NFT game powered by Dual Launch via OccamRazer IDO and PancakeSwap IFO, Duelist King is built on BSC and Cardano and supported by Fantom, Polygon and Cronos (powered by ) to provide a cross-chain supporting platform for NFTs and later Duelist King tokens. We started off Duelist King by building our own in-house Decentralized Autonomous Organization (DAO), Oracle and Random Number Generator. Also we dream of Duelist King as the first generational NFT card game that aims to transform the way people play, participate and earn in card games through top-notch game play, asset designs and advocacy for the sustainable future of #Win2Earn. crypto.com We believe the current gaming space is heavily tilted towards bigger game studios who can manipulate the gaming market for their own good. Blockchain has the potentials to democratize the gaming space and combined with NFT, undoubtedly gaming will be able to stand strong through bull and bear market as a valid use case for blockchain. 3. How did you end up on your current career path? Do you like it? My career path has always been a rough and uneasy one (you know how picky and critical the community might be sometime). Yet the feeling of being able to empower people and convince them to believe in a plethora of life-changing opportunities thanks to blockchain and technology is just something so special that you can’t find in other technologies - and in this case we have the community to feel especially thankful for! 4. What tech are you most excited or passionate about right now and why? Blockchain, blockchain and blockchain! The importance of decentralizing trust and empowering the community through collective trust can expedite technological adoption and empower us to transform a certain industry and the whole world with blockchain. Some people might have already experienced financial freedom with blockchain and on top of that, starting up a business and becoming a tech entrepreneur can turn so much more accessible and potential thanks to blockchain. 5. What tech are you most worried about right now and why? Blockchain does come with its downside - rug pulls, scams, users lacking basic knowledge to verify where to put their assets into etc. The reason being the novelty of the technology, lack of educational efforts and the lucrative businesses it can bring might explain why some people still remain skeptical with blockchain. 6. If we gave you 10 million dollars to invest in something today, what would you invest in and why? I would invest in a cross-chain infrastructure that will be able to support asset and liquidity management across different blockchains. This will help fix the chronic issue of fractionalized liquidity in blockchain and help aggregate resources and liquidity at a larger scope and pave the way for more world-changing games, DeFi and Fintech products. 7. What are you currently learning? Mindfulness meditation as it helps me see through the mirror of truth and be more aware of what is reality and what really matters. 8. What’s the best advice you’ve ever given someone? Invest in people not in papers! Numbers might deflate yet talents and values will definitely inflate. 9. What’s the best advice you’ve ever received? Go down the uncharted path till the very end as the leader instead of being a follower for the remaining of one’s life - and that’s how I found blockchain! About HackerNoon’s 2021 Noonie Awards The annual Noonie Awards celebrate the best and brightest of the tech industry, bringing together all who are making the Internet and the world of tech what it is today. Please be sure to , nominate, and vote for the people and companies who you think are making the biggest impact on the tech industry today. check out our award categories The 2021 Noonies are sponsored by: , , and . Thank you so much to these sponsors who are helping us celebrate the accomplishments of all our nominees. bybit Dottech Domains Avast

Duelist King Nominated as Blockchain Game Dev of the Year

With online card games such as Gwent and Hearthstone taking up a large chunk of the video game market, it was only a matter of time before blockchain-based card games would start gaining traction. The added value NFTs bring to players could give players an extra incentive to play games like Duelist King, where players own the NFTs of the cards they play with. Recently, the HackerNoon community spoke to Nicole Nguyen, the founder of Duelist King to learn more about the project. This thread by Limarc Ambalina, Zaeem Shoaib, Amy Shah, Mónica Freitas, Nicole Nguyen and Jack Boreham occurred in slogging's official #gaming channel, and has been edited for readability. Slogging Limarc Ambalina Sep 30, 2021, 8:46 AM Hey everyone please welcome Nicole Nguyen the founder of Duelist King, an awesome up and coming Blockchain card game that I had the pleasure of demoing. Please feel free to ask her anything about Duelist King, Blockchain games, game development, nfts, and anything else that comes to mind! Limarc Ambalina Sep 30, 2021, 8:46 AM Thanks for joining us Nicole I'd like to hear your explanation of just what Duelist King is and what sets it apart from normal online card games like Gwent or Hearthstone as well as how you differentiate yourselves from competitors like Splinterlands? Nicole Nguyen Sep 30, 2021, 3:36 PM Powered by in-house Decentralized Autonomous Organization (DAO), Oracle and Random Number Generator, Duelist King offers dual values for investors from token utilities and card sales (projected at roughly 40 million USD). Duelist King aims to transform the way people play, participate and earn in card games through top-notch game play, asset designs and advocacy for the sustainable future of Win2Earn. In short our differentiators can be recapped as follows: - Absolutely randomized, verifiable and fair distribution infrastructure infused with gambling excitement through mystery box offering - Playable demo available ready - Top-notch game designs and game play in progress - Last but definitely not least, community empowerment. - Advocacy for #Win2Earn - encouraging people to Enjoy - Engage - Earn from the game for a more sustainable business model and earnings model for the community We do not just engage the community as gamers but collectors, designers and investors are more than welcome. Also our community members can get to vote and pick their favorite game play, game design and also propose how the game operations should take place. This will follow the DAO governing model that we have designed for Duelist King and also can be leveraged for other gaming studios who wish to pursue the fair distribution and community empowerment features that we advocate. Also our team are domain experts in blockchain and also gaming side so I believe you have every reason to be bullish about us. About us: Differentiators: Zaeem Shoaib Sep 30, 2021, 8:49 AM Welcome, Nicole! I would like to ask how and in what ways the advent of Blockchain technology has changed the way developers now approach starting development on a game that incorporates that technology as opposed to traditional games? 🔥 1 Nicole Nguyen Sep 30, 2021, 3:40 PM this question might be best addressed through our platform We built Duelist King as the first showcase of DKDAO - the ultimate platform for blockchain gaming. We offer no-frill solutions for gaming studios and developers from tokenization, fundraising, DAO governing model to other infrastructures. I think the most powerful feature of blockchain is to allow crowdsourcing for projects especially in gaming sector to pull together a vast array of resources from development, financing to community engagement (also the most powerful of all). Also blockchain game functions in a more decentralized and user-centric fashion vs. distributor-centric model in traditional games. https://www.dkdao.network/ Amy Shah Sep 30, 2021, 10:58 AM Nicole Nguyen Welcome! I am sort of confused by blockchain....How would you explain Duelist King to someone like me? Also, do you think the future for NFTs is particularly bright?  If so, why? Also, what draws you do game development as a career? Do you have some prior experiences that have shaped your interest in this field? Thanks so much for your responses! So excited to learn more about your company and game. 🔥 1 Nicole Nguyen Sep 30, 2021, 3:44 PM Amy Shah I think the core values of Duelist King can be recapped in fair distribution, different playmode/earning models for gamers and engagement of creators, gamers, collectors alike. NFTs are increasingly finding its use cases and utilities in a number of industries like gaming, arts and collectibles. Adding metaverse effect, the sky's the limit for NFT growth as a whole new emerging business instead of a contender to the traditional space. We aim to issue our cards as NFTs used in game as assets and also collectibles with more rare NFTs. So I believe we have real use cases with our NFTs and we have a big base and funnel to provide utilities and offerings for newbies, mainstream gamers and those who can afford more rare NFTs and collectibles. I've chosen blockchain as my career in the last 5 years - gaming is one of the first areas ripe for disruption back in 2016 and now with NFTs, gaming is poised to be the top area for blockchain adoption. My experiences for newbies is definitely get as much exposure as you can - talk to people, join community, get your voice heard and never stop learning. Only in blockchain do I feel alive and grow at an exceptional pace 😄 Mónica Freitas Sep 30, 2021, 12:38 PM Hi Nicole Nguyen. You organized the first Vietnam Blockathon. How would you describe that experience? And how do you think such events can inspire the next generation of blockchain innovators? Nicole Nguyen Sep 30, 2021, 3:48 PM Mónica Freitas Vietnam Blockathon was the first blockchain hackathon in Vietnam back in 2017. There was lots of sweat and tears for sure since we were the first to educate the market and the community through groundworks and real projects. But the feeling of being the first, the PIONEERS is just out of this world - and it can propel you to new heights that you can never imagine when taking the first step. I believe we need more activities like that - with specific and real practitioners and projects to showcase how we can actually change ourselves and change the world with blockchain. Practitioners in NFT gaming space encompass not only gamers but also artists, creators, developers and investors. We are doing some arts creativity contest at Duelist King with a number of partners to demonstrate real use cases of NFTs and how indie artists can actually reap the benefits of this daunting and robust space. https://medium.com/ludena-protocol/duelist-king-digital-nft-art-competition-start-on-october-4th-in-korea-with-ludena-protocol-833202b1e54c Limarc Ambalina Oct 5, 2021, 6:06 AM Thanks for your thoughtful responses Nicole Nguyen. I've played the demo and can vouch for the animation, beautiful card design, sound effects, and more. However, one thing I'm worried about with every card game (not just DK) is how often big card games on the market foster a pay-to-win economy. "If I just buy the strongest cards, I have the strongest deck and no one can touch me" Does the gameplay mechanics and rules of Duelist King anyway offset pay-to-win and reward skill and strategy over who has the biggest bank account? Nicole Nguyen Oct 6, 2021, 4:40 AM Limarc Ambalina thats a really great question. Our cards are distributed randomly so it doesn't mean you can buy powerful cards at your own will. All cards are sold at a flat price of 1 USD in a box of 5 cards. You still can definitely pay to buy the more powerful cards on the secondary market but you need to be more strategic with your pocket as they are not cheap at all 😄 Our game economics do offer more earning opportunities and incentives for people who challenge others and participate in tournaments and battles. Also any upgrading option would be reset prior to tournaments to make sure people can participate on equal ground. The only way for you to climb up the ranking ladder is to duel against people of the same ranks so make sure you play your card rights and engage in the game as much as you can! Jack Boreham Oct 5, 2021, 7:55 AM Hi Nicole Nguyen, thanks for joining us. What do you think about the future of NFT’s? Do you think they will take off and become mainstream? How long do you think this will take? Nicole Nguyen Oct 6, 2021, 4:44 AM Jack Boreham Since joining the blockchain space in 2016, I believe NFTs are some of the few trends that will be here to stay. NFTs are increasingly finding its use cases and utilities in a number of industries like gaming, arts and collectibles. Adding metaverse effect, the sky's the limit for NFT growth as a whole new emerging business instead of a contender to the traditional space. Another important implication of this trend is increasing cross-chain collaboration - the NFT and metaverse so far have opened up significantly more crossover development and community development efforts and I think this is very healthy and conducive to further growth of the whole industry. There are still lots of things that need to be done on the onboarding, payment, and solution to expedite NFT adoption but I believe it's a matter of time till that becomes a reality. NFTs have started going mainstream and by next year I hope we can entice much more adopters. Mónica Freitas Oct 5, 2021, 8:05 AM Nicole Nguyen Thank you for your answer. How do you think we could get more people outside of this "blockchain world" and even the gaming world, to get interested? How can we translate the meaning of blockchain in a way that seems valuable for someone foreign to these topics? Nicole Nguyen Oct 6, 2021, 4:48 AM Mónica Freitas this is a industry wide challenge, however gaming is among the most popular catalyst for blockchain adoption. Personally I think it has a lot to do with specific use cases, simple onboarding for users and also the user experiences in game. From the last conference I've been to, people have been talking about the issue of lack of education, unpolished game play and designs and lack of user-friendly onboarding for mainstream audience. Once these challenges are addressed either by existing blockchain gaming startups or traditional gaming studios, blockchain can manifest itself as the underlying solution to games and people will hardly notice the difference between blockchain gaming and traditional gaming. The key value is still user's experience - the more we can enhance game UX the more valid blockchain use cases are to the audience. Limarc Ambalina Oct 6, 2021, 6:19 AM Very happy to hear you've thought about the competitive aspect Nicole Nguyen! What have you got in the pipeline in terms of development? There is a demo out now, but when do you imagine the full game to be released? Nicole Nguyen Oct 6, 2021, 9:05 AM Limarc Ambalina Upon receiving feedback from our partners and KOLs, we are fine-tuning the game play and game design so the Player vs. Environment feature will be ready early next year. Till then we are planning to do 2-3 rounds of card sales plus community activities like game play voting/art contest so people can gather a decent deck to try out the game and engage with the project. Make sure you follow our social media and check out our upcoming designs and game - our first NFT offerings of 250,000 NFTs have been sold out in less than 45 minutes 😄 Limarc Ambalina Oct 6, 2021, 9:07 AM That's awesome and where can people join said community activities? Can you drop us the links? Nicole Nguyen Oct 6, 2021, 9:41 AM Limarc Ambalina our news feed have been jammed with activities, AMAs and other community activities. Please make sure you are not lagging behind 😄 ✅Website: ✅Twitter: ✅Telegram group: ✅ Discord: https://duelistking.com/ https://twitter.com/duelistkingnft https://t.me/DuelistKingNFT https://discord.gg/6HzQR7Mf Limarc Ambalina Oct 7, 2021, 12:42 PM Thanks so much for your thorough responses and your time Nicole! We hope the best for Duelist King and hope we can do more collaborations with you in the future. 🔥 1 Nicole Nguyen Oct 8, 2021, 3:36 PM Thanks everyone for a very energetic AMA - looking forward to more collaboration in the future. Don't forget to check us out on Twitter and Telegram at @DuelistKingNFT and stay up-to-date with Duelist King! This article is part of The hosted by HackerNoon in partnership with Gaming Metaverse Writing Contest The Sandbox . Submit your today for your chance to win up to $2000. #gaming-metaverse story

Duelist King is a New NFT Blockchain Card Game with Promising Gameplay

“What I cannot create, I do not understand.” — Richard Feynman GANs generated by a computer The above images look real, but more than that, they look familiar. They resemble a famous actress that you may have seen on television or in the movies. They are not real, however. A new type of neural network created them. Generative Adversarial Networks (GANs), sometimes called generative networks, generated these fake images. The NVIDIA research team used this new technique by feeding thousands of photos of celebrities to a neural network. The neural network produced thousands of pictures, like the ones above, that resembled the famous faces. They look real, but machines created them. GANs allow researchers to build images that look like the real ones that share many of the features the neural network was fed. It can be fed photographs of objects from tables to animals, and after being trained, it produces pictures that resemble the originals. Of the two images above, can you tell the real from the fake?¹ For the NVIDIA team to generate these images, it set up two neural networks. One that produced the pictures and the other that determined if they were real or fake. Combining these two neural networks produced a GAN. They play a cat and mouse game, where one creates fake images that look like the real ones fed into it, and the other decides if they are real. Does this remind you of anything? The Turing Test. Think of the networks as playing the guessing game of whether it is a real or fake. After training the GAN, one of the neural networks creates fake images that look like the real ones fed to the GAN. The resulting pictures look exactly like the real ones. This technique can generate a lot of fake data that can help researchers predict the future or even construct simulated worlds. That is why for Yann LeCun, Director of Facebook A.I. Research, “Generative Adversarial Networks is the most interesting idea in the last ten years in machine learning.” GANs will be helpful for creating images and maybe creating software simulations of the real world, where developers can train and test other types of software. For example, for companies writing self-driving software for cars, they can train and check their software in simulated worlds. These simulated worlds and situations are now hand-crafted by developers, but some believe that in the future these scenarios will all be created by GANs. GANs generate new images and videos from very compressed data. Thus, you could use the two neural networks for a GAN to save data and then reinstate it. Instead of zipping your files, you could use one neural network to compress it and the other to generate the original videos or images. It is no coincidence that in the human brain some of the apparatus used for imagination is the same as the one for memory recall. Demis Hassabis, the founder of DeepMind, published a paper that “showed systematically for the first time that patients with damage to their hippocampus, known to cause amnesia, were also unable to imagine themselves in new experiences.” The finding established a link between the constructive process of imagination and the reconstructive process of episodic memory recall. Increasingly realistic synthetic faces generated by variations on Generative Adversarial Networks (GANs) The Creator of GANs , the creator of GANs, came up with the idea at a bar in Montreal when he was with fellow researchers discussing and trying to figure out what goes into creating photographs. The initial plan was to understand the statistics that determined what created photos, and then feed them to a machine so that it could produce them. thought that the idea would never work because there are too many statistics to make that work. So, he thought about using a tool, a neural network. He could teach neural networks to figure out what are the underlying characteristics of the pictures fed to the machine and generate new ones. Ian Goodfellow Goodfellow Goodfellow then added two neural networks so that they could together build realistic photographs. One created fake images and the other determined if they were real. The idea was that one of the adversary networks would teach the other how to produce images that could not be distinguished from the real ones. On the same night that he came up with the idea, he went home a little bit drunk and stayed up that night coding the initial concept of a GAN on his laptop. It worked on the first try. A few months later, he and a few other researchers published the seminal paper on GANs at a conference. The trained GAN used handwriting digits from a well-known training image set called MNIST. In the following years, hundreds of papers were published using the idea of GANs to produce not only images but also videos and other data. Now at Google Brain, Goodfellow leads a group that is making this technique of using two neural networks very reliable to train. The result from this work are services that are far better at generating images and learning sounds, among other things. “The models learn to understand the structure of the world,” Goodfellow says. “And that can help systems learn without being explicitly told as much.” Synthetically generated word images GANs could eventually help neural networks learn with less data, generating more synthetic images that are then used to identify and create better neural networks. Recently, a group of researchers at Dropbox improved their mobile document scanner by using synthetically generated images. GANs produced new word images that, in turn, were used to train the neural network. And, that is just the start. Researchers believe that the same technique can be applied to develop artificial data that can be shared openly on the internet not revealing the primary source, making sure that the original data stays private. This would allow researchers to create and share healthcare information without sharing sensitive data about patients. GANs also show promise for predicting the future. It may sound like science fiction now, but that might change over time. LeCun is working on writing software that can generate video of future situations based on the current video. He believes that human intelligence lies in the fact that we can predict the future, and therefore, GANs will be a powerful force for Artificial Intelligence systems in the future The Birthday Paradox Test Even though GANs are generating new images and sounds, some people ask if GANS generate new information. A collection of data trains GANs. Can GANs produce data that contain information that is outside of the data given to them? Can they create images that are entirely different from the ones fed to it? A way of analyzing that is by what is called the Birthday Paradox Test. This test derives its name from the implication that if you put 23 — two soccer teams plus a referee — random people in a room, the chance that two of them have the same birthday is more than 50%. This effect happens because when there are 365 days in a year, you need around the square root of that to see an occurrence of duplicate birthdays. The Birthday Paradox says that for a discrete distribution that has support N_,_ then a random sample size of √N would likely contain a duplicate. What does that mean? Let me break it down. If there are 365 days a year, then you need the square root of 365 — √365 — to have a probable chance of duplicate birthdays, which means about 19 people. But this also works for the other side of the equation. If you do not know the number of days in a year, then you can select a fixed number of people and ask them if there are two people with the same birthday. Then, you can infer the number of days based on the number of people when two people have the same birthday with high probability. If you have 22 people in a room, then the number of days in a year is the square of 22–22² — about 484 days per year. The same test can check the size of the original distribution of the generated images of a GAN. If the result reveals that K images have duplicate images with reasonable probability, then you can suspect that the number of original images is about K². So, if it shows that it is very likely to find a duplicate with 20 images, then it means that the size of original set of images equals 400. With the following test in hand, researchers have shown that images generated by famous GANs do not generalize beyond what the training data provides. Now, what is left to prove is if GANs can generalize beyond the training data or if there are ways of generalizing beyond the original images by using other methods to improve the dataset. Humans receive data, including sounds, images through vision, and whatnot. A question that some researchers ask is if humans would pass the Birthday Paradox Test, meaning “Can human brains generalize more than the data that is presented to them?” Bad Intent There are concerns that people can use this technique with ill intent. With so much attention on fake media, we could face an even broader range of attacks with fake data. “The concern is that these methods will rise to the point where it becomes very difficult to discern truth from falsity,” said Tim Hwang, who previously oversaw A.I. policy at Google and is now director of the Ethics and Governance of Artificial Intelligence Fund, an effort to fund ethical A.I. research. “You might believe that accelerates problems we already have.” Even though this technique cannot create still images of high quality, researchers believe that the same technology could produce videos, games, and virtual reality. The work to start generating videos has already begun. Researchers are also using a wide range of other Machine Learning methods to generate faux data. In August of 2017, a group of researchers at the University of Washington was featured in the headlines when they built a system that could put words in Barack Obama’s mouth in a video. An app with this technique is already on the App Store. The results were not completely convincing, but the rapid progress in the area of GANs and other techniques point to a future where it becomes tough for people to differentiate between real videos and generated ones. Some researchers claim that GAN is just another tool like others and can be used for good or evil and that there will be more technology to figure out if the newly created videos and images are real. Not only that, but researchers uncovered ways of using GANs to generate audio that sounds like something to humans but that machines understand differently. For example, you can develop audio that sounds to humans like “Hi, how are you?” and to machines, “Alexa, buy me a drink.” Or, audio that sounds like a Bach symphony, but for the machine, it sounds like “Alexa, go to this website.” The future has unlimited potential. All my blog posts can be found here: https://giacaglia.github.io/blog/ Notes: Both images were generated by GANs

Dueling Neural Networks

How CKB and RGB++ are Quietly Redefining Crypto Networks

কীভাবে CKB এবং RGB++ শান্তভাবে ক্রিপ্টো নেটওয়ার্কগুলিকে পুনরায় সংজ্ঞায়িত করছে৷

Wie CKB und RGB++ still und leise Krypto-Netzwerke neu definieren

Cómo CKB y RGB++ están redefiniendo silenciosamente las redes criptográficas

Comment CKB et RGB++ redéfinissent silencieusement les réseaux cryptographiques

कैसे CKB और RGB++ चुपचाप क्रिप्टो नेटवर्क को पुनर्परिभाषित कर रहे हैं

CKB と RGB++ が暗号ネットワークを静かに再定義する方法

CKB와 RGB++가 암호화폐 네트워크를 조용히 재정의하는 방법

Como CKB e RGB++ estão redefinindo silenciosamente as redes criptográficas

Как CKB и RGB++ незаметно переосмысливают криптосети

CKB ve RGB++ Kripto Ağlarını Sessizce Nasıl Yeniden Tanımlıyor?

CKB và RGB++ đang âm thầm định nghĩa lại mạng lưới tiền điện tử như thế nào

CKB 和 RGB++ 如何悄然重新定义加密网络

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details D IMPLEMENTATION DETAILS In this section we add remaining implementation details that were not discussed in the main paper. We break down our implementation details into two subsections, one for the next-token prediction task on CodeParrot and one for our Crammed BERT (Geiping & Goldstein, 2023) masked language modelling experiments pretrained on the Pile dataset (Gao et al., 2020) and fine-tuned to downstream GLUE benchmark (Wang et al., 2019). To avoid repetition, any details that are mentioned in one subsection but not the other are shared between both subsections. All runtime results on CodeParrot were run on a single A5000 GPU. D.1 CODEPARROT NEXT-TOKEN PREDICTION As mentioned, much of our setup is derived from https://huggingface.co/learn/ nlp-course/chapter7/6. Model The model is a 18-layer GPT-style auto-regressive decoder-only transformer. We use width d = 768, and H = 12 heads in multi-head attention. We remove dropout entirely as our focus is on training speed, and we are always in a single-epoch regime so regularisation hurts training speed. The MLP uses ReLU activation unless stated otherwise, and we use MLP hidden dimension 3072 = 4d. The only exception to this is in Fig. 6, where we reduce the MLP hidden dimension to 1536 = 2d to account for the increased memory requirements of larger depths. For any of our simplified model we initialise βFF = 0.1 in Eq. (1) to account for the lack of skip, apart from the 18-layer models in Fig. 6, where βFF = 0.2 due to the narrower width. We use RMSNorm (Zhang & Sennrich, 2019) where applicable with epsilon 1e − 8, and add a final normalisation after the decoder. Sinusoidal positional encodings are used and added at the embedding level. Training We use AdamW optimiser (Loshchilov & Hutter, 2017) with weight decay 0.1 which we tuned on a small grid, and found to work well for both baselines and our models. We do not apply weight decay to any scalar gain parameter. We clip gradients with with clipping parameter 1, and use epsilon of 1e − 8 and default betas of (0.9, 0.999) in AdamW. As discussed, we use a linear decay rate with 5% of all steps used for linear warmup. The optimal learning rate was tuned in all cases, and for our best (SAS and SAS-P) models, was found to be 1e−3, which exactly matched that of the default Pre-LN. This held true also when we scaled to 72 layers. V-SkipInit needed a lower learning rate for the depth scaling experiments (3e−4 and 1e−4 for depths 18 and 72 respectively). We use batch size of 128 with microbatches of size 32. Dataset The Codeparrot dataset is a large corpus of 20 million python files from GitHub. We take the dataset, pre-processing and tokeniser from https://huggingface.co/learn/ nlp-course/chapter7/6. We use sequence length T = 128 throughout, and our tokeniser has 50K vocabulary size. Our base experiments train for around 43K steps on batch size 128 and sequence length 128 which is around 700M tokens. In Fig. 8 we scale this to 2B tokens. Task The model is trained on next-token prediction using cross-entropy loss. D.2 BERT ENCODER-ONLY As discussed in Sec. 5, we inherit much of our hyperparameters from the Cramming setup of Geiping & Goldstein (2023), and also base our implementation from their excellent codebase.[9] We highlight important implementation details here. Model We use a 16-layer encoder only model, with width d = 768 and 12 heads. We use MLP width 3072 = 4d, but now we use GLU (Dauphin et al., 2017) with GeLU activation, which essentially halves the hidden dimension. We use LayerNorm (Ba et al., 2016) for normalisation where applicable with epsilon 1e − 12 as taken from Geiping & Goldstein (2023); we always use a final LN after all the layers. Again, we remove all dropout, and use a sequence length of 128. We found our simplified skipless models prefered smaller MLP block scales and initialise βFF = 0.05. Parameter Initialisation The initialisations are identical to those in Codeparrot, and are detailed above. Datasets Like Geiping & Goldstein (2023), we train on the Pile dataset (Gao et al., 2020), with a WordPiece tokeniser of vocabulary size 32768, and a sequence length of 128. Our fastest runs took around 600K steps with microbatch size 64 in 24 hours, which corresponds to around 5B tokens. Training We again trained with AdamW optimiser, with weight decay 0.1. AdamW had hyparameters [β1, β2] = [0.9, 0.98], and epsilon 1e − 12. We used a microbatch of 64 (to fit on a RTX-2080Ti), and scale the batch size to reach 8192 linearly after 60% of total training like in Geiping & Goldstein (2023). We use the same aggressive learning rate as Geiping & Goldstein (2023), which increase linearly to max value after 75% of all training steps, before linear decay, and tune the maximum learning rate to 3e − 3 for our SAS and SAS-P models. This was slightly too large for the SAS-P model without normalisation, so we reduce to 2e − 3. We inherit the clipping parameter of 0.5 from Geiping & Goldstein (2023). Fine-tuning We followed the same protocol as Geiping & Goldstein (2023), In particular, we finetune for 5 epochs with fixed hyperparameters across tasks. We found dropout to be important for good downstream performane (unlike during pre-training), and set dropout probability p = 0.1. We use batch size 32, with a maximum learning of 1.5e − 4. We keep other hyperparameters, e.g. the choice of cosine decay and AdamW epsilon 1e − 6, like from Geiping & Goldstein (2023). Task The model is trained on the masked language modelling task with masking probability 0.25, as in Geiping & Goldstein (2023). This paper is available on arxiv under CC 4.0 license. [9] https://github.com/JonasGeiping/cramming Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details D IMPLEMENTATION DETAILS In this section we add remaining implementation details that were not discussed in the main paper. We break down our implementation details into two subsections, one for the next-token prediction task on CodeParrot and one for our Crammed BERT (Geiping & Goldstein, 2023) masked language modelling experiments pretrained on the Pile dataset (Gao et al., 2020) and fine-tuned to downstream GLUE benchmark (Wang et al., 2019). To avoid repetition, any details that are mentioned in one subsection but not the other are shared between both subsections. All runtime results on CodeParrot were run on a single A5000 GPU. D.1 CODEPARROT NEXT-TOKEN PREDICTION As mentioned, much of our setup is derived from https://huggingface.co/learn/ nlp-course/chapter7/6. Model The model is a 18-layer GPT-style auto-regressive decoder-only transformer. We use width d = 768, and H = 12 heads in multi-head attention. We remove dropout entirely as our focus is on training speed, and we are always in a single-epoch regime so regularisation hurts training speed. The MLP uses ReLU activation unless stated otherwise, and we use MLP hidden dimension 3072 = 4d. The only exception to this is in Fig. 6, where we reduce the MLP hidden dimension to 1536 = 2d to account for the increased memory requirements of larger depths. Model For any of our simplified model we initialise βFF = 0.1 in Eq. (1) to account for the lack of skip, apart from the 18-layer models in Fig. 6, where βFF = 0.2 due to the narrower width. We use RMSNorm (Zhang & Sennrich, 2019) where applicable with epsilon 1e − 8, and add a final normalisation after the decoder. Sinusoidal positional encodings are used and added at the embedding level. Training We use AdamW optimiser (Loshchilov & Hutter, 2017) with weight decay 0.1 which we tuned on a small grid, and found to work well for both baselines and our models. We do not apply weight decay to any scalar gain parameter. We clip gradients with with clipping parameter 1, and use epsilon of 1e − 8 and default betas of (0.9, 0.999) in AdamW. As discussed, we use a linear decay rate with 5% of all steps used for linear warmup. The optimal learning rate was tuned in all cases, and for our best (SAS and SAS-P) models, was found to be 1e−3, which exactly matched that of the default Pre-LN. This held true also when we scaled to 72 layers. V-SkipInit needed a lower learning rate for the depth scaling experiments (3e−4 and 1e−4 for depths 18 and 72 respectively). We use batch size of 128 with microbatches of size 32. Dataset The Codeparrot dataset is a large corpus of 20 million python files from GitHub. We take the dataset, pre-processing and tokeniser from https://huggingface.co/learn/ nlp-course/chapter7/6. We use sequence length T = 128 throughout, and our tokeniser has 50K vocabulary size. Our base experiments train for around 43K steps on batch size 128 and sequence length 128 which is around 700M tokens. In Fig. 8 we scale this to 2B tokens. Dataset Task The model is trained on next-token prediction using cross-entropy loss. Task D.2 BERT ENCODER-ONLY As discussed in Sec. 5, we inherit much of our hyperparameters from the Cramming setup of Geiping & Goldstein (2023), and also base our implementation from their excellent codebase.[9] We highlight important implementation details here. Model We use a 16-layer encoder only model, with width d = 768 and 12 heads. We use MLP width 3072 = 4d, but now we use GLU (Dauphin et al., 2017) with GeLU activation, which essentially halves the hidden dimension. We use LayerNorm (Ba et al., 2016) for normalisation where applicable with epsilon 1e − 12 as taken from Geiping & Goldstein (2023); we always use a final LN after all the layers. Again, we remove all dropout, and use a sequence length of 128. We found our simplified skipless models prefered smaller MLP block scales and initialise βFF = 0.05. Model Parameter Initialisation The initialisations are identical to those in Codeparrot, and are detailed above. Parameter Initialisation Datasets Like Geiping & Goldstein (2023), we train on the Pile dataset (Gao et al., 2020), with a WordPiece tokeniser of vocabulary size 32768, and a sequence length of 128. Our fastest runs took around 600K steps with microbatch size 64 in 24 hours, which corresponds to around 5B tokens. Datasets Training We again trained with AdamW optimiser, with weight decay 0.1. AdamW had hyparameters [β1, β2] = [0.9, 0.98], and epsilon 1e − 12. We used a microbatch of 64 (to fit on a RTX-2080Ti), and scale the batch size to reach 8192 linearly after 60% of total training like in Geiping & Goldstein (2023). We use the same aggressive learning rate as Geiping & Goldstein (2023), which increase linearly to max value after 75% of all training steps, before linear decay, and tune the maximum learning rate to 3e − 3 for our SAS and SAS-P models. This was slightly too large for the SAS-P model without normalisation, so we reduce to 2e − 3. We inherit the clipping parameter of 0.5 from Geiping & Goldstein (2023). Training Fine-tuning We followed the same protocol as Geiping & Goldstein (2023), In particular, we finetune for 5 epochs with fixed hyperparameters across tasks. We found dropout to be important for good downstream performane (unlike during pre-training), and set dropout probability p = 0.1. We use batch size 32, with a maximum learning of 1.5e − 4. We keep other hyperparameters, e.g. the choice of cosine decay and AdamW epsilon 1e − 6, like from Geiping & Goldstein (2023). Fine-tuning Task The model is trained on the masked language modelling task with masking probability 0.25, as in Geiping & Goldstein (2023). Task This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [9] https://github.com/JonasGeiping/cramming

Simplifying Transformer Blocks: Implementation Details

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details C ADDITIONAL EXPERIMENTS In this section, we provide additional experiments and ablations on top of those provided in the main paper. The experiments in this section are ordered to follow the chronological order of where they are referenced (or most relevant) in the main paper. Linear vs Cosine decay LR schedule Fig. 11 compares linear and cosine decay LR schedule. We see that linear decay provides better final performance across both our models and baselines, and use linear decay throughout the rest of the paper. Shaped Attention vs Value-SkipInit Fig. 12 explains our reasons for using Shaped Attention Eq. (5) (Noci et al., 2023) over the modified attention matrix, αI + βA(X), Eq. (4), that was introduced by He et al. (2023) in Value-SkipInit. We see that Shaped Attention gives a small but consistent gain throughout training. The experiments here follow the same training and hyperparameter setup as those in Sec. 4. Sensitivity to MLP block gain initialisation In Sec. 4.1, we motivated downweighting the initialisation of trainable MLP block weight βFF (c.f. Eqs. (1) and (7)) in skipless architectures to replicate the implicit downweighting mechanism of Pre-LN skips. Fig. 13 shows the sensitivity of final loss to our initialisation for trainable βFF. Fig. 14 is the equivalent of Fig. 3 but with LL-tied (yellow line with diamond markers) and all-tied (blue line with star markers) included. In Fig. 14, we see that matching the functional output (as in LL tied) with orthogonal initialisation provides a slight improvement to close the gap between the random orthogonal (green line) and identity (purple line) initialisations. Matching the attention sub-block outputs (as in all-tied) further improves performance, but does not fully close the gap to identity initialisation. Interestingly, it seems like orthogonally initialised values and projections do benefit from being trainable (i.e. small but non-zero βV , βP ). We leave a further exploration of these observations to future work. Figs. 16 to 19 plot the corresponding trajectories, of the model in Fig. 4, for various other trainable scalar parameters we have: namely, the three shaped attention parameters 1) α on the identity (Fig. 16), 2) β on the softmax attention output (Fig. 17), 3) γ on the centring matrix (Fig. 18), as well as 4) βFF on the MLP block (Fig. 19). We see that none of these other scalar parameters converge to zero. The shaped attention parameters have error bars denoting standard deviations across (the 12) heads. Identity values and projections with default Pre-LN block In Sec. 4.2 we observed that removing values and projection parameters by setting them to the identity improves the convergence speed per parameter update of transformer blocks with skipless attention sub-blocks. This raises the question of whether the same would occur in the standard block that uses attention sub-blocks skip connections i.e. the Pre-LN block. In Fig. 20 we compare the default Pre-LN block to one with values and projections set to be identity. We see that in this case identity values and projections actually slightly hurt performance in terms of loss reduction per update, in constrast to the skipless case. We also tried to downweight the attention block residual (βSA < 1 in the notation of Eq. (1)) with identity values and projections to see if the scale of the attention skip and residual was the reason for this difference, but this did not change the findings of Fig. 20. We do not have a satisfying explanation for this, but our intuition is that identity value and projections (as opposed to e.g. Gaussian/random orthogonal initialisation) with attention skip means that the two branches of the skip are no longer independent at initialisation and interfere with each other, though it is unclear that this would continue to hold during training. Linearising MLP activations As stated in Sec. 4.3, we tried to use the recent idea of “linearising” activation functions in order to obtain better signal propagation (Martens et al., 2021; Zhang et al., 2022; Li et al., 2022) in deep NNs with skip connections, and recover lost training speed when the MLP skip is removed. In particular, Li et al. (2022) show for Leaky ReLU, We see that all blocks without MLP skip train significantly slower than our SAS block (which matches the training speed of the Pre-LN block). In fact, linearising ReLU into LReLU seemed to hurt training speed rather than help it. These findings are consistent with those of previous works with AdamW optimiser (Martens et al., 2021; Zhang et al., 2022; He et al., 2023). We note that a big reason behind this is that the architectures with skipless MLP sub-blocks required an order of magnitude smaller learning rate (1e − 4 vs 1e − 3) otherwise training was unstable. Loss vs training step In Fig. 23, we provide the equivalent plot to Fig. 5, but in terms of loss over the steps taken. Our SAS and SAS-P essentially match the Pre-LN model in terms of loss reduction per step, whilst removing normalisation slightly hurts performance. Crammed Bert loss vs training step In Fig. 24 we plot the MLM loss in the Crammed Bert setting on the Pile dataset, as a function of the number of microbatch steps taken. Because our models have higher throughput they are able to take more steps within the 24 hour allotted time. GLUE breakdown In Table 2, we provide a breakdown of the GLUE results in Table 1 in terms of different tasks. This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details C ADDITIONAL EXPERIMENTS In this section, we provide additional experiments and ablations on top of those provided in the main paper. The experiments in this section are ordered to follow the chronological order of where they are referenced (or most relevant) in the main paper. Linear vs Cosine decay LR schedule Fig. 11 compares linear and cosine decay LR schedule. We see that linear decay provides better final performance across both our models and baselines, and use linear decay throughout the rest of the paper. Linear vs Cosine decay LR schedule Shaped Attention vs Value-SkipInit Fig. 12 explains our reasons for using Shaped Attention Eq. (5) (Noci et al., 2023) over the modified attention matrix, αI + βA(X), Eq. (4), that was introduced by He et al. (2023) in Value-SkipInit. We see that Shaped Attention gives a small but consistent gain throughout training. The experiments here follow the same training and hyperparameter setup as those in Sec. 4. Shaped Attention vs Value-SkipInit Sensitivity to MLP block gain initialisation In Sec. 4.1, we motivated downweighting the initialisation of trainable MLP block weight βFF (c.f. Eqs. (1) and (7)) in skipless architectures to replicate the implicit downweighting mechanism of Pre-LN skips. Fig. 13 shows the sensitivity of final loss to our initialisation for trainable βFF. Sensitivity to MLP block gain initialisation Fig. 14 is the equivalent of Fig. 3 but with LL-tied (yellow line with diamond markers) and all-tied (blue line with star markers) included. In Fig. 14, we see that matching the functional output (as in LL tied) with orthogonal initialisation provides a slight improvement to close the gap between the random orthogonal (green line) and identity (purple line) initialisations. Matching the attention sub-block outputs (as in all-tied) further improves performance, but does not fully close the gap to identity initialisation. Interestingly, it seems like orthogonally initialised values and projections do benefit from being trainable (i.e. small but non-zero βV , βP ). We leave a further exploration of these observations to future work. Figs. 16 to 19 plot the corresponding trajectories, of the model in Fig. 4, for various other trainable scalar parameters we have: namely, the three shaped attention parameters 1) α on the identity (Fig. 16), 2) β on the softmax attention output (Fig. 17), 3) γ on the centring matrix (Fig. 18), as well as 4) βFF on the MLP block (Fig. 19). We see that none of these other scalar parameters converge to zero. The shaped attention parameters have error bars denoting standard deviations across (the 12) heads. Identity values and projections with default Pre-LN block In Sec. 4.2 we observed that removing values and projection parameters by setting them to the identity improves the convergence speed per parameter update of transformer blocks with skipless attention sub-blocks. This raises the question of whether the same would occur in the standard block that uses attention sub-blocks skip connections i.e. the Pre-LN block. In Fig. 20 we compare the default Pre-LN block to one with values and projections set to be identity. We see that in this case identity values and projections actually slightly hurt performance in terms of loss reduction per update, in constrast to the skipless case. We also tried to downweight the attention block residual (βSA < 1 in the notation of Eq. (1)) with identity values and projections to see if the scale of the attention skip and residual was the reason for this difference, but this did not change the findings of Fig. 20. We do not have a satisfying explanation for this, but our intuition is that identity value and projections (as opposed to e.g. Gaussian/random orthogonal initialisation) with attention skip means that the two branches of the skip are no longer independent at initialisation and interfere with each other, though it is unclear that this would continue to hold during training. Identity values and projections with default Pre-LN block Linearising MLP activations As stated in Sec. 4.3, we tried to use the recent idea of “linearising” activation functions in order to obtain better signal propagation (Martens et al., 2021; Zhang et al., 2022; Li et al., 2022) in deep NNs with skip connections, and recover lost training speed when the MLP skip is removed. In particular, Li et al. (2022) show for Leaky ReLU, Linearising MLP activations We see that all blocks without MLP skip train significantly slower than our SAS block (which matches the training speed of the Pre-LN block). In fact, linearising ReLU into LReLU seemed to hurt training speed rather than help it. These findings are consistent with those of previous works with AdamW optimiser (Martens et al., 2021; Zhang et al., 2022; He et al., 2023). We note that a big reason behind this is that the architectures with skipless MLP sub-blocks required an order of magnitude smaller learning rate (1e − 4 vs 1e − 3) otherwise training was unstable. Loss vs training step In Fig. 23, we provide the equivalent plot to Fig. 5, but in terms of loss over the steps taken. Our SAS and SAS-P essentially match the Pre-LN model in terms of loss reduction per step, whilst removing normalisation slightly hurts performance. Loss vs training step Crammed Bert loss vs training step In Fig. 24 we plot the MLM loss in the Crammed Bert setting on the Pile dataset, as a function of the number of microbatch steps taken. Because our models have higher throughput they are able to take more steps within the 24 hour allotted time. Crammed Bert loss vs training step GLUE breakdown In Table 2, we provide a breakdown of the GLUE results in Table 1 in terms of different tasks. GLUE breakdown This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Simplifying Transformer Blocks: Additional Experiments 

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details B BLOCK LAYOUTS In Fig. 9 and Fig. 10 we show the layouts of our SAS block (Sec. 4.2) and parallel SAS-P block (Sec. 4.3). These are the equivalent plots to the layouts in Fig. 1. Mathematically, our SAS attention sub-block computes (in the notation of Eq. (2)): This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details B BLOCK LAYOUTS In Fig. 9 and Fig. 10 we show the layouts of our SAS block (Sec. 4.2) and parallel SAS-P block (Sec. 4.3). These are the equivalent plots to the layouts in Fig. 1. Mathematically, our SAS attention sub-block computes (in the notation of Eq. (2)): This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Simplifying Transformer Blocks: Block Layouts

Simplifying Transformer Models for Faster Training and Better Performance

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details 5 FURTHER EXPERIMENTAL ANALYSIS Having detailed all of our simplifications in Sec. 4, we now provide further experimental results of our simplified blocks in a range of settings, as well as details of the efficiency gains afforded by our simplifications. In the interest of space, additional experimental details can be found in App. D. Depth Scaling Given that signal propagation theory often focuses on large depths, where signal degeneracies usually appear, it is natural to ask whether the improved training speeds of our simplified transformer blocks also extend to larger depths. In Fig. 6, we see that scaling depth from 18 to 72 blocks leads to an increase in performance in our models as well as the Pre-LN transformer, indicating that our simplified models are able to not only train faster but also to utilise the extra capacity that more depth provides. Indeed, the per-update trajectories of our simplified blocks and Pre-LN are near-indistinguishable across depths, when using normalisation. On the other hand, we see that Value-SkipInit (He et al., 2023) actually trains slower per update at depth 72 compared to 18 despite the increase in capacity and parameter count. Moreover, the gap in performance between Value-SkipInit and the other models increases with larger depth, which implies poor scalability of the previous method. We note that 72 blocks is already reasonably deep by publically-available modern standards (Hoffmann et al., 2022; Touvron et al., 2023). BERT Next, we demonstrate our simplified blocks’ performance extends to different datasets and architectures besides autoregressive decoder-only, as well as on downstream tasks. We choose the popular setting of the bidirectional encoder-only BERT model Devlin et al. (2018) for masked language modelling, with downstream GLUE benchmark. In particular, we adopt the “Crammed” BERT setup of Geiping & Goldstein (2023), which asks how well one can train a BERT model with a modest training budget: 24 hours on a single consumer GPU. The authors provide an architecture, data pipeline and training setup that has been optimised for this low resource setting. We note that the Crammed architecture uses the Pre-LN block, and describe other setup details in App. D. We plug-in our simplified blocks, keeping the existing optimised hyperparameters, besides tuning learning rate and weight decay. In Fig. 7, we see that our simplified blocks (especially with normalisation) match the pre-training speed on the masked language modelling task compared to the (Crammed) Pre-LN baseline within the 24 hour runtime. On the other hand, the removal of skip connections without modifying the values and projections (as in He et al. (2023)) once again leads to a significant loss of training speed. In Fig. 24, we provide the equivalent plot in terms of microbatch steps. Moreover in Table 1, we find that our methods match the performance of the Crammed BERT baseline after finetuning on the GLUE benchmark. We provide a breakdown over the downstream tasks in Table 2. We use the same finetuning protocol as Geiping & Goldstein (2023) (5 epochs, constant hyperparameters across tasks, dropout regularisation) for a fair comparison. Interestingly, Value-SkipInit is largely able to recover from its poor pre-training in the fine-tuning phase. This, combined with the need for dropout when fine-tuning, suggests that factors besides pre-training speed are also important for fine-tuning. As the focus of our work primarily concerns training speed from random initialisations, we leave this to future work. Relatedly, we found removing normalisations (Sec. 4.4) to cause instabilities when fine-tuning, where a small minority of sequences in some downstream datasets had NaN values in the initial forward pass from the pre-trained checkpoint. Efficiency Gains In Table 1, we also detail the parameter count and training speeds of models using different Transformers blocks on the masked language modelling task. We compute the speed as the ratio of the number of microbatch steps taken within the 24 hours of pre-training, relative to the baseline Pre-LN Crammed BERT. We see that our models use 16% fewer parameters, and SAS-P & SAS are 16% & 9% faster per iteration, respectively, compared to the Pre-LN block in our setting. We note that in our implementation the Parallel block is only 5% faster than the Pre-LN block, whereas Chowdhery et al. (2022) observed 15% faster training speeds, suggesting that further throughout increases may be possible with a more optimised implementation. Our implementation, like Geiping & Goldstein (2023), uses automated operator fusion in PyTorch (Sarofeen et al., 2022) Longer training Finally, given the current trends of training smaller models for longer on more data (Hoffmann et al., 2022; Touvron et al., 2023), we investigate if our simplified blocks continue to match the training speeds of the Pre-LN block with longer training. To do this, we take our models from Fig. 5 on CodeParrot and train with 3× tokens. To be precise, we train for around 120K (rather than 40K) steps with batch size 128 and sequence length 128, which results in around 2B tokens. In Fig. 8, we do indeed see that our simplified SAS and SAS-P blocks continue to match or outer perform the PreLN block in training speed when trained on more tokens. This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details 5 FURTHER EXPERIMENTAL ANALYSIS Having detailed all of our simplifications in Sec. 4, we now provide further experimental results of our simplified blocks in a range of settings, as well as details of the efficiency gains afforded by our simplifications. In the interest of space, additional experimental details can be found in App. D. Depth Scaling Given that signal propagation theory often focuses on large depths, where signal degeneracies usually appear, it is natural to ask whether the improved training speeds of our simplified transformer blocks also extend to larger depths. In Fig. 6, we see that scaling depth from 18 to 72 blocks leads to an increase in performance in our models as well as the Pre-LN transformer, indicating that our simplified models are able to not only train faster but also to utilise the extra capacity that more depth provides. Indeed, the per-update trajectories of our simplified blocks and Pre-LN are near-indistinguishable across depths, when using normalisation. Depth Scaling On the other hand, we see that Value-SkipInit (He et al., 2023) actually trains slower per update at depth 72 compared to 18 despite the increase in capacity and parameter count. Moreover, the gap in performance between Value-SkipInit and the other models increases with larger depth, which implies poor scalability of the previous method. We note that 72 blocks is already reasonably deep by publically-available modern standards (Hoffmann et al., 2022; Touvron et al., 2023). BERT Next, we demonstrate our simplified blocks’ performance extends to different datasets and architectures besides autoregressive decoder-only, as well as on downstream tasks. We choose the popular setting of the bidirectional encoder-only BERT model Devlin et al. (2018) for masked language modelling, with downstream GLUE benchmark. BERT In particular, we adopt the “Crammed” BERT setup of Geiping & Goldstein (2023), which asks how well one can train a BERT model with a modest training budget: 24 hours on a single consumer GPU. The authors provide an architecture, data pipeline and training setup that has been optimised for this low resource setting. We note that the Crammed architecture uses the Pre-LN block, and describe other setup details in App. D. We plug-in our simplified blocks, keeping the existing optimised hyperparameters, besides tuning learning rate and weight decay. In Fig. 7, we see that our simplified blocks (especially with normalisation) match the pre-training speed on the masked language modelling task compared to the (Crammed) Pre-LN baseline within the 24 hour runtime. On the other hand, the removal of skip connections without modifying the values and projections (as in He et al. (2023)) once again leads to a significant loss of training speed. In Fig. 24, we provide the equivalent plot in terms of microbatch steps. Moreover in Table 1, we find that our methods match the performance of the Crammed BERT baseline after finetuning on the GLUE benchmark. We provide a breakdown over the downstream tasks in Table 2. We use the same finetuning protocol as Geiping & Goldstein (2023) (5 epochs, constant hyperparameters across tasks, dropout regularisation) for a fair comparison. Interestingly, Value-SkipInit is largely able to recover from its poor pre-training in the fine-tuning phase. This, combined with the need for dropout when fine-tuning, suggests that factors besides pre-training speed are also important for fine-tuning. As the focus of our work primarily concerns training speed from random initialisations, we leave this to future work. Relatedly, we found removing normalisations (Sec. 4.4) to cause instabilities when fine-tuning, where a small minority of sequences in some downstream datasets had NaN values in the initial forward pass from the pre-trained checkpoint. Efficiency Gains In Table 1, we also detail the parameter count and training speeds of models using different Transformers blocks on the masked language modelling task. We compute the speed as the ratio of the number of microbatch steps taken within the 24 hours of pre-training, relative to the baseline Pre-LN Crammed BERT. We see that our models use 16% fewer parameters, and SAS-P & SAS are 16% & 9% faster per iteration, respectively, compared to the Pre-LN block in our setting. We note that in our implementation the Parallel block is only 5% faster than the Pre-LN block, whereas Chowdhery et al. (2022) observed 15% faster training speeds, suggesting that further throughout increases may be possible with a more optimised implementation. Our implementation, like Geiping & Goldstein (2023), uses automated operator fusion in PyTorch (Sarofeen et al., 2022) Efficiency Gains Longer training Finally, given the current trends of training smaller models for longer on more data (Hoffmann et al., 2022; Touvron et al., 2023), we investigate if our simplified blocks continue to match the training speeds of the Pre-LN block with longer training. To do this, we take our models from Fig. 5 on CodeParrot and train with 3× tokens. To be precise, we train for around 120K (rather than 40K) steps with batch size 128 and sequence length 128, which results in around 2B tokens. In Fig. 8, we do indeed see that our simplified SAS and SAS-P blocks continue to match or outer perform the PreLN block in training speed when trained on more tokens. Longer training This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Depth Scaling and Efficiency Gains with Simplified Transformer Blocks

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details 4 SIMPLIFYING TRANSFORMER BLOCKS We now describe how we arrive at our simplest Transformer block, Fig. 1 (top right), starting from the Pre-LN block, using a combination of signal propagation theory and empirical observations. Each subsection here will remove one block component at a time without compromising training speed, and we aim to provide an intuitive account of our progress in simplifying the Pre-LN block. All experiments in this section use an 18-block 768-width causal decoder-only GPT-like model on the CodeParrot dataset,[1] which is sufficiently large that we are in a single epoch regime with minimal generalisation gap (Fig. 2), allowing us to focus on training speed. We provide depth scaling, and non-causal encoder-only, experiments, in Sec. 5. We use a linear decay learning rate (LR) schedule[2] with AdamW (Loshchilov & Hutter, 2017), with linear warmup for the first 5% steps. The maximum LR is tuned on training loss, using a logarithmic grid. Additional experimental details are in App. D. 4.1 REMOVING THE ATTENTION SUB-BLOCK SKIP CONNECTION We first consider removing the skip connection in the attention sub-block. In the notation of Eq. (1), this corresponds to fixing αSA to 0. Naively removing the attention skip leads to a signal degeneracy called rank collapse (Dong et al., 2021), which causes poor trainability (Noci et al., 2022). Setup He et al. (2023) outline modifications needed to the self-attention mechanism in order to correct these signal degeneracies at large depths, and train such deep skipless networks for the first time. One method they introduce, Value-SkipInit, modifies the self-attention matrix to compute: 4.2 REMOVING VALUE AND PROJECTION PARAMETERS 4.3 REMOVING THE MLP SUB-BLOCK SKIP CONNECTION So far we have simplified the Pre-LN transformer block by removing, without loss of training speed, three key components: 1) the attention sub-block skip connection, as well as 2) value and 3) projection matrices. We next turn to removing the remaining skip connection in the MLP sub-block. This proved more challenging. Like previous works (Martens et al., 2021; Zhang et al., 2022; He et al., 2023), we found that making activations more linear, motivated through signal propagation, still resulted in a significant loss of per-update training speed without MLP skips when using Adam, as shown in Fig. 22. We also experimented with variants of the Looks Linear initialisation (Balduzzi et al., 2017), with Gaussian, orthogonal or identity weights, to no avail. As such, we use standard activations (e.g. ReLU in this section) and initialisations in the MLP sub-block throughout our work. 4.4 REMOVING NORMALISATION LAYERS The final simplification we consider is removing normalisation layers, which leaves us with our simplest block (Fig. 1, top right). From a signal propagation initialisation perspective, it has been possible to remove normalisation at any stage of our simplifications in this section: the idea is that normalisation in Pre-LN blocks implicitly downweights residual branches, and this beneficial effect can be replicated without normalisation by another mechanism: either explicitly downweighting residual branches when skips are used, or biasing attention matrices to the identity/transforming MLP non-linearities to be “more” linear otherwise. As we account for these mechanisms in our modifications (e.g. downweighted MLP βFF & Shaped Attention), there is no need for normalisation. Of course, these modifications have effects on training speeds and stability beyond initialisation, which are harder to predict from existing theory alone. In Fig. 5 we see that removing normalisation allows even our simplest transformer block, which does not have skips, sequential sub-blocks, values, projections or normalisations, to match the training speed of the Pre-LN block in terms of runtime. Having said that, we do observe a slight degradation in training speed per iteration, as seen in Fig. 23, suggesting that normalisation layers have some beneficial properties for training speed beyond what is captured by signal propagation theory. We thus treat our SAS (Fig. 9) and SAS-P (Fig. 10) blocks, with normalisation, as our main approaches. On this note, we point out that Dehghani et al. (2023) found extra normalisation on the queries and keys to provide improved training stability in ViT-22B, going against the recent trend of researchers seeking to remove normalisation. This paper is available on arxiv under CC 4.0 license. [1] Our setting is taken from https://huggingface.co/learn/nlp-course/chapter7/6. [2] We found linear decay to slightly outperform cosine decay for both our models and baselines (c.f. Fig. 11). [3] For example, when there is no masking, C becomes the uniform T × T stochastic matrix: 1 T 11⊤ [4] We set aside the MLP sub-block here for simplicity, but point out that all of our experiments use MLPs so our findings carry over to the full setting. [5] Although the initial forward pass is identical regardless of βV , due to zero initialised ∆WV Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details 4 SIMPLIFYING TRANSFORMER BLOCKS We now describe how we arrive at our simplest Transformer block, Fig. 1 (top right), starting from the Pre-LN block, using a combination of signal propagation theory and empirical observations. Each subsection here will remove one block component at a time without compromising training speed, and we aim to provide an intuitive account of our progress in simplifying the Pre-LN block. All experiments in this section use an 18-block 768-width causal decoder-only GPT-like model on the CodeParrot dataset,[1] which is sufficiently large that we are in a single epoch regime with minimal generalisation gap (Fig. 2), allowing us to focus on training speed. We provide depth scaling, and non-causal encoder-only, experiments, in Sec. 5. We use a linear decay learning rate (LR) schedule[2] with AdamW (Loshchilov & Hutter, 2017), with linear warmup for the first 5% steps. The maximum LR is tuned on training loss, using a logarithmic grid. Additional experimental details are in App. D. 4.1 REMOVING THE ATTENTION SUB-BLOCK SKIP CONNECTION We first consider removing the skip connection in the attention sub-block. In the notation of Eq. (1), this corresponds to fixing αSA to 0. Naively removing the attention skip leads to a signal degeneracy called rank collapse (Dong et al., 2021), which causes poor trainability (Noci et al., 2022). Setup He et al. (2023) outline modifications needed to the self-attention mechanism in order to correct these signal degeneracies at large depths, and train such deep skipless networks for the first time. One method they introduce, Value-SkipInit, modifies the self-attention matrix to compute: Setup 4.2 REMOVING VALUE AND PROJECTION PARAMETERS 4.3 REMOVING THE MLP SUB-BLOCK SKIP CONNECTION So far we have simplified the Pre-LN transformer block by removing, without loss of training speed, three key components: 1) the attention sub-block skip connection, as well as 2) value and 3) projection matrices. We next turn to removing the remaining skip connection in the MLP sub-block. This proved more challenging. Like previous works (Martens et al., 2021; Zhang et al., 2022; He et al., 2023), we found that making activations more linear, motivated through signal propagation, still resulted in a significant loss of per-update training speed without MLP skips when using Adam, as shown in Fig. 22. We also experimented with variants of the Looks Linear initialisation (Balduzzi et al., 2017), with Gaussian, orthogonal or identity weights, to no avail. As such, we use standard activations (e.g. ReLU in this section) and initialisations in the MLP sub-block throughout our work. 4.4 REMOVING NORMALISATION LAYERS The final simplification we consider is removing normalisation layers, which leaves us with our simplest block (Fig. 1, top right). From a signal propagation initialisation perspective, it has been possible to remove normalisation at any stage of our simplifications in this section: the idea is that normalisation in Pre-LN blocks implicitly downweights residual branches, and this beneficial effect can be replicated without normalisation by another mechanism: either explicitly downweighting residual branches when skips are used, or biasing attention matrices to the identity/transforming MLP non-linearities to be “more” linear otherwise. As we account for these mechanisms in our modifications (e.g. downweighted MLP βFF & Shaped Attention), there is no need for normalisation. Of course, these modifications have effects on training speeds and stability beyond initialisation, which are harder to predict from existing theory alone. In Fig. 5 we see that removing normalisation allows even our simplest transformer block, which does not have skips, sequential sub-blocks, values, projections or normalisations, to match the training speed of the Pre-LN block in terms of runtime. Having said that, we do observe a slight degradation in training speed per iteration, as seen in Fig. 23, suggesting that normalisation layers have some beneficial properties for training speed beyond what is captured by signal propagation theory. We thus treat our SAS (Fig. 9) and SAS-P (Fig. 10) blocks, with normalisation, as our main approaches. On this note, we point out that Dehghani et al. (2023) found extra normalisation on the queries and keys to provide improved training stability in ViT-22B, going against the recent trend of researchers seeking to remove normalisation. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [1] Our setting is taken from https://huggingface.co/learn/nlp-course/chapter7/6. [2] We found linear decay to slightly outperform cosine decay for both our models and baselines (c.f. Fig. 11). [3] For example, when there is no masking, C becomes the uniform T × T stochastic matrix: 1 T 11⊤ [4] We set aside the MLP sub-block here for simplicity, but point out that all of our experiments use MLPs so our findings carry over to the full setting. [5] Although the initial forward pass is identical regardless of βV , due to zero initialised ∆WV

Simplifying Transformer Blocks: Step-by-Step Component Removal for Faster Training

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details 3 PRELIMINARIES A deep transformer architecture of depth L is formed by sequentially stacking L transformer blocks. The most common block is Pre-LN, depicted in Fig. 1 (left), which we treat as a baseline for comparing training speed, both in terms of per-update and runtime. It differs from the original Post-LN block only in the position of the normalisation layers relative to the skip connections, but is more popular as the Post-LN block suffers from poor training stability and signal propagation in deep layers (Xiong et al., 2020; Liu et al., 2020; Noci et al., 2022; He et al., 2023). with scalar gain weights αFF, βFF, αSA, βSA fixed to 1 by default. Here, “MHA” stands for MultiHead Attention (detailed below), and “Norm” denotes a normalisation layer (Ba et al., 2016; Zhang & Sennrich, 2019). In words, we see that the Pre-LN transformer block consists of two sequential sub-blocks (one attention and one MLP), with normalisation layers and residual connections for both sub-blocks, and crucially the normalisation layers are placed within the residual branch. The MLP is usually single hidden-layer, with hidden dimension that is some multiple of d (e.g. 4 (Vaswani et al., 2017) or 8/3 (Touvron et al., 2023)), and acts on each token in the sequence independently. The MHA sub-block allows tokens to share information between one another using self-attention. For input sequence X, the self-attention mechanism outputs: This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details 3 PRELIMINARIES A deep transformer architecture of depth L is formed by sequentially stacking L transformer blocks. The most common block is Pre-LN, depicted in Fig. 1 (left), which we treat as a baseline for comparing training speed, both in terms of per-update and runtime. It differs from the original Post-LN block only in the position of the normalisation layers relative to the skip connections, but is more popular as the Post-LN block suffers from poor training stability and signal propagation in deep layers (Xiong et al., 2020; Liu et al., 2020; Noci et al., 2022; He et al., 2023). with scalar gain weights αFF, βFF, αSA, βSA fixed to 1 by default. Here, “MHA” stands for MultiHead Attention (detailed below), and “Norm” denotes a normalisation layer (Ba et al., 2016; Zhang & Sennrich, 2019). In words, we see that the Pre-LN transformer block consists of two sequential sub-blocks (one attention and one MLP), with normalisation layers and residual connections for both sub-blocks, and crucially the normalisation layers are placed within the residual branch. The MLP is usually single hidden-layer, with hidden dimension that is some multiple of d (e.g. 4 (Vaswani et al., 2017) or 8/3 (Touvron et al., 2023)), and acts on each token in the sequence independently. The MHA sub-block allows tokens to share information between one another using self-attention. For input sequence X, the self-attention mechanism outputs: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Improving Training Stability in Deep Transformers: Pre-LN vs. Post-LN Blocks

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details 2 RELATED WORK Simplifying deep NNs by removing block components has received a lot of attention, both in transformers and other architectures. In these works, signal propagation theory often acts as inspiration. It has been shown that judicious use of weight initialisations and architectural tools, like skip connections and normalisation layers, can improve signal propagation degeneracies and the trainability of deep NNs. Such considerations have motivated principled modifications with simpler architectures. De & Smith (2020) show that an implicit mechanism of Pre-LN skip connections is to downweight the residual branch relative to the skip branch, leading to better signal propagation. They also show that explicitly downweighting the residual branch allows normalisation layers to be removed without affecting performance. The idea of downweighting residuals for improved signal propagation & trainability has been studied extensively in the literature (Zhang et al., 2018; Hanin & Rolnick, 2018; Tarnowski et al., 2019; Zhang et al., 2019; Arpit et al., 2019; Xu et al., 2020; Bachlechner et al., 2021; Touvron et al., 2021; Hayou et al., 2021; Hayou & Yang, 2023; Martens et al., 2021; Davis et al., 2021; Noci et al., 2022; Wang et al., 2022a; Huang et al., 2020; Wang et al., 2022b). For skip connections (He et al., 2016), it has been shown that transforming non-linear activation functions in MLPs and CNNs to be more linear according to a given deep architecture can enable good signal propagation even without skip connections (Martens et al., 2021; Zhang et al., 2022; Li et al., 2022). He et al. (2023) apply similar considerations to the self-attention mechanism, where the key insight is that attention matrices need to be more identity-like in order to prevent signal degradation in skipless transformers. However, these works find that skipless architectures suffer from significant losses in training speed compared to their residual counterparts, when using standard optimisers like SGD or Adam. Such differences were not observed with stronger optimisers like K-FAC (Martens & Grosse, 2015) on CNNs, and this inability to explain training phenomena highlights a current limitation of signal propagation theory. Ding et al. (2021; 2023) design a CNN, RepVGG, that can be trained like a residual architecture for fast per-update convergence, but reparameterised to be skipless at test time for significantly higher inference throughput. This reparameterisation is related to our considerations of value and projection parameters in Sec. 4. Many works have considered simplifications or improvements specific to the transformer architecture. Most relevant to our work is the parallel block of Wang & Komatsuzaki (2021) (pictured Fig. 1, bottom right), which computes the MLP and attention sub-blocks in parallel for efficiency gains, with minimal performance loss. Trockman & Kolter (2023) observe that the product of value and projection parameters often has a large identity component in trained transformers, and design an initialisation mimicking this to improve performance in standard transformers on small datasets. We find these matrices can be fixed to the identity without loss of performance, which removes them from our simplified architecture. Other works have considered reducing the frequency of MLP sub-blocks (Sridhar et al., 2022; Pires et al., 2023) or efficient replacements to softmax attention (Katharopoulos et al., 2020; Schlag et al., 2021; Choromanski et al., 2021). Sukhbaatar et al. (2019) remove the MLP by integrating it into the attention sub-block, augmented with persistent memory This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details 2 RELATED WORK Simplifying deep NNs by removing block components has received a lot of attention, both in transformers and other architectures. In these works, signal propagation theory often acts as inspiration. It has been shown that judicious use of weight initialisations and architectural tools, like skip connections and normalisation layers, can improve signal propagation degeneracies and the trainability of deep NNs. Such considerations have motivated principled modifications with simpler architectures. De & Smith (2020) show that an implicit mechanism of Pre-LN skip connections is to downweight the residual branch relative to the skip branch, leading to better signal propagation. They also show that explicitly downweighting the residual branch allows normalisation layers to be removed without affecting performance. The idea of downweighting residuals for improved signal propagation & trainability has been studied extensively in the literature (Zhang et al., 2018; Hanin & Rolnick, 2018; Tarnowski et al., 2019; Zhang et al., 2019; Arpit et al., 2019; Xu et al., 2020; Bachlechner et al., 2021; Touvron et al., 2021; Hayou et al., 2021; Hayou & Yang, 2023; Martens et al., 2021; Davis et al., 2021; Noci et al., 2022; Wang et al., 2022a; Huang et al., 2020; Wang et al., 2022b). For skip connections (He et al., 2016), it has been shown that transforming non-linear activation functions in MLPs and CNNs to be more linear according to a given deep architecture can enable good signal propagation even without skip connections (Martens et al., 2021; Zhang et al., 2022; Li et al., 2022). He et al. (2023) apply similar considerations to the self-attention mechanism, where the key insight is that attention matrices need to be more identity-like in order to prevent signal degradation in skipless transformers. However, these works find that skipless architectures suffer from significant losses in training speed compared to their residual counterparts, when using standard optimisers like SGD or Adam. Such differences were not observed with stronger optimisers like K-FAC (Martens & Grosse, 2015) on CNNs, and this inability to explain training phenomena highlights a current limitation of signal propagation theory. Ding et al. (2021; 2023) design a CNN, RepVGG, that can be trained like a residual architecture for fast per-update convergence, but reparameterised to be skipless at test time for significantly higher inference throughput. This reparameterisation is related to our considerations of value and projection parameters in Sec. 4. Many works have considered simplifications or improvements specific to the transformer architecture. Most relevant to our work is the parallel block of Wang & Komatsuzaki (2021) (pictured Fig. 1, bottom right), which computes the MLP and attention sub-blocks in parallel for efficiency gains, with minimal performance loss. Trockman & Kolter (2023) observe that the product of value and projection parameters often has a large identity component in trained transformers, and design an initialisation mimicking this to improve performance in standard transformers on small datasets. We find these matrices can be fixed to the identity without loss of performance, which removes them from our simplified architecture. Other works have considered reducing the frequency of MLP sub-blocks (Sridhar et al., 2022; Pires et al., 2023) or efficient replacements to softmax attention (Katharopoulos et al., 2020; Schlag et al., 2021; Choromanski et al., 2021). Sukhbaatar et al. (2019) remove the MLP by integrating it into the attention sub-block, augmented with persistent memory This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Simplifying Transformer Blocks: Related Work

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details ABSTRACT A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable. In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical observations, we motivate modifications that allow many block components to be removed with no loss of training speed, including skip connections, projection or value parameters, sequential sub-blocks and normalisation layers. In experiments on both autoregressive decoder-only and BERT encoder-only models, our simplified transformers emulate the per-update training speed and performance of standard transformers, while enjoying 15% faster training throughput, and using 15% fewer parameters 1 INTRODUCTION The transformer architecture (Vaswani et al., 2017) is arguably the workhorse behind many recent successes in deep learning. A simple way to construct a deep transformer architecture is by stacking multiple identical transformer “blocks” one after another in sequence. Each block, however, is more complicated and consists of many different components, which need to be combined in specific arrangements in order to achieve good performance. Surprisingly, the base transformer block has changed very little since its inception, despite attracting the interest of many researchers. In this work, we study whether the standard transformer block can be simplified. More specifically, we probe the necessity of several block components, including skip connections, projection/value matrices, sequential sub-blocks and normalisation layers. For each considered component, we ask if it can be removed without loss of training speed (both in terms of per-update step & runtime), and what architectural modifications need to be made to the transformer block in order to do so. We believe the problem of simplifying transformer blocks without compromising training speed is an interesting research question for several reasons. First, modern neural network (NN) architectures have complex designs with many components, and it is not clear the roles played by these different components in NN training dynamics, nor how they interact with each other. This is particularly pertinent given the existing gap between theory and practice in deep learning, where theorists working to understand the mechanisms of deep learning often only consider simplified architectures due to convenience, not necessarily reflective of modern architectures used in practice. Simplifying the NN architectures used in practice can help towards bridging this divide On a related theoretical note, our work highlights both strengths and current limitations of signal propagation: a theory that has proven influential due to its ability to motivate practical design choices in deep NN architectures. Signal propagation (Poole et al., 2016; Schoenholz et al., 2017; Hayou et al., 2019) studies the evolution of geometric information in an NN at initialisation, captured through inner products of layerwise representations across inputs, and has inspired many impressive results in training deep NNs (Xiao et al., 2018; Brock et al., 2021; Martens et al., 2021; Zaidi et al., 2023). However, the current theory only considers a model at initialisation, and often considers only the initial forward pass. As such, signal propagation at present is unable to shed light on many intricacies of deep NN training dynamics, for example the benefits of skip connections for training speed. Though signal propagation is crucial in motivating our modifications, we would not have arrived at our simplified transformer blocks from theory alone, and relied also on empirical insights. Finally, on the practical side, given the exorbitant cost of training and deploying large transformer models nowadays, any efficiency gains in the training and inference pipelines for the transformer architecture represent significant potential savings. Simplifying the transformer block by removing non-essential components both reduces the parameter count and increases throughput in our models. In particular, we show that it is possible to remove skip connections, value parameters, projection parameters and sequential sub-blocks, all while matching the standard transformer in terms of training speed and downstream task performance. As a result, we reduce parameter count by up to 16% and observe throughput increases of 16% at both train and inference time. Our starting point for simplifying Transformer blocks is He et al. (2023), who show that respecting signal propagation principles allows one to train deep Transformers without skip connections or normalisation layers, but at significantly reduced convergence speeds per parameter update. We first show that regulating the updates to values and projection parameters (Sec. 4.1), or in fact removing them entirely (Sec. 4.2), improves the performance of skipless attention sub-blocks, and recovers the lost per-update training speed reported by He et al. (2023). This removes half of the parameters and matrix-multiplications in the attention sub-block. In Sec. 4.3, we show our simplifications combine profitably with parallel sub-blocks (Wang & Komatsuzaki, 2021), which allows us to remove all remaining skip connections and sequential sub-blocks without compromising per-update training speed, whilst further boosting the throughput increase to be 16%, in our implementation. Finally, in Sec. 5, we show that our simplified blocks improve when scaled to larger depths, work well in both encoder-only and decoder-only architectures, and that our findings also hold when scaling training length. We conclude with a discussion of limitations and future work in Sec. 6. This paper is available on arxiv under CC 4.0 license. Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details ABSTRACT A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable. In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical observations, we motivate modifications that allow many block components to be removed with no loss of training speed, including skip connections, projection or value parameters, sequential sub-blocks and normalisation layers. In experiments on both autoregressive decoder-only and BERT encoder-only models, our simplified transformers emulate the per-update training speed and performance of standard transformers, while enjoying 15% faster training throughput, and using 15% fewer parameters 1 INTRODUCTION The transformer architecture (Vaswani et al., 2017) is arguably the workhorse behind many recent successes in deep learning. A simple way to construct a deep transformer architecture is by stacking multiple identical transformer “blocks” one after another in sequence. Each block, however, is more complicated and consists of many different components, which need to be combined in specific arrangements in order to achieve good performance. Surprisingly, the base transformer block has changed very little since its inception, despite attracting the interest of many researchers. In this work, we study whether the standard transformer block can be simplified. More specifically, we probe the necessity of several block components, including skip connections, projection/value matrices, sequential sub-blocks and normalisation layers. For each considered component, we ask if it can be removed without loss of training speed (both in terms of per-update step & runtime), and what architectural modifications need to be made to the transformer block in order to do so. We believe the problem of simplifying transformer blocks without compromising training speed is an interesting research question for several reasons. First, modern neural network (NN) architectures have complex designs with many components, and it is not clear the roles played by these different components in NN training dynamics, nor how they interact with each other. This is particularly pertinent given the existing gap between theory and practice in deep learning, where theorists working to understand the mechanisms of deep learning often only consider simplified architectures due to convenience, not necessarily reflective of modern architectures used in practice. Simplifying the NN architectures used in practice can help towards bridging this divide On a related theoretical note, our work highlights both strengths and current limitations of signal propagation: a theory that has proven influential due to its ability to motivate practical design choices in deep NN architectures. Signal propagation (Poole et al., 2016; Schoenholz et al., 2017; Hayou et al., 2019) studies the evolution of geometric information in an NN at initialisation, captured through inner products of layerwise representations across inputs, and has inspired many impressive results in training deep NNs (Xiao et al., 2018; Brock et al., 2021; Martens et al., 2021; Zaidi et al., 2023). However, the current theory only considers a model at initialisation, and often considers only the initial forward pass. As such, signal propagation at present is unable to shed light on many intricacies of deep NN training dynamics, for example the benefits of skip connections for training speed. Though signal propagation is crucial in motivating our modifications, we would not have arrived at our simplified transformer blocks from theory alone, and relied also on empirical insights. Finally, on the practical side, given the exorbitant cost of training and deploying large transformer models nowadays, any efficiency gains in the training and inference pipelines for the transformer architecture represent significant potential savings. Simplifying the transformer block by removing non-essential components both reduces the parameter count and increases throughput in our models. In particular, we show that it is possible to remove skip connections, value parameters, projection parameters and sequential sub-blocks, all while matching the standard transformer in terms of training speed and downstream task performance. As a result, we reduce parameter count by up to 16% and observe throughput increases of 16% at both train and inference time. Our starting point for simplifying Transformer blocks is He et al. (2023), who show that respecting signal propagation principles allows one to train deep Transformers without skip connections or normalisation layers, but at significantly reduced convergence speeds per parameter update. We first show that regulating the updates to values and projection parameters (Sec. 4.1), or in fact removing them entirely (Sec. 4.2), improves the performance of skipless attention sub-blocks, and recovers the lost per-update training speed reported by He et al. (2023). This removes half of the parameters and matrix-multiplications in the attention sub-block. In Sec. 4.3, we show our simplifications combine profitably with parallel sub-blocks (Wang & Komatsuzaki, 2021), which allows us to remove all remaining skip connections and sequential sub-blocks without compromising per-update training speed, whilst further boosting the throughput increase to be 16%, in our implementation. Finally, in Sec. 5, we show that our simplified blocks improve when scaled to larger depths, work well in both encoder-only and decoder-only architectures, and that our findings also hold when scaling training length. We conclude with a discussion of limitations and future work in Sec. 6. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about DUELIT

Duelit’s Stories on HackerNoon

Duelit's News & Mentions