Mezmo

mezmo

Tech is both the future and the road that will get us there. Whether it’s an adjacent digital reality- that integrates seamlessly with a tangible one- or an app that tracks your heart rate, we depend more and more on innovative solutions to problems we didn’t even know we had. The 2022 Noonies’ Award Category, presented by ,  puts the spotlight on the builders that improve our lives on a daily basis, the creative minds that help build these solutions, and by doing so light our path to an exciting future. Programming Slogging Here are the stats: 4,743 nominees. 371 Hacker Noon awards for 2022 Across 5 award categories: Web3, Emerging Tech, Gaming, Internet Heroes, and Programming. And a monumental 143,156 votes cast over at noonies.tech To see the winners simply visit , pick an award, and discover which name appears at the top. That’s the winner! noonies.tech The category alone is announcing 110 winners. Let’s congratulate the winners! Programming HackerNoon Contributor of the Year- JAVA is a Software Engineer and published author currently working as a Developer Advocate at MariaDB Corporation. Alejandro Duarte HackerNoon Contributor of the Year- KUBERNETES Aside from being a Content Strategist at Valasys Media, is  a Psychic, Tarot Reader and Violet Flame Alchemist. Priya Kumari HackerNoon Contributor of the Year- CODING is a Technical Leader with a hobby of competitive programming. Ruslan Rakhmedov HackerNoon Contributor of the Year- SQL is a Software Engineer and published author currently working as a Developer Advocate at MariaDB Corporation. Alejandro Duarte HackerNoon Contributor of the Year- DEVOPS Aside from being a Content Strategist at Valasys Media, is  a Psychic, Tarot Reader and Violet Flame Alchemist. Priya Kumari HackerNoon Contributor of the Year- CODING-SKILLS is a Senior Ruby on Rails Developer with 2+ YoE in developing e-commerce, travel, and blockchain projects. Alexey Shepelev HackerNoon Contributor of the Year- DATABASE is a Software Engineer and published author currently working as a Developer Advocate at MariaDB Corporation. Alejandro Duarte HackerNoon Contributor of the Year- CICD has 9+ years of professional experience as a Solutions architect, Tech lead, Cloud Architect and in DevOps. Viachaslau Matsukevich HackerNoon Contributor of the Year- CLOUD-COMPUTING Congratulations to , who loves coding in C# and Cloud Computing. Uladzislau Baryshchyk HackerNoon Contributor of the Year- AWS Congratulations to once again! Uladzislau Baryshchyk HackerNoon Contributor of the Year- MICROSERVICES brings over 19 years of experience in Software Development and a University degree in Computer Science. Oleg Melnic HackerNoon Contributor of the Year- GOLANG is a seasoned Software engineer Skilled in Go, Java, and JavaScript. Yuda Prasetiya Haqqy HackerNoon Contributor of the Year- CSS is a backend software engineer at Snap Inc,  interested in distributed systems, web security, and building cool things. Stanley Lim HackerNoon Contributor of the Year- PROGRAMMING is an Experienced Software Quality Assurance Tester at Libertex Group. Sergei Shaikin Coding Guru of 2022 is a Senior Software Engineer at Layermark. Alexander Makeev HackerNoon Contributor of the Year- MONITORING is a backend software engineer at Snap Inc,  interested in distributed systems, web security, and building cool things. Stanley Lim HackerNoon Contributor of the Year- GITHUB is a Software engineer with 11 years of experience focused on fault tolerant, distributed systems, PaaS, golang, HA. Anton Gubarev HackerNoon Contributor of the Year- DOCKER is a Software Engineer and published author, winning this year Noonies! Alejandro Duarte Tech Tutorial Teacher of the Year will solve all of your coding issues! Code basics HackerNoon Contributor of the Year- REACT is a Skilled Front-End engineer with 7 years of experience in developing Web and SmartTV applications. Yan Tsishko HackerNoon Contributor of the Year- FLUTTER is the Chief Technology Officer at Estatic Infotech Pvt. Ltd. Kiran Beladiya HackerNoon Contributor of the Year- API is an amazing data engineer and rockstar python teacher, winning this year's Noonies. Veronika Vasileva HackerNoon Contributor of the Year- CLEAN-CODE is a Senior software engineer & agile lover who specializes in declarative designs and S.O.L.I.D. Maximiliano Contieri HackerNoon Contributor of the Year- HOW-TO is a Senior Software Engineer at Akma Trading. Vladislav Gukasov HackerNoon Contributor of the Year- FUNCTIONAL-PROGRAMMING is a Staff Software Engineer at Lab49 with large hands-on experience on efficiently coding web apps. Georgii Kliukovkin Software Developer of The Year Asides managing Developer Experience & Relations at StepZen, is a Tech Author, Speaker and Entrepreneur. Roy Derks HackerNoon Contributor of the Year- SOFTWARE-ARCHITECTURE is a Senior Software Engineer with 7+ YoE building massively scalable systems both from scratch and diving into a codebase. Sergey Golitsyn HackerNoon Contributor of the Year- KOTLIN is a Senior Software Engineer at Ultimeta with over 10+ years of experience in software development with Java, Kotlin, and Spring. Fedor Yaremenko Top Programming Guru is a Principal Product Engineer at Esri. Hussein Nasser HackerNoon Contributor of the Year- JAVASCRIPT is a Frontend Developer and Technical Writer. Amarachi Iheanacho HackerNoon Contributor of the Year- GIT is a Software Engineer and published author currently working as a Developer Advocate at MariaDB Corporation. Alejandro Duarte iOS Writer of the Year Congratulations to , the Chief Technology Officer at Estatic Infotech Pvt. Ltd. Kiran Beladiya HackerNoon Contributor of the Year- DATA-SCIENCE Congratulations to , an avid data science writer currently studying Computer Science at Penn State University. Max Vidavsky HackerNoon Contributor of the Year- PRODUCT-DEVELOPMENT Congratulations to , the Chief Technology Officer at Estatic Infotech Pvt. Ltd. Kiran Beladiya Android Advocate of The Year Congratulations to once again!. Kiran Beladiya HackerNoon Contributor of the Year- SOFTWARE-TESTING Congrats to , a software engineer with a love for backend testing! Sergei Shaikin HackerNoon Contributor of the Year- MONGODB , Senior Ruby on Rails Developer, wins this award! Alexey Shepelev HackerNoon Contributor of the Year- LINUX The winner is , on his way to enlightenment! Aleksandr Tabakov HackerNoon Contributor of the Year- TESTING Congratulations to , Head of Product Marketing at Livecycle! Zevi Reinitz HackerNoon Contributor of the Year- DATA The award goes to… l, a Data Scientist, applied scientist, research consultant, freelancer and startup builder. Shaurya Uppa HackerNoon Contributor of the Year- DOTNET is the winner this year! Rahul Rai HackerNoon Contributor of the Year- DEBUGGING Congratulations to ! He is a professional fullstack software engineer, currently working for REVOLUT as a software backend engineer. Artem Sutulov HackerNoon Contributor of the Year- AUTHENTICATION is the winner! Rodrigo Kamada HackerNoon Contributor of the Year- FRONTEND Congratulations to , Frontend Developer And Technical Writer. Amarachi Iheanacho HackerNoon Contributor of the Year- AZURE Congratulations to , a solutions architect with expertise in public cloud, kubernetes, PAAS and CI\CD. Viachaslau Matsukevich HackerNoon Contributor of the Year- BACKEND wins his second award! Artem Sutulov HackerNoon Contributor of the Year- SERVERLESS is a solutions architect with expertise in public cloud, kubernetes, PAAS and CI\CD. Viachaslau Matsukevich HackerNoon Contributor of the Year- QUALITY-ASSURANCE is on fire! Congratulations! Ruslan Muradov HackerNoon Contributor of the Year- COMPUTER-SCIENCE Congratulations to ! Rishabh Agarwal HackerNoon Contributor of the Year- HEROKU , senior python developer, wins this award. Maksim Kuznetsov HackerNoon Contributor of the Year- REST-API Congratulations to ! Alexey Shepelev HackerNoon Contributor of the Year- NPM Congratulations to , Web/Blockchain Engineer, Technical Writer. Idris Abdul-Lateef HackerNoon Contributor of the Year- PERFORMANCE Congratulations to , a Senior Software Engineer. Yura Abharian HackerNoon Contributor of the Year- LOW-CODE , a passionate software engineer, wins this award. Aleksandr Zakharov HackerNoon Contributor of the Year- ANGULAR is the winner! Rodrigo Kamada HackerNoon Contributor of the Year- AGILE Congratulations to , a Software Engineering Manager at UTMOST, a Devops Advocate, passionate about Startups and continuous improvement pursuer Alessandro Diaferia HackerNoon Contributor of the Year- AGILE-SOFTWARE-DEVELOPMENT Congratulations to once again! Alessandro Diaferia Open Source Writer of the Year ’s motto is "EVERY DAY IS A LEARNING DAY". She is a Growing Developer, Python and Open-Source and a Tech Enthusiasts & Writer. Hillary Nyakundi Data Science Demon Congratulations to , a Data Scientist, a Problem Solving Expert and a Research Consultant. Shaurya Uppal HackerNoon Contributor of the Year- WEBSITE-DEVELOPMENT brings home another award! Rodrigo Kamada HackerNoon Contributor of the Year- OPTIMIZATION , a Data Engineer and a Python Tutor wins! Nikita Vasilev Master on the Edge of Computing Congratulations to for another win! Viachaslau Matsukevich HackerNoon Contributor of the Year- SOFTWARE-ENGINEERING Congratulations to ! Artem Sutulov HackerNoon Contributor of the Year- SCALING wins! Congratulations! Ralph Benko HackerNoon Contributor of the Year- CONTAINERS has been a pioneer in video software development since 2005. Congrats! Fora Soft HackerNoon Contributor of the Year- TYPESCRIPT , currently developing and enjoying life, and meanwhile winning Noonies. Anatolii Kabanov Rawr for ROR , a 6+ years Java Developer, crashed this award! Alexander Makeev HackerNoon Contributor of the Year- SWIFT ,  a senior IOS Software Engineer at BestDoctor has won the HackerNoon Contributor of the Year- Swift. Swift is designed to function as the intuitive language for ios, ipados, macos, watchos, and tvos. Pavel Plotnikov Most Authentic Developer Advocate of The Year has your back! Dev Advocate at DraftBit and ReactNative & Expo Enthusiast and now, the most authentic dev advocate of the year. Aman Mittal HackerNoon Contributor of the Year- NEURAL-NETWORKS The crux of deep learning algorithms: Neural Networks is taken over by , a researcher / engineer with 10+ YoE in deep learning, computer vision, and autonomous driving. Dmitrii Khizbullin HackerNoon Contributor of the Year- HTML Build better websites with , a Senior Ruby on Rails Developer who has won the HTML Contributor of the Year at HackerNoon. Alexey Shepelev HackerNoon Contributor of the Year- HACKING wins the HackerNoon Contributor of the Year: Hacking. They write tutorials on Cybersecurity and Bug Bounty on our website and HackerNoon. Kali Linux Tutorials HackerNoon Contributor of the Year- INFRASTRUCTURE Build better websites with , a Senior Ruby on Rails Developer who has won the HTML Contributor of the Year at HackerNoon. Alexey Shepelev HackerNoon Contributor of the Year- REDUX takes the award for Contributor of the Year- Redux. Redux is an open source javascript library, commonly used with libraries like react or angular for building user interfaces. Anatolii Kabanov No No No NodeJs wins their second title for NodeJs. Congratulations X2! Anatolii Kabanov HackerNoon Contributor of the Year- ENCRYPTION is an eater of bitcoin stories, spinner of blockchain tales, and lastly the winner of the HackerNoon Contributor of the Year- Encryption. StevieSats HackerNoon Contributor of the Year- REFACTORING This award is for the highest quality refactoring stories on hackernoon and it goes to . Abbey is a full-stack web developer, crafter, blogger, cosplayer, fiber artist, yoga teacher, and gamer. Abbey Perini HackerNoon Contributor of the Year- ENGINEERING is not only a Tech Enthusiast, he is also the HackerNoon Contributor of the Year for Engineering. Rishabh Agarwal HackerNoon Contributor of the Year- RASPBERRY-PI loves Robotics and IoT. Nazar has provided us with the highest quality raspberry-pi stories for HackerNoon and has won the HackerNoon Contributor of the year award. Chingiz Nazar HackerNoon Contributor of the Year- DEVELOPER-TOOLS is a founder at Adadot.com, the world’s first fitness tracker for work, helping developers improve the way they work and feel. His contributions for HackerNoon will be remembered forever. Congratulations Alex! Alex Harris Developer Defying Patriarchy (While Being an Awesome Developer) is a software Engineer at Mezmo and mentor for female tech-newbies and advocate of slaying the imposter syndrome dragon. Sullivan, the defier of patriarchy and the HackerNoon Contributor of the Year, Congratulations! Shanna Sullivan HackerNoon Contributor of the Year- NATURAL-LANGUAGE-PROCESSING is a coder and a writer, with a goal of becoming a streamer someday. Tan is surely a step closer after winning HackerNoon Contributor of the Year for Natural Language Processing. Liling Tan Bug Finder of the Year is the HackerNoon Bug Finder of the Year, reporting bugs and significant vulnerabilities in a tech project. Andrej is also a dedicated writer and digital evangelist. Andrej Kovacevic HackerNoon Contributor of the Year- OPEN-SOURCE is focused on the open source software supply chain to build a better digital future for all of us. We can all appreciate Sal for his contributions and congratulate them for winning the HackerNoon Contributor of the year for Open Source. Sal Kimmich AI Artist Award is a trained neural network called DALL·E that creates images from text captions for a wide range of concepts expressible in natural language. 2022 has seen AI drawing software proliferate like no other and Dall E is one of them. Congratulations! Dall E HackerNoon Contributor of the Year- CODE-QUALITY Whatever your metrics for judgment are, code quality stories on hackernoon provide pointers that ease your way. helps make your Python code easier to read, understand, and work with through automatic refactoring suggestions. Let's congratulate them for being the HackerNoon Contributor of the Year for Code Quality! Sourcery HackerNoon Contributor of the Year- MOBILE is the CTO and Co-Founder of @incorainc, where we can turn your ideas into products. Incorainc is THE website that caters to your short time span. Congratulations on being the HackerNoon Contributor of the Year for Mobile. Tetiana Stoyko Pythonistas' Paradise is THE paradise for pythonistas.  They are a professional provider of data protection solutions for enterprises. Congrats on being the HackerNoon Contributor of the Year! Vinchin Backup & Recovery HackerNoon Contributor of the Year- ARCHITECTURE is a full-stack engineer and architect. Michelangelo would be proud of Andronchik’s contribution and being the HackerNoon Contributor of the Year. George Andronchik The Skynet Award Unconcerned with a global takeover at the hands of AI, this award honors the writers, inventors, or companies that brought us closer to the singularity this year. , an AI research and deployment company with the mission to ensure that artificial general intelligence benefits all of humanity has won the Hackernoon’s Skynet Award. OpenAI HackerNoon Contributor of the Year- APPLICATION Application is a set of instructions encoded in the language a computer understands. This was best reflected by , a Smart Web Vulnerability Scanner and their contributions to HackerNoon. Congratulations on winning the HackerNoon Contributor of the Year for Applications. SmartScanner HackerNoon Contributor of the Year- SCRUM Scrum represents complex products, sustainability and adaptive solutions, in the constantly evolving work and technological landscape. , the principal data consultant at ThoughtWorks has proven themselves to be the HackerNoon Contributor of the Year- Scrum. Ryan Dawson HackerNoon Contributor of the Year- ENGINEERING-MANAGEMENT is a human specialist and former VPE at New Relic, Gremlin. Jade wins the award  for the highest quality engineering-management stories on HackerNoon Jade Rubick HackerNoon Contributor of the Year- FIREBASE Congratulations to , a software engineer at Udgama for winning the HackerNoon Contributor of the Year for Firebase. Snehal Narkar HackerNoon Contributor of the Year- PROJECT-MANAGEMENT Project managers are so meticulous that they remember all the details. is one of them and wins this award with grace! Jade Rubick The Calmer of Storms Debugging is often the most grueling sweat of one’s brow but it easier by providing open-source error tracking to monitor and fix bugs and crashes anywhere in your application in real time. Sentry makes HackerNoon Contributor of the Year- APP-DEVELOPMENT Congratulations to , a Layer-3 peer-to-peer network to securely exchange liquidity & facilitate trading, clearing, settlement, and compliance. Yellow Network SaaSiest CEO of the year , SaaSiest Ceo of Notion, a project management software designed to help members of a company and individual users coordinate objectives, working processes and so forth is the winner! Ivan Zhao Tip your Hat Tutorials - Best Tutorials has some of the best tutorials on backend development, deployment, frameworks and much more. DigitalOcean Blog HackerNoon Contributor of the Year- CODE crashed this award! Oscar Daum HackerNoon Contributor of the Year- DATA-ANALYSIS Congratulations to , a global data labeling company. Toloka HackerNoon Contributor of the Year- GOOGLE-CLOUD-PLATFORM Congratulations to , a Java programmer and blogger! Javin Paul HackerNoon Contributor of the Year- PHP is an experienced Full-Stack Developer and CEO of BootesNull. Surinder Rawat The Best Alternatives to Notepad++ is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as "vi" with most UNIX systems and with Apple OS X. Vim HackerNoon Contributor of the Year- CSS3 We all need a Sensei to guide us through our learning journey and has been our most favorite in CSS3 voyage. Congratulations! Anderson Osayerie HackerNoon Contributor of the Year- SOFTWARE is a Chief Product Officer for Chargify and SaaSOptics. He is passionate about creating products that can maximize his customer’s ease. Barrow Hamilton HackerNoon Contributor of the Year- CMS is a passionate writer interested in new technologies, a dog lover and a digital nomad. Kat HackerNoon Contributor of the Year- DJANGO Congratulations to who is a Software engineer, blogger, lifelong learner with an added badge to his armor as this year’s DJANGO master . Ismail Tlemcani HackerNoon Contributor of the Year- INFORMATION-SECURITY Congratulations to for keeping us well informed on how to stay safe in the virtual world jasonhiggins HackerNoon Contributor of the Year- LARAVEL To the artisan that made learning Laravel easier for all of us. Congratulations ! Adam Schmideg

#Noonies2022 Awards: The List of Winners in the Programming Category

Ever laugh about “leaky abstractions” with your dev buddies at a conference? Yeah, it’s hilarious, until you’re the one stuck debugging a memory leak at 3 a.m., cursing your code and chugging energy drinks. If you’ve been there, you know that sinking feeling when your app starts choking, and you’re scrambling to figure out why. Memory leaks are the worst. They’re like roaches in your codebase, sneaking around, gobbling up memory until your app slows to a crawl or just crashes. Hard. And the kicker? Even your cleanest, most brilliant code can turn into a hot mess if leaks get out of control. But you don’t have to just sit there and take it. I’ve got your back with 5 dead-simple, no-BS ways to bulletproof your code against memory leaks. These are must-haves if you’re working on high-traffic apps or services that need to stay up forever. Because, honestly, nobody gives a damn about your perfect code if your app keeps tanking. Let’s squash those leaks for good. Ready? 5 Practical Ways To Harden Your Code Against Memory Leaks 1. Avoid overriding finalize() in Java Java’s finalize() sounds useful, but in reality? It’s a bit of a disaster under the hood. It creates more problems than solutions. The issue is that once you override finalize(), Java has to move through extra hoops. And when an object has a finalize() method, Java puts it into a special queue, and a special thread runs it. Only after that can the memory actually be cleaned up, and this delays garbage collection time. Objects with finalize() can take longer to clean up, which is why your app might be slowing down without warning. In fact, some garbage collectors don’t play well with finalize(), which leads to more collection cycles and extra work. The result? Your app gets to take the hit. Here’s the fix: Here’s the fix: Here’s the fix: Implement Autoclosable with a clean close() method.
Use try-with-resources so that Java can handle cleanup automatically.
Always double-check subclasses so that they won’t inherit finalize() logic silently.
Use WeakReference or PhantomReference for caching.
Constantly clean up native resources like file handles and sockets Implement Autoclosable with a clean close() method. Use try-with-resources so that Java can handle cleanup automatically. Always double-check subclasses so that they won’t inherit finalize() logic silently. Use WeakReference or PhantomReference for caching. Constantly clean up native resources like file handles and sockets 2. Use Object Pooling in .NET Applications Object pooling is an effective way to optimize memory usage and application performance. Sometimes, when your application is experiencing instability, you might only have to reduce how objects are created, used, and reused. These are the issues object pooling aims to fix. At its core, object pooling is a clever way of reusing existing objects rather than creating new ones from scratch. How is this smart? By reusing objects, the pressure on the garbage collector is lifted. And this will simply increase smooth app performance and avert fewer pauses. This approach has two extra benefits: it saves memory and cuts down time to allocate and deallocate resources. That sounds like a win-win to me. Not to be a party pooper here. But, here’s a little warning: Pooling can slow things down if you don’t need it. This is why Microsoft recommends testing in real-life scenarios before implementing. Follow these steps to find out how to implement object pooling in your .NET applications: Here’s the fix: Here’s the fix: Here’s the fix: Use dotMemory or any efficient profiling tool to find objects that are often created, yet have short lives.
Make custom ObjectPool policies that clear leftover data before reusing objects.
Use try/finally blocks to ensure borrowed objects are returned to the pool.
Benchmark your application before and after pooling to measure performance. Use dotMemory or any efficient profiling tool to find objects that are often created, yet have short lives. Make custom ObjectPool policies that clear leftover data before reusing objects. Use try/finally blocks to ensure borrowed objects are returned to the pool. Benchmark your application before and after pooling to measure performance. 3. Execute Cleanup in React useEffect Hooks When something starts running in the background that shouldn’t, your app will most likely begin to behave oddly. This type of memory leak happens in React apps when components hold on to things even after being unmounted. This is typically a result of asynchronous tasks or persistent references outliving the components that started them. A common instance is when an event listener is still active even after the absence of the component. Another typical example is subscriptions to data sources that were never unsubscribed, along with several other issues. Thankfully, React provides a cleanup hook, useEffect, to solve this problem. Now, developers can find a way to clean up components before re-running or unmounting. The benefits of this function are almost endless. Finally, you can clear timers, cancel subscriptions, and remove event listeners. All by executing a simple step? Yes. But what tops all of them is that this simple step frees up memory and keeps your app efficient. Want your React applications to be stable over time? Then, cleanup is a must. Let me show you how to go about it. Here’s the fix: Here’s the fix: Here’s the fix: useEffect ( () => { … } , [ ]);
This hook should run once after the component mounts. It’s also where you place side effects. useEffect ( () => { … } , [ ]); This hook should run once after the component mounts. It’s also where you place side effects. let  isMounted = true;
This tracks whether the component is still mounted. let  isMounted = true; This tracks whether the component is still mounted. const fetchData = async () => { … }
This fetches data from an external API. const fetchData = async () => { … } This fetches data from an external API. const controller = new AbortController ();
This cancels a fetch request if the component unmounts. const controller = new AbortController (); This cancels a fetch request if the component unmounts. const response = await fetch ( ‘ https: //api.youandme.com/data’, { signal } );
This sends the actual HTTP request to the API. const response = await fetch ( ‘ https: //api.youandme.com/data’, { signal } ); This sends the actual HTTP request to the API. const data = await response. json();
This parses the returned response as JSON then recieve information fetched from the API const data = await response. json(); This parses the returned response as JSON then recieve information fetched from the API if (isMounted) { setData (data); } if (isMounted) { setData (data); } Catch (error) { … }
This catches and logs every errors during fetch and ignores AbortError. Catch (error) { … } This catches and logs every errors during fetch and ignores AbortError. fetchData ();
This requests the async function to commence the data-fetching process. fetchData (); This requests the async function to commence the data-fetching process. return () => { … }
This is the cleanup function of useEffect that runs right before the components unmounts. return () => { … } This is the cleanup function of useEffect that runs right before the components unmounts. isMounted = false;
This ensures that we don’t update state after the component is gone. isMounted = false; This ensures that we don’t update state after the component is gone. Controller.abort ();
This cancels the fetch request if it’s still ongoing and prevents memory leaks. Controller.abort (); This cancels the fetch request if it’s still ongoing and prevents memory leaks. 4. Fix equals () and hashCode () in Java Collections It’s so easy for us to focus on the big things in Java development, but sometimes it’s the small details that cause the biggest problems. One of those little details: the proper use of equals () and hashCode(). I won’t be surprised if you are wondering how these simple methods could cause a memory leak. Java development Well, these two seemingly simple methods are the core of how Java handles objects in HashMap or HashSet. And if implemented incorrectly? Things may go downhill. Many developers slip up by overriding equals() and forget to override hashCode(). This would result in the collection receiving and storing duplicates. Over time, the application would hold on to objects that it shouldn’t. The result? Memory bloat. This won’t crash your app right away, but it will make it unresponsive. Here’s the fix: Here’s the fix: Here’s the fix: @0verride public boolean equals(0bject o) { /*...*/ } @0verride public int hashCode()  { /*...*/ }
Do well to override both methods together and not just one method. @0verride public boolean equals(0bject o) { /*...*/ } @0verride public int hashCode()  { /*...*/ } Do well to override both methods together and not just one method. return 0bjects .hash (id, name); // same fields as equals()
Do well to make hashCode() return consistent results for your equal objects. return 0bjects .hash (id, name); // same fields as equals() Do well to make hashCode() return consistent results for your equal objects. private final String name; // immutable field used in equals/hashCode
Do well to use immutable fields for calculations. private final String name; // immutable field used in equals/hashCode Do well to use immutable fields for calculations. equals: return this.id == o.id; hashCode: return Integer.hashCode (id);
Include the exact fields in both methods. equals: return this.id == o.id; hashCode: return Integer.hashCode (id); Include the exact fields in both methods. Use Weak References For Cache Management Use Weak References For Cache Management Memory management often spring up as a concern when building a cache. Of course, no developer would want their application to hold on to memory longer than it should. That’s exactly where weak references step in. Why are weak references so valuable in managing cache? Well, it all comes down to their abilities to allow memory to be reclaimed especially when it isn’t in use. This means that if a memory isn’t in use, it can be cleaned and the reference can be cleared. It’s very important to note that different platforms offer different variety of weak references. In JavaScript, WeakMap and WeakSet always come in handy. WeakMap is perfect for attaching temporary metadata to objects without affecting their lifespan in memory. On the other hand, WeakSet is ideal when grouping objects that you don’t necessary want to keep alive. And in Java, WeakReference is great when building collections that hold onto objects only when the app till needs them. Final Thoughts One hard truth? Preventing memory leaks is not something you do once and forget. You have to build it into the way you code, test, and deploy. It also requires consistency, so you can’t just go a week of heroic leak fixing and go back to old habits. Teams that apply these good memory management practices that I have shared are guaranteed to have apps that thrive. At the end of the day, it’s a long-lasting, thriving application that gets to be appreciated.

5 Ways To Fortify Your Code Against Memory Leaks

As neuroscience research and adjacent programs grapple with funding and staff cuts, the question is this: Where did neuroscience research falter in the last few years? The Substance Abuse and Mental Health Services Administration [SAMHSA] is losing funding and headcount. The National Institutes of Health [NIH] is undergoing a major surgery. There are high-level questions against psychiatric medications. Universities are cutting down on graduate admissions for several biomedical programs. losing funding and headcount undergoing a major surgery What were the major events in the mainstream, in the last few years, that neuroscience research did not show its might or importance, resulting in gaps that may have contributed to its current fate? Mental health was on the headlines, yet with all the talk of therapy, policy, and so forth, the field of neuroscience could not define what mental health is, or is not, or why some therapies work, or do not, at least conceptually—within the mechanisms of the relevant components in the cranium. Addiction was in the news with major reports on overdose deaths. Although reversal medications were useful, some of the answers about what addiction might beand how to prevent it, that would have percolated into prevention messages, were not available from neuroscience. define what mental health is, or is not, or why some therapies work what addiction might be However, there was one major situation where it appeared that neuroscience self-subverted. It was in the aftermath of the report of Special Counsel Robert Hur on then-President Joe Biden's possession of classified documents post-vice-presidency. There were several news stories with quotes from experts about memory and aging. There were opinion articles as well. While some of them noted that it was possible for memory not to be as efficient in old age, there was a generalized theme of better judgment, wisdom, proper memory tests, impulse control, knowledge, and several other points. The problem is not that some of the points may not be accurate. But, if a neuroscience expert were asked about memory and there was talk of wisdom, what is the difference between how the brain mechanizes memory and how it mechanizes wisdom—or judgment? The question is beyond the use of common terms for descriptions because of the general news audience since what is called memory is a label—not a mechanism within the cranium. mechanism within the cranium How does human memory work? This is different from stating that the unit of memory is an engram, or that there is long-term memory, or working memory, or that there is memory consolidation, encoding, retrieval, interference, central executive, default mode, and other terms that are used to describe observations about the human memory. It is possible to be articulate in all these expressions and not be describing the mechanisms of the human memory. The labels are not the actual science of human memory. It would be like saying that velocity is the engineering of automobiles. The question is that if an individual sees a chair, what is the configuration or representationof that chair in the mind [as a memory]? How is it possible to identify that a remote control in the room is misplaced and to correct it? what is the configuration or representation If the consensus is that there is no clear understanding of how the human memory works or how the brain organizes information, it, maybe, would have been better to state that and then explain up to the current evidence in neuroscience, as simple [and as balanced] as possible for the general audience. how the brain organizes information While an expert [who needs federal funding for research] could be cautious in discussing the memory of an incumbent [who might win reelection] in the press, it would have been better, so to speak, to find a way to strike a balance, because even if the present was scary, the future, being unknown, should have been a cautionary tale. It would have been better to stay within neuroscience in explanations—discussing neurons and their signals. Then, stating that these signals are also involved in all functions, including wisdom, judgment, emotions, feelings, intelligence, [regulation or control of] digestion, muscles, and so forth. All functions are within the mechanisms of electrical and chemical signals—conceptually and—up to the current evidence in neuroscience. A simple way to put it is that electrical signals have intensity, and chemical signals have volume. A reason, conceptually, that digestion regulation for digestion capacity or efficiency might wane with age could also be due to electrical signals’ lower intensity or chemical signals’ lower volume—as a rough example. that electrical signals have intensity, and chemical signals have volume This means that it could be possible that memory too, might be affected with age. While life experiences may inform judgment and wisdom, effectiveness may not be as rounded or regular, per se, with senescence. Offering explanations that are nearer to how the brain works, while using [non-cranium components or] labels cautiously, as well as from not just a few experts, but say from some major associations for neuroscience, neurology or others, could have ensured the possibility to be careful about directly correlating advanced age with memory decline or offering why that might be possible, without muddling memory [a label] with [other labels like] wisdom, judgment, knowledge and impulse control—charging instead, at philosophy. No one knows if grievance, in part, may be responsible for some of the cataclysmic airstrikes—currently—on neuroscience, but it may have been a lost opportunity to have the field, appear to describe aging and memory vaguely, without actually discussing the brain, and then the presidential debate struck, discrediting the subtlety of the experts while making the field of neuroscience [and its research] a collateral damage. There is an analysis [July 7, 2024] in The Washington Post, Debate brings scrutiny of whether aides shielded signs of Biden’s agingstating that, “When special counsel Robert K. Hur concluded in a February report that President Biden should not be prosecuted for mishandling classified documents in part because a jury would view the president as “a sympathetic, well-meaning, elderly man with a poor memory,” Biden and his aides hit back hard. Now, after his halting performance at a debate against Trump brought questions about Biden’s mental sharpness and physical stamina to the forefront, Biden’s senior aides face worries among fellow Democrats and sharp accusations from Republicans that they took steps to conceal the effect the president’s aging has had on his ability to carry out his duties.” The Washington Post Debate brings scrutiny of whether aides shielded signs of Biden’s aging There is a recent [March 19, 2025] report in Axios, Fallout from Trump research cuts expands across academia, stating that, “Trump administration spending cuts and freezes to federal grants are roiling major academic medical research programs, prompting layoffs, and leading administrators to abandon studies and rescind admissions offers to graduate students. The University of Massachusetts Chan Medical School rescinded provisional offers to dozens of biomedical Ph.D. students this month. Johns Hopkins University last week moved to eliminate more than 2,200 staff positions, including 247 in the U.S., following the termination of more than $80 million in USAID funding, primarily to its school of public health and a global health affiliate. Elsewhere, Stanford announced a hiring freeze tied to NIH cuts, Baylor's College of Medicine said it will scale back expansion plans and trim its incoming graduate school class while Emory's public health school warned of "potentially radical adjustments."” Axios Fallout from Trump research cuts expands across academia Feature image source Feature image source source

Did Special Counsel Robert Hur’s Report, Referencing Joe Biden’s Memory, Derail Neuroscience?

Turning AI Into Better Thinkers With Pointer-Based Memory

Authors:
(1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia;
(2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia;
(3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia;
(4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia;
(5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Methods Methods Part 2 Experimental Results Experimental Results Part 2 Related Works, Discussion, & References Appendix A, B, & C Appendix D 3. Experimental Results In §3.1-3.3, our chosen tasks are representative of various types of symbolic reasoning and wellknown benchmarks to measure the symbol-processing capability of ML models. To showcase that these tasks are non-trivial, we report how Chat-GPT failed on our tasks in Appendix D.6. We validate the contribution of our methods in other practical tasks in §3.4. We also explain the choice of competing baselines in Appendix D. 3.1 Algorithmic Reasoning In our first experiment, we study the class of symbol processing problems where an output sequence is generated by a predefined algorithm applied to any input sequence (e.g., copy and sort). The tokens in the sequences are symbols from 0 to 9. The input tokens can be coupled with meta information related to the task such as the priority score in Priority Sort task. During training, the input sequences have length up to L tokens and can grow to L + 1, 2(L + 1), 4(L + 1) or 8(L + 1) during testing. Our setting is more challenging than previous generalization tests on algorithmic reasoning because of four reasons: (1) the task is 10-class classification, harder than binary prediction in Graves et al. [2014], Le and Venkatesh [2022], (2) the testing data can be eight time longer than the training and the training length is limited to L ≈ 10, which is harder than Grefenstette et al. [2015], (3) there is no curriculum learning as in Kurach et al. [2015], and (4) the training label is the one-hot value of the token, which can be confusing in case one token appears multiple times in the input sequence and tougher than using label as the index/location of the token as in Vinyals et al. [2015]. Baselines are categorized into 4 groups: (1) Traditional RNNs such as LSTM [Hochreiter and Schmidhuber, 1997], (2) Sequential attention models: Content Attention [Bahdanau et al., 2014], Location Attention [Luong et al., 2015], Hybrid Attention (our baseline concatenates the attention vectors from content and location attention), (3) MANNs such as NTM [Graves et al., 2014], DNC [Graves et al., 2016], Neural Stack [Grefenstette et al., 2015] and Transformer [Vaswani et al., 2017], and (4) pointer-aware models: NRAM [Kurach et al., 2015], PtrNet [Vinyals et al., 2015], ESBN [Webb et al., 2020] and our method PANM. In this synthetic experiment, we adopt LSTM as the encoder for PANM. All baselines are trained with fixed number of steps (100K for ID Sort and 50K for the rest), which is enough for the training loss to converge. For each task, each baseline is trained 5 times with different random seeds and we use the best checkpoint on L + 1 mode validation to evaluate the baselines. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Authors: Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Abstract & Introduction Methods Methods Methods Part 2 Methods Part 2 Experimental Results Experimental Results Experimental Results Part 2 Experimental Results Part 2 Related Works, Discussion, & References Related Works, Discussion, & References Appendix A, B, & C Appendix A, B, & C Appendix D Appendix D 3. Experimental Results In §3.1-3.3, our chosen tasks are representative of various types of symbolic reasoning and wellknown benchmarks to measure the symbol-processing capability of ML models. To showcase that these tasks are non-trivial, we report how Chat-GPT failed on our tasks in Appendix D.6. We validate the contribution of our methods in other practical tasks in §3.4. We also explain the choice of competing baselines in Appendix D. 3.1 Algorithmic Reasoning In our first experiment, we study the class of symbol processing problems where an output sequence is generated by a predefined algorithm applied to any input sequence (e.g., copy and sort). The tokens in the sequences are symbols from 0 to 9. The input tokens can be coupled with meta information related to the task such as the priority score in Priority Sort task. During training, the input sequences have length up to L tokens and can grow to L + 1, 2(L + 1), 4(L + 1) or 8(L + 1) during testing. Our setting is more challenging than previous generalization tests on algorithmic reasoning because of four reasons: (1) the task is 10-class classification, harder than binary prediction in Graves et al. [2014], Le and Venkatesh [2022], (2) the testing data can be eight time longer than the training and the training length is limited to L ≈ 10, which is harder than Grefenstette et al. [2015], (3) there is no curriculum learning as in Kurach et al. [2015], and (4) the training label is the one-hot value of the token, which can be confusing in case one token appears multiple times in the input sequence and tougher than using label as the index/location of the token as in Vinyals et al. [2015]. Baselines are categorized into 4 groups: (1) Traditional RNNs such as LSTM [Hochreiter and Schmidhuber, 1997], (2) Sequential attention models: Content Attention [Bahdanau et al., 2014], Location Attention [Luong et al., 2015], Hybrid Attention (our baseline concatenates the attention vectors from content and location attention), (3) MANNs such as NTM [Graves et al., 2014], DNC [Graves et al., 2016], Neural Stack [Grefenstette et al., 2015] and Transformer [Vaswani et al., 2017], and (4) pointer-aware models: NRAM [Kurach et al., 2015], PtrNet [Vinyals et al., 2015], ESBN [Webb et al., 2020] and our method PANM. In this synthetic experiment, we adopt LSTM as the encoder for PANM. All baselines are trained with fixed number of steps (100K for ID Sort and 50K for the rest), which is enough for the training loss to converge. For each task, each baseline is trained 5 times with different random seeds and we use the best checkpoint on L + 1 mode validation to evaluate the baselines. Baselines This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

This Model Teaches AI to Handle Long-Term Memory Like a Computer

A Memory Hack That Makes AI Smarter With Sequences

Authors:
(1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia;
(2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia;
(3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia;
(4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia;
(5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Methods Methods Part 2 Experimental Results Experimental Results Part 2 Related Works, Discussion, & References Appendix A, B, & C Appendix D Abstract We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data. PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities. PANM facilitates pointer assignment, dereference, and arithmetic by explicitly using physical pointers to access memory content. Remarkably, it can learn to perform these operations through end-to-end training on sequence data, powering various sequential models. Our experiments demonstrate PANM’s exceptional length extrapolating capabilities and improved performance in tasks that require symbol processing, such as algorithmic reasoning and Dyck language recognition. PANM helps Transformer achieve up to 100% generalization accuracy in compositional learning tasks and significantly better results in mathematical reasoning, question answering and machine translation tasks. 1. Introduction Systematic generalization underpins intelligence, and it relies on the ability to recognize abstract rules, extrapolating them to novel contexts that are distinct yet semantically similar to the seen data. Current neural networks or statistical machine learning fall short of handling novel data generated by symbolic rules even though they have achieved state-of-the-art results in various domains. Some approaches can show decent generalization for single or set input data [Bahdanau et al., 2018, Gao et al., 2020, Webb et al., 2020]. Yet, neural networks in general still fail in sequential symbol processing tasks, even with slight novelty during inference [Lake and Baroni, 2018, Del´etang et al., 2022]. For instance, these models can easily learn to duplicate sequences of 10 items, but they will fail to copy sequences of 20 items if they were not part of the training data. These models overfit the training data and perform poorly on out-of-distribution samples such as sequences of greater length or sequences with novel compositions. The issue also affects big models like Large Language Models, making them struggle with symbolic manipulation tasks [Qian et al., 2023]. This indicates that current methods lack a principled mechanism for systematic generalization. From a neuroscience perspective, it has been suggested that the brain can execute symbol processing through variable binding and neural pointers, wherein the sensory data are conceptualized into symbols that can be assigned arbitrary values [Kriete et al., 2013]. Like the brain, computer programs excel at symbolic computations. Programmers use address pointers to dynamically access data or programs, and have flexible control over the variable. Their programs can work appropriately with unseen inputs. Building on these insights, we propose a pointer-based mechanism to enhance generalization to unseen length in sequence prediction, which is a crucial problem that unifies all computable problems [Solomonoff, 2010]. Our mechanism is based on two principles: (I) explicitly modeling pointers as physical addresses, and (II) strictly isolating pointer manipulation from input data. As such, we need to design a memory that supports physical pointers, and create a model that manipulates the pointers to perform abstract rules and access to the memory. Our memory, dubbed Pointer-Augmented Neural Memory (PANM), is slot-based RAM [Von Neumann, 1993] where each memory slot consists of two components: data and address. Unlike initial endeavors that implicitly model pointers as attention softmax [Vinyals et al., 2015, Kurach et al., 2015, Le et al., 2018, Khan et al., 2021], our addresses are generated to explicitly simulate physical memory addresses, i.e., incremental binary numbers, which is critical for generalization to longer sequences. To manipulate a pointer, we create an address bank that contains physical addresses corresponding to the input sequence, and use a neural network called Pointer Unit that is responsible for transforming pointers from an initial address in the address bank. Through attention to the address bank, a new pointer is generated as a mixture of the physical addresses, which can point to different memory slots to follow the logic of the task. We aim to let the Pointer Unit learn the symbolic rules of the task in an end-to-end manner. Finally, given a (manipulated) pointer, the model can access the data through 2 modes of pointer-based access: pointer dereference (Mode-1) and relational access (Mode-2). Our memory can be plugged into common encoder-decoder backbones such as LSTM or Transformer. Our contribution is a novel memory architecture that incorporates explicit pointer and symbol processing, working seamlessly with sequential models to generalize better. We examine our model in symbol-processing domains such as algorithms and context-free grammar where PANM effectively works with LSTM and StackRNN. We apply PANM to improve the generalization of Transformer models on compositional learning, using SCAN and mathematics datasets. Also, we observe PANM’s superior performance in more realistic question answering and machine translation tasks. Our focus is not on striving for state-of-the-art results requiring specialized designs tailored to specific tasks. Our objective is to highlight the generalization improvement achieved by integrating our memory module into fundamental sequential models, with minimal architectural changes, and showcase the importance of using fundamental generalizing principles to address limitations of current deep learning. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Authors: Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Abstract & Introduction Methods Methods Methods Part 2 Methods Part 2 Experimental Results Experimental Results Experimental Results Part 2 Experimental Results Part 2 Related Works, Discussion, & References Related Works, Discussion, & References Appendix A, B, & C Appendix A, B, & C Appendix D Appendix D Abstract We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data. PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities. PANM facilitates pointer assignment, dereference, and arithmetic by explicitly using physical pointers to access memory content. Remarkably, it can learn to perform these operations through end-to-end training on sequence data, powering various sequential models. Our experiments demonstrate PANM’s exceptional length extrapolating capabilities and improved performance in tasks that require symbol processing, such as algorithmic reasoning and Dyck language recognition. PANM helps Transformer achieve up to 100% generalization accuracy in compositional learning tasks and significantly better results in mathematical reasoning, question answering and machine translation tasks. 1. Introduction Systematic generalization underpins intelligence, and it relies on the ability to recognize abstract rules, extrapolating them to novel contexts that are distinct yet semantically similar to the seen data. Current neural networks or statistical machine learning fall short of handling novel data generated by symbolic rules even though they have achieved state-of-the-art results in various domains. Some approaches can show decent generalization for single or set input data [Bahdanau et al., 2018, Gao et al., 2020, Webb et al., 2020]. Yet, neural networks in general still fail in sequential symbol processing tasks, even with slight novelty during inference [Lake and Baroni, 2018, Del´etang et al., 2022]. For instance, these models can easily learn to duplicate sequences of 10 items, but they will fail to copy sequences of 20 items if they were not part of the training data. These models overfit the training data and perform poorly on out-of-distribution samples such as sequences of greater length or sequences with novel compositions. The issue also affects big models like Large Language Models, making them struggle with symbolic manipulation tasks [Qian et al., 2023]. This indicates that current methods lack a principled mechanism for systematic generalization. This indicates that current methods lack a principled mechanism for systematic generalization. From a neuroscience perspective, it has been suggested that the brain can execute symbol processing through variable binding and neural pointers, wherein the sensory data are conceptualized into symbols that can be assigned arbitrary values [Kriete et al., 2013]. Like the brain, computer programs excel at symbolic computations. Programmers use address pointers to dynamically access data or programs, and have flexible control over the variable. Their programs can work appropriately with unseen inputs. Building on these insights, we propose a pointer-based mechanism to enhance generalization to unseen length in sequence prediction, which is a crucial problem that unifies all computable problems [Solomonoff, 2010]. Our mechanism is based on two principles: (I) explicitly modeling pointers as physical addresses, and (II) strictly isolating pointer manipulation from input data. As such, we need (I) (II) to design a memory that supports physical pointers, and create a model that manipulates the pointers to perform abstract rules and access to the memory. Our memory, dubbed Pointer-Augmented Neural Memory (PANM), is slot-based RAM [Von Neumann, 1993] where each memory slot consists of two components: data and address. Unlike initial endeavors that implicitly model pointers as attention softmax [Vinyals et al., 2015, Kurach et al., 2015, Le et al., 2018, Khan et al., 2021], our addresses are generated to explicitly simulate physical memory addresses, i.e., incremental binary numbers, which is critical for generalization to longer sequences. To manipulate a pointer, we create an address bank that contains physical addresses corresponding to the input sequence, and use a neural network called Pointer Unit that is responsible for transforming pointers from an initial address in the address bank. Through attention to the address bank, a new pointer is generated as a mixture of the physical addresses, which can point to different memory slots to follow the logic of the task. We aim to let the Pointer Unit learn the symbolic rules of the task in an end-to-end manner. Finally, given a (manipulated) pointer, the model can access the data through 2 modes of pointer-based access: pointer dereference (Mode-1) and relational access (Mode-2). Our memory can be plugged into common encoder-decoder backbones such as LSTM or Transformer. Our contribution is a novel memory architecture that incorporates explicit pointer and symbol processing, working seamlessly with sequential models to generalize better. We examine our model in symbol-processing domains such as algorithms and context-free grammar where PANM effectively works with LSTM and StackRNN. We apply PANM to improve the generalization of Transformer models on compositional learning, using SCAN and mathematics datasets. Also, we observe PANM’s superior performance in more realistic question answering and machine translation tasks. Our focus is not on striving for state-of-the-art results requiring specialized designs tailored to specific tasks. Our objective is to highlight the generalization improvement achieved by integrating our memory module into fundamental sequential models, with minimal architectural changes, and showcase the importance of using fundamental generalizing principles to address limitations of current deep learning. showcase the importance of using fundamental generalizing principles to address limitations of current deep learning. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

A New Neural Memory Trick Helps AI Handle Much Longer Sequences

Authors:
(1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia;
(2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia;
(3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia;
(4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia;
(5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Methods Methods Part 2 Experimental Results Experimental Results Part 2 Related Works, Discussion, & References Appendix A, B, & C Appendix D A. More Discussion on Related Works The proposed address attention in our paper is comparable to two known mechanisms: (1) location based attention [Luong et al., 2015, Dubois et al., 2020] and (2) memory shifting [Graves et al., 2014, Yang, 2016]. The former uses neural networks to produce attention weights to the memory/sequence, which cannot help when the memory grows during inference since the networks never learn to generate weights for the additional slots. Inspired by Turing Machine, the latter aims to shift the current attention weight associated with a memory slot to the next or previous slot. Shifting-like operations can handle any sequence length. However, it cannot simulate complicated manipulation rules. Unlike our PU design which obeys § 1’s principle II, the attention weight and the network trained to shift it depend on the memory content M. That is detrimental to generalization since new content can disturb both the attention weight and the shifting network as the memory grows. Another line of works tackles systematic generalization through meta-learning training [Lake, 2019], while our method employs standard supervised training. These approaches are complementary, with our method concentrating on enhancing model architecture rather than training procedures, making it applicable in diverse settings beyond SCAN tasks. Additionally, the study by Hu et al. (2020) addresses syntactic generalization [Hu et al., 2020], a different problem compared to our paper, which emphasizes length extrapolation across various benchmarks. Notably, our paper considers similar baselines, such as LSTM and Transformer, as those examined in the referenced papers. There are other lines of research targeting reasoning and generalization using image input[Wu et al., 2020, Eisermann et al., 2021]. They are outside the scope of our paper, which specifically addresses generalization for longer sequences of text or discrete inputs. Our address bank and physical pointers can be viewed as some form of positional encoding. However, we do not use simple projections or embeddings to force the attention to be position-only. Instead, we aim to learn a series of transformations that simulate the position-based symbolic rules. At each time step, a new pointer ("position") is dynamically generated that reflects the manipulation rule required by the task (e.g. move to the next location), which is unlike the positional encoding approaches such as RPE [Dai et al., 2019] which aims to provide the model with information on the relative position or distance of the timesteps. We summarise the difference between our method and Transformer in Table 5. B. More Discussion on Base Address Sampling Mechanism We provide a simple example to illustrate how base address sampling help in generalization. Assume the training sequence length is 10, and the desired manipulation is p ′ = p + 1 (copy task). Assume the possible address range is 0, 1, ..., 19, which is bigger than any sequence length. If pB = 0 , the training address bank contains addresses: 0, 1, ...8, 9. Without base address sampling, the model always sees the training address bank of 0, 1, ...8, 9 and thus can only learn manipulating function for 0 ≤ p ≤ 9, thereby failing when testing address bank includes addresses larger than 9. C. More Discussion on Model and Architecture This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Authors: Authors: (1) Hung Le, Applied AI Institute, Deakin University, Geelong, Australia; (2) Dung Nguyen, Applied AI Institute, Deakin University, Geelong, Australia; (3) Kien Do, Applied AI Institute, Deakin University, Geelong, Australia; (4) Svetha Venkatesh, Applied AI Institute, Deakin University, Geelong, Australia; (5) Truyen Tran, Applied AI Institute, Deakin University, Geelong, Australia. Table of Links Abstract & Introduction Abstract & Introduction Methods Methods Methods Part 2 Methods Part 2 Experimental Results Experimental Results Experimental Results Part 2 Experimental Results Part 2 Related Works, Discussion, & References Related Works, Discussion, & References Appendix A, B, & C Appendix A, B, & C Appendix D Appendix D A. More Discussion on Related Works The proposed address attention in our paper is comparable to two known mechanisms: (1) location based attention [Luong et al., 2015, Dubois et al., 2020] and (2) memory shifting [Graves et al., 2014, Yang, 2016]. The former uses neural networks to produce attention weights to the memory/sequence, which cannot help when the memory grows during inference since the networks never learn to generate weights for the additional slots. Inspired by Turing Machine, the latter aims to shift the current attention weight associated with a memory slot to the next or previous slot. Shifting-like operations can handle any sequence length. However, it cannot simulate complicated manipulation rules. Unlike our PU design which obeys § 1’s principle II, the attention weight and the network trained to shift it depend on the memory content M. That is detrimental to generalization since new content can disturb both the attention weight and the shifting network as the memory grows. Another line of works tackles systematic generalization through meta-learning training [Lake, 2019], while our method employs standard supervised training. These approaches are complementary, with our method concentrating on enhancing model architecture rather than training procedures, making it applicable in diverse settings beyond SCAN tasks. Additionally, the study by Hu et al. (2020) addresses syntactic generalization [Hu et al., 2020], a different problem compared to our paper, which emphasizes length extrapolation across various benchmarks. Notably, our paper considers similar baselines, such as LSTM and Transformer, as those examined in the referenced papers. There are other lines of research targeting reasoning and generalization using image input[Wu et al., 2020, Eisermann et al., 2021]. They are outside the scope of our paper, which specifically addresses generalization for longer sequences of text or discrete inputs. Our address bank and physical pointers can be viewed as some form of positional encoding. However, we do not use simple projections or embeddings to force the attention to be position-only. Instead, we aim to learn a series of transformations that simulate the position-based symbolic rules. At each time step, a new pointer ("position") is dynamically generated that reflects the manipulation rule required by the task (e.g. move to the next location), which is unlike the positional encoding approaches such as RPE [Dai et al., 2019] which aims to provide the model with information on the relative position or distance of the timesteps. We summarise the difference between our method and Transformer in Table 5. B. More Discussion on Base Address Sampling Mechanism We provide a simple example to illustrate how base address sampling help in generalization. Assume the training sequence length is 10, and the desired manipulation is p ′ = p + 1 (copy task). Assume the possible address range is 0, 1, ..., 19, which is bigger than any sequence length. If pB = 0 , the training address bank contains addresses: 0, 1, ...8, 9. Without base address sampling, the model always sees the training address bank of 0, 1, ...8, 9 and thus can only learn manipulating function for 0 ≤ p ≤ 9, thereby failing when testing address bank includes addresses larger than 9. C. More Discussion on Model and Architecture This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about MEZMO

Categories

Trending Topics

Stories, News & Mentions about MEZMO

Mezmo’s Stories on HackerNoon

Categories

Trending Topics