Amilo

Do You Know the Three Amigos? Don’t worry. This doesn’t mean we have to write our stories in Spanish! Instead, we’ll explore one of the agile ways of working, the Three Amigos. We’ll discover what the Three Amigos is with an example of how a session can help a team better understand requirements. What Are the Three Amigos? The “three amigos,” translated to “three friends” in English, is an agile way of working where three individuals with different perspectives come together to refine user stories. These three perspectives, as described by , consist of: Agile Alliance Business — What problem are we trying to solve? Development — How might we build a solution to solve that problem? Testing — What about this? What could happen? Therefore, when refining a user story, a developer, tester, and business analyst (BA) should be present to understand the requirements sufficiently. How Do the Three Amigos Work? Let’s go through an example of a team discussing a user story in a three amigos session. The team sits down together with the following user story to refine: As an employee, I want to submit my timesheet so I can get paid Developer: So, what do we need to capture on the timesheet? BA: We need to capture the number of hours worked each day in a calendar month. Developer: No problem at all. We can create a form where users can enter their work hours daily. Then, when the user hits submit, can we send an email to HR to process the timesheet. BA: Unfortunately, an email isn’t going to cut it. We need to send this information to our HR system as we have an automated process for payroll. Developer: OK, in that case, we can send it to the HR system via the HR API. Tester: What happens if they enter incorrect information? Developer: We can provide feedback to the user. For example, we can show a validation message when a user provides invalid data. Tester: What is invalid data? Can a user enter a negative number for hours worked? Is there a maximum number of hours a user can work on a given day? BA: No, a user should only be able to enter positive numbers up to twelve. Tester: OK, we should write a test case to cover this. Developer: I’ll also ensure we have the appropriate validation for the number of hours. BA: Also, I just remembered we need to send an email notification to the line manager for approval. Developer: So, we need to make sure we submit the timesheet in a “review” state, so the employee doesn’t get paid until their line manager approves it? BA: Correct. Tester: What if the line manager is absent? BA: That’s a good question. We should also email line manager B. Tester: What if the line manager wants to reject the submitted timesheet? Developer: This is turning out to be a lot of development work. May I suggest we create new user stories for these additional scenarios? The team agrees the user story is now quite large, so they split out the email notification work into separate user stories for later refinement: As a line manager, I want to get notified if one of my staff submits a timesheet so I can review it As a line manager, I want to be able to approve my staff timesheets so I can ensure my staff get paid As a line manager, I want to be able to reject a timesheet so I can ensure my staff aren't paid incorrectly Developer: Is there anything else we need to consider for capturing the timesheet details? BA: I think that’s everything covered. We can iterate on this in the future if we need to. The team moves on to the next story. What Did the Team Discover From This Session? For this single user story, the team discovered the following: Firstly, the number of hours worked for each day of the month must be captured. Second, the number of hours worked must be positive, up to twelve. Third, the form must be validated and feedback provided to the user. Finally, when the user submits their timesheet, the system must send the data to the HR API in a state of review. On top of that, the team has created three new user stories. They are now better prepared to estimate and take on this work. Best Practices There’s no right or wrong way to refine user stories using three amigos. Instead, I’d suggest trying it out to find what works best for your team. However, here are a few good practices that can help: Timebox sessions — Timeboxing helps focus the team; you can always return to a story later if you don’t finish refining it. 45 minutes to an hour is generally a good rule of thumb. Rotate team members each session — this helps share knowledge amongst the team. Have one member from each area — different perspectives provide additional value. Don’t rush — it takes as long as it takes. Refining stories upfront means we’re not asking as many questions when we’re in development. Takeaways From the example above, we can see the value of the Three Amigos. The developer helped to define the solution to the problem. The tester provided value by highlighting potential issues. The BA provided information right there and then, so the team didn’t have to go away to find the answers they desperately needed. The team is now more confident in taking on the story in their next sprint! First published here

Three Amigos Teach Us a Lot About Writing Better Agile User Stories 

Listen to the interview on or watch on . iTunes YouTube In this episode and interview , President at NEM.io Foundation and NEM’s CTO . Linh Dao Smooke Utsav Jaiswal Alex Tinsman Nate D’Amico They discuss cryptocurrency adoption, difference between NEM, a peer-to-peer cryptocurrency and blockchain platform, and Nem.IO Foundation, gender role in crypto tech spheres, blockchain decentralization, Nem’s operation mindset and the future of NEM. “What's pretty interesting about the trajectory with the path for the NEM is that it was so forward thinking with the security features that it has always overcome and remained when all these other coins on platforms kind of went away. I think it has a bright future with technology. Platforms like NEM stick around because they are forward thinking with their features and their abilities.” “I want to be a part of change and the only way to do so is to educate yourself. And the only way to educate yourself is to get dirty. To get dirty with the code, get dirty with the channels, understand. Get into the thick of it. And then it gets really exciting [blockchain and cryptocurrency) because it opens up the opportunity for anyone. Anyone, no matter what your skills set is, everyone has the same level of playing field.  So I don't really think of it as a female thing, I think of it as an empowering tool that anyone and everyone can be a part of. So to me it's not gender, it's a movement.” - Alex Tinsman “If someone on even local, city council level wanted to do a similar voting style that we are using for the NEM ecosystem, they could launch a private or public chain and it doesn't really matter as functionally of the network would be doing the same thing. They could launch a private chain or public chain and still make it auditable, so anyone could have a verifiable data that they would check and download. And then they could configure that network by just tuning a couple labs when they launch it to decide how the consensus works. So if you’re launching a six server node of private network for local city council loading and you’re gonna use it once and then archive it for historical verification purposes, and then every year stand up a new network - you could configure it in a very simplified way that gives you total control on how to protect the network.” - Nate D'Amico P.S. If you dig the new , consider Hacker Noon Podcast giving us a 5 star review on iTunes. Also check out . today’s homepage

Challenges and Solutions with The NEM Foundation's President Alex Tinsman and NEM CTO Nate D’Amico

The past two weeks have had me very very confused. Every Tuesday and Thursday I sit in FNCE 101 (intermediate macroeconomics) attempting to understand how interest rates affect the GDP, why taxes may not shift consumption, and why Bernanke says it’s great to run a current account deficit. The trouble is I’m a skeptic, and for the past couple of weeks I haven’t believed half of what I am being taught. What I haven’t been able to grasp so far is why GDP even matters, how it’s okay for the US to run what looks like a ponzi scheme, and why productivity seems to have such a contrived definition. Those last three points will be included in future posts as I begin this series of reflections on economics, and share my opinions as to what really matters. If you have at all felt confused about what macroeconomic mumbo jumbo means, I invite you to read along; and if you aren’t confused, I still hope you’ll read this article and then properly explain economics to me. To begin, I want to talk about money because that is where the root of all my problems started. Everything is always priced in terms of dollars; it is the unit of economics. When we discuss the economists’ universal fetish (EUF), otherwise known as gross domestic product (GDP), we speak about it in terms of dollars, real or nominal. The fact that real and nominal are not the same, should key people off to the fact that something spooky is going on. GDP is meant to convey economic growth and is a measure of the production of goods and services. Goods and services are real tangible things that have utility i.e. they help increase people’s satisfaction. When we discuss them in the aggregate we use money because that is how we compare their values. The thing is other than the fact that we use it as an . The term fiat currency is especially apt because dollars, pounds, yen, etc. are literally fiats, that is they are arbitrary commands coming from an authoritative body that said, “Thou must pay me thine taxes in dollars,” (complete anachronism). Since taxes had to be paid in fiat, and the government expected people to accept that fiat when it wanted goods and services, the currency became the new means of exchange. money has no intrinsic value intermediate As an aside, to all those who declare that money and currency are different because money (i.e. gold and silver) is a store of value while currency is not, enjoy paying for your tea parties and first edition Atlas Shrugged hardcovers with gold bullion. For the most part, gold and silver have historically been no different than greenbacks in that their only value was being pretty to look at and difficult to dig out of the ground. You couldn’t eat them or make anything particularly useful out of them (or at least create nothing more useful than could be made out of another material). In reality, they had very little utility, and so I would argue very little intrinsic value. Going even further, I believe that the dollar is not a value itself, but an equals sign. What I mean by this is that a currency functions as a statement of equality which say for example $1 equals 4 gum balls equals 6 large eggs equals 2.04 seconds of Amy Gutmann’s “labor” (in 2015 dollars assuming a 2000 hour workweek). When we speak about a GDP we are discussing all of the goods and services that are produced in a country in a year. That is, we mean all of the stuff that we do for each other, or give to each other in a year. We can call it $18 trillion or 42 quadrillion AmitCoins©, but the point of the number is to say that I made more stuff than you did, or I made more stuff this year than I did last year. The issue with this is that it is loosely a measure of how much you made (for real GDP) but there is no measure of quality, and more than that, all we are really measuring is the utilization of visible dollars as a medium of exchange. Someone is going to tell me I am wrong, but I posit that in theory the value of a unit of currency is loosely equal to the trade conducted through that currency /circulating supply of the currency. The purpose of a currency is to mediate transactions of goods and services, and each unit of currency is an entitlement to a good or service. If nobody is using the currency for transactions, or the amount of goods and services being exchanged (i.e. economic activity) is decreasing, then the value of the currency decreases. If more notes of the currency exist, the value of the currency decreases because each unit it is now a smaller fraction of the overall amount of money in action. However, if some of that money is not being used at all, then it does not necessarily cause an appreciable dip in value because it is effectively not part of the money supply since it cannot be transacted with. The value of money is the value of trade conducted with that money. So after all that buildup, here comes , i.e. a digital currency. First two comments: 1. the is stupid, 2. Jamie Dimon is an idiot (sort of). Bitcoin is really not that different from a dollar or a pound. It is a way for people to conduct transactions, and its value stems from how much transacting people want to do with its circulating supply. Since, like a dollar, it has no intrinsic value, i.e. I can’t farm it for food, utilize it it to build something, or use it to enable a service (sort of), Bitcoin is not, and should not be considered property; it’s money. Bitcoin derives value from the fact that people like to use it for making exchanges. In the beginning I believe there were five main segments using it: Bitcoin IRS considering Bitcoin property Techno geeks: These evangelists saw an technological breakthrough that was unlike anything they had seen before, and were inspired by this new system. Decentralists (Anarchists?): These people railed against the idea of the Fed running the world, and wanted something pure of central bank taint. Decentralists generally believe dispersing data across a , versus having a central node, inherently makes a system more secure. Plus, they generally mistrust any institution with the power of maintaining a network by itself. network Cool Kids: These followers didn’t understand how blockchains work whatsoever, but wanted to be part of the in crowd and look like trendsetters. Smugglers: These people saw a great way to buy drugs without the government monitoring them. Free marketers (Tax Evaders): These people wanted to avoid taxes, and send money internationally without restriction. The issue today is that many of those initial groups (outside of the smugglers) were not very actively exchanging the bitcoins for goods and services, or at least exchanging it for another fiat to represent goods and services. Added to that mess entered all of the investors (i.e. speculators) who saw bitcoins as high-alpha penny stocks, and miners who thought they found the next gold rush. Where Dimon may have a point is that Bitcoin is not actually being used to trade very much. People are treating this as if it’s a stock, but a stock is a share of a company that has earnings and will eventually pay those earnings out to you. Bitcoin is never going to pay anyone anything, it’s a string of numbers that says that let’s people trade with each other. Bitcoin’s integral strength lies in the fact that it makes trade really easy, but if people do not start using it to pay for things, it will continue to be volatile and the price eventually will drop. Right now Bitcoin is propped up by wishfulness mixed with its de facto use as the crypto dollar or reserve currency against which most other crypto is priced. Once it is used as a proper means of exchange, the price will stabilize because nobody wants to hear that today gasoline is .001 BTC and tomorrow it is .002 BTC. Business becomes impossible to conduct when price swings create that much uncertainty and instability, so the currency stops trading. The only other option is that, like gold, it becomes a hedge against the dollar, but given its volatility, due to a lack of use as a medium for value exchange, I do not see that happening. If the general public, however, sees that Bitcoin is a great medium for exchange, and stops fearing digital currencies, which may eventually replace paper money, the price of Bitcoin will soar due to its limited supply. Unlike Bitcoin, Ethereum actually does have some intrinsic value since many things can be made from it. In some sense, Ethereum is essentially a commodity like steel because different smart contracts can be built with it. Thus unlike Bitcoin, Ethereum ideally sources its value from the demand in all of the applications being built on top of it. Computer processing of Ethereum-based smart contracts, any transactions that occur on the Ethereum network, have gas prices that are paid in Ethereum, so as long as people want to build and use ERC20 tokens Ethereum will have a commodity value. Ethereum holds a fiat-style monopoly on transactions within its network which is why its value is tied strongly to the value of the overall network, including the sume of ERC20 tokens. The issue is that Ethereum too is over speculated (by people like me), and is oftentimes being treated as a security, which it isn’t. The potential downfall for Ethereum, is that it is being used to create hundreds of illegal securities through ICOs as founders decide to avoid SEC regulations and institutional investors by going straight to the gamblers trying to get rich quick (also me). If good products and companies are built on Ethereum, its value will soar, but if ICOs devolve into scams and governments decide to ban them completely, Ethereum is doomed. Let me know if you (dis)agree.

Amitomics Part 1: Is Bitcoin Money?

2024’s here, and DIY LLM toys? Totally a thing now. No tech wizardry needed, just your curiosity. I took the plunge, mixing a bit of coding with heaps of fun, and bam — created my own talking toy. If you’re up for crafting an AI buddy with ease, you’re in the right spot. Let’s demystify tech together and bring your AI friend to life. In a world where technology increasingly intersects with daily life, creating your own LLM toy not only demystifies AI but also provides a personalized gateway to the wonders of interactive technology. Let’s take a look at the final effect first. https://www.youtube.com/embed/FKEYL6VgHo4?embedable=true Honestly, it’s pretty awesome. Ready to start? Let’s dive in! How Does It Work? There are three key steps: Recording: Receive real-time recording data sent by the toy through UDP, and call the STT (Sound-To-Text) API to convert the sound into text. Thinking: After receiving the previous text, the LLM (Large-Language-Model) API will be immediately called to obtain sentences generated by the LLM in a streaming manner. Then, the TTS (Text-To-Sound) API is called to convert the sentences into human speech. Play audio: Toys will receive TTS (Text-To-Sound) audio file streams generated by the FoloToy Server and play them according to the order. Preparation Before Development Before you start making your LLM toy, it is crucial to understand the necessary hardware, software, and technical knowledge. This section will guide you in preparing all the essentials to ensure a smooth start. Hardware The ChatGPT AI Voice Conversation Core Board serves as the brain of your project, enabling voice interactions with AI. Folotoy Core: Essentials like a microphone, speaker, buttons, switches, and power supply are necessary. I’m going with the for its ready-to-use setup. Toy Components: Alilo Honey Bunny G6 (any other choice): Ideal for those looking to retrofit existing toys with AI capabilities. Octopus Dev Suit Server Utilizing your own machine, like a MacBook Pro, ensures that your toy has a reliable backend to process and respond to voice interactions. Alternatively, cloud services like Google Cloud Engine (GCE) can scale your project for broader applications. Prerequisite Knowledge (required): Understanding Docker is crucial for deploying software in containers, making your project portable and scalable. I use it to start the Folo server. Docker (required): Version control is essential for managing your project’s codebase. I use it for managing the Folo server codebase. Git (optional): If you’re aiming for advanced customization, familiarity with MQTT (a lightweight messaging protocol) will be beneficial for communicating between the toy and the server. MQTT (optional): Open-source MQTT broker for IoT, IIoT, and connected vehicles. Used for managing your toys. EMQX (optional): If you want to run large language models locally, it’s a good choice. ollama Services & Registration To make your toy come to life, you’ll need access to specific AI services. For this project, I’ve chosen to utilize OpenAI’s offerings: TTS (Text-to-Speech): OpenAI’s Whisper service converts text responses from the AI into speech, making the interaction natural. LLM (Large Language Model): Utilizing OpenAI’s models for understanding and generating human-like text responses. STT (Speech-to-Text): OpenAI’s TTS service transcribes spoken words into text, allowing the AI to understand voice commands or queries. All you need do is register on the OpenAI platform and , like sk-…i7TL. create a key Assemble Your Toy Now it’s time to put all the pieces together and make your own LLM toy. https://www.youtube.com/embed/hR7V1izzeZw?si=T3cY8y4Oy2Bxkg0H&embedable=true The general steps are as follows, it is recommended to watch the above tutorial first. Check if the recording and playback of Alilo G6 are normal. Use a screwdriver to unscrew the 6 screws on the back of Alilo G6. Carefully open the casing of Alilo G6, unplug all plugs on the motherboard, and first unplug the power plug, there is glue on the plug, you can use an art knife to gently cut it open, be sure not to cut your hand. Unscrew the 4 screws on the motherboard and remove it. Replace the original motherboard with FoloToy’s motherboard and tighten 3 screws to fix it. Plug in the microphone, light, speaker, and power sockets, finally, plug in the power socket. After plugging in all sockets, do not close the casing or install screws yet. First, turn the on/off switch on the rabbit’s tail to see if the newly installed light can turn on and flash slowly in blue color. If there is no problem, install the casing and tighten the screws to complete the replacement process. Set Up the Server Strong backend support is key to making your LLM toys understand and respond to voice commands. This section will teach you how to clone the server code base, configure the server, and start Docker containers to ensure that your toys have stable backend support. First, clone the Folo server codebase from GitHub. git clone git@github.com:FoloToy/folotoy-server-self-hosting.git Then change the base server configuration in file to your own. docker-compose.yml Name Description Example OPENAI_OPENAI_KEY Your OpenAI API key. sk-...i7TL OPENAI_TTS_KEY Your OpenAI API key. sk-...i7TL OPENAI_WHISPER_KEY Your OpenAI API key. sk-...i7TL AUDIO_DOWNLOAD_URL The URL of the audio file. http://192.168.x.x:8082 SPEECH_UDP_SERVER_HOST The IP address of your server. 192.168.x.x Then configure your roles in the file, here is a minimal example, for full configuration, please refer to the . config/roles.json Folotoy documentation {
  "1": {
    "start_text": "Hello, what can I do for you?",
    "prompt": "You are a helpful assistant."
  }
} Then start the docker containers. docker compose up -d I run the Folo server in my own machine, if you want to run it in the cloud, almost the same. One thing to note is that you need to expose the port 1883, 8082, 8085, 18083 and 8083 to the public network. For more information, please refer to the . Folotoy documentation Chat with Your Toy Once everything is ready, it’s time to interact with your LLM toy. Turn on the switch on the back of the toy to power it on. The blue blinking light in the ears indicates that the toy has entered pairing mode. Turn on your phone or computer, and select the “FoloToy-xxxx” wireless network. After a moment, your phone or computer will automatically open a configuration page where you can set which WiFi network (SSID and password) to connect to, as well as the server address (like 192.168.x.x) and port number (keep the default 1883). After the network is configured and connected to the server, press the big round button in the middle to start the conversation. After you stop speaking, FoloToy will emit a beep to indicate the end of the recording. The 7 round small buttons around are role-switching buttons. After clicking, the role switch takes effect. Debugging Whether it is a server or a toy, you may encounter some technical problems. This section will provide some basic debugging tips and tools to help you diagnose and solve possible problems and ensure that your LLM toys can run smoothly. Server Debugging To check the server logs, run the following command. docker compose logs -f LOG_LEVEL can be set in the file to control the log level. docker-compose.yml Toy Debugging Folo Toy provides an easy way to debug the toy base on a USB serial port. You can use the to debug the toy. Folo Toy Web Tool Connect the toy with your computer using a USB cable. Open the Folo Toy Web Tool, and then click the “Console” button to connect to the toy. Once connected, you should be able to see real-time logs from your device in the console. Also, there is a LED on the toy, it will light up in . different colors to indicate the status of the toy MQTT Debugging Open the to check the MQTT messages. The default username is admin and the password is public. Anyways, change the password to a secure one after you log in. EMQX Dashboard Advanced Customization For advanced users who want to further explore and customize their LLM toys, this section will introduce how to locally run large language models, use tools such as CloudFlare AI Gateway, and customize the voice of characters. This will open up a broader world of DIY LLM toys for you. Run LLM Locally Running large language models locally is an interesting thing. You can run Llama 2, Gemma, and all kinds of open-source large models from around the world, even models trained by yourself. Using ollama, you can do it easily. , and then run the following command to run the Llama 2 model. Install ollama first ollama run llama2 Then, to use the local LLM model. change the role configuration {
  "1": {
    "start_text": "Hello, what can I do for you?",
    "prompt": "You are a helpful assistant.",
    "llm_type": "ollama",
    "llm_config": {
      "api_base": "http://host.docker.internal:11434",
      "model": "llama2"
    }
  }
} The api_base should be the address of your ollama server address, and don’t forget to restart the Folo server to make the changes take effect. docker compose restart folotoy That’s all, change model to Gemma or other models as you like, and enjoy it. Use CloudFlare AI Gateway Cloudflare’s AI Gateway allows you to gain visibility and control over your AI apps. By connecting your apps to AI Gateway, you can gather insights on how people are using your application with analytics and logging and then control how your application scales with features such as caching, rate limiting, as well as request retries, model fallback, and more. First, you need to . create a new AI Gateway Then edit the file to change OPENAI_OPENAI_API_BASE to the address of your AI Gateway, like this: docker-compose.yml services:
  folotoy:
    environment: OPENAI_OPENAI_API_BASE=https://gateway.ai.cloudflare.com/v1/${ACCOUNT_TAG}/${GATEWAY}/openai Then you have a dashboard to see metrics on requests, tokens, caching, errors, and cost. And a logging page to see individual requests, such as the prompt, response, provider, timestamps, and whether the request was successful, cached, or if there was an error. That’s fantastic, isn’t it? Role Voice Customization You can customize the voice of the role by changing the field in the role configuration file. voice_name {
  "1": {
    "tts_type": "openai-tts",
    "tts_config": {
      "voice_name": "alloy"
    }
  }
} Find the voice you like in the . OpenAI TTS Voice List Edge tts has many voices to choose from, enjoy it like this: {
  "1": {
    "tts_type": "edge-tts",
    "tts_config": {
      "voice_name": "en-NG-EzinneNeural"
    }
  }
} Knowledge base support ​ For higher levels of customization, such as knowledge base support. It is recommended to use Dify, which combines the concepts of Backend as Service and LLMOps, covering the core technology stack required to build generative AI native applications, including a built-in RAG engine. With Dify, you can self-deploy capabilities like Assistants API and GPTs based on any model. Let’s focus on the built-in RAG engine, which is a retrieval-based generative model that can be used for tasks such as question and answer, dialogue, and document summarization. Dify includes various RAG capabilities based on full-text indexing or vector database embedding, allowing direct upload of various text formats such as PDF and TXT. Upload your knowledge base, so you don’t have to worry about the toy-making nonsense because you don’t know the background knowledge. Dify can be deployed by itself or use the cloud version directly. The configuration on Folo is also very simple: {
  "1": {
    "llm_type": "dify",
    "llm_config": {
      "api_base": "http://192.168.52.164/v1",
      "key": "app-AAAAAAAAAAAAAAAAAAa"
    }
  }
} Custom toy shape ​ In terms of working principle, any toy can be modified. Folo Toy also offers the Octopus AI Development Kit, which can transform any ordinary toy into an intelligent talking toy. The chip is small and lightweight and can easily fit into any type of toy, whether plush, plastic, or wooden. I DIYed a Shaanxi-speaking cactus. Use your imagination, you can put it into your favorite toys, and it’s not particularly complicated to do it: open the toy Put the Octopus AI development kit into it close the toy The server still uses the same one. You can assign different roles to different toys through sn, which will not be expanded here. You can check the configuration document on the official website. Security Notes Please note, never put the key in a public place, such as GitHub, or it will be abused. If your key is leaked, delete it immediately on the OpenAI platform and generate a new one. You can also use environment variables in and pass them in when starting the container, so as to avoid exposing the key in the code. docker-compose.yml services:
  folotoy:
    environment:
      - OPENAI_OPENAI_KEY=${OPENAI_OPENAI_KEY} OPENAI_OPENAI_KEY=sk-...i7TL docker compose up -d In case you wish to make FoloToy Server publicly available on the Internet, it is strongly recommended to secure the EMQX service and allow access to EMQX only with a password. Learn more about . EMQX Security Conclusion Crafting your own LLM toy is an exciting journey into the world of AI and technology. Whether you’re a DIY enthusiast or a beginner, this guide provides the roadmap to create something truly interactive and personalized. If you encounter challenges acquiring the Folotoy Core or face any issues along the way, joining our Telegram group offers community support and expert advice. For those preferring a ready-made solution, the finished product is available for purchase . This option delivers the same interactive experience without the need for assembly. Folo toys also offer many other products which can be found . here here

DIY LLM Toys: How to Create Your Own AI Buddy with Ease

This paper is available on arxiv under CC 4.0 license. Authors: (1) Md Masud Rana, Department of Mathematics, University of Kentucky; (2) Duc Duy Nguyen, Department of Mathematics, University of Kentucky & ducnguyen@uky.edu. Table of Links Abstract & Introduction Datasets and Results Methods Conclusion, Data and Software Availability, Competing interests, Acknowledgments & References 3 Methods 3.1 Graph Theory and Atom-level Interactions in Biomolecules Graph theory provides a mathematical framework that is widely applied in the study of biomolecules such as proteins, DNA, and RNA. For a biomolecule, a graph G(V, E) is a collection of nodes V and edges E that can represent the connectivity and relationships between different atoms or residues within the molecule. A refinement to this representation is graph coloring, a technique that assigns unique labels to different atom types within the biomolecule. This enriched, colored graph encodes diverse atomic interactions, paving the way for a collective and coarse-grained description of the dataset. In this representation, atoms with assigned labels are organized into subgraphs, and the colored edges between them represent atom-specific interactions. The advantage of using subgraphs lies in their ability to focus on specific regions or components of the biomolecule. By isolating relevant subsets of atoms, subgraphs allow us to identify localized patterns, interactions, or clusters that might not be evident in the global graph representation. This targeted approach provides a more nuanced understanding of the structural and functional properties of biomolecules. To extract atom-level interaction information, we consider specific atom types based on their names in the PDB structure such as carbon alpha (CA), carbon beta (CB), carbon delta-1 (CD1), etc. These atom names serve as identifiers for specific positions within a protein’s three-dimensional structure. They help define the individual atoms that constitute amino acids, the building blocks of proteins, and provide crucial information about their spatial orientation and chemical properties. We consider a total of 37 distinct atom names that are frequently found in protein structures within the PDB database. These atom types are represented by the set A. To simplify the notation, we assume the set A is sorted in alphanumeric order, A = {C, CA, CB, · · · , N, ND1, ND2, · · · , O, OD1, · · · , SD, SG}, serves as a measure of the combined strength of interaction between chosen pairs of atom types, providing valuable insights into the molecular structure and properties. 3.2 Geometric Subgraph Representation of PPIs In the context of studying protein-protein interactions (PPIs) and predicting the effects of mutations on these interactions, it is important to focus on the relevant regions where the interactions occur. While protein-protein complexes can consist of a large number of atoms, the interactions between proteins primarily take place at specific regions known as interfaces. To streamline computational costs and concentrate on pertinent information, it is common practice to consider only the protein atoms near the binding sites. The binding site, in this context, refers to the region within a certain cutoff distance c from the chain where the mutation occurred. By defining the binding site in this way, we can narrow our focus to the specific area where the interaction and subsequent effects of the mutation are most pronounced. Furthermore, when analyzing the effects of mutations, it is crucial to incorporate geometric graph information from the mutation sites and their neighboring regions. The mutation site is defined as the region within a cutoff distance c from the mutated residue, allowing us to capture the structural changes resulting from the mutation. To construct a site-specific multiscale weighted colored geometric subgraph (MWCGS) representation for a PPI, both the wild-type and mutant-type proteins are considered. This leads to four sets of features for each PPI, corresponding to the two sites and the two types of proteins involved. Each set consists of 37 × 37 = 1369 MWCGS features, representing the interactions between the atom types involved in the PPI. These features encompass diverse chemical and biological properties, such as the presence of specific interatomic interactions involving oxygen and nitrogen atoms, the hydrophobic nature of certain regions, and the ability of atoms to undergo polarization, among other relevant molecular characteristics. By utilizing these site-specific MWCGS features, we can uncover valuable insights into the effects of mutations and the underlying molecular interactions, revealing significant information and characteristics embedded within the PPI system. 3.3 Geometric Graph Learning for PPIs Accurately predicting the changes in binding affinity induced by mutations in protein-protein complexes poses a significant challenge due to the complex nature of these systems. The interactions between proteins are highly intricate, and the effects of mutations can be subtle and context-dependent. Machine learning techniques offer a promising approach to tackle this problem by leveraging the power of data-driven models to capture complex patterns and relationships. Machine learning algorithms can aid in predicting mutation-induced binding affinity changes by learning from a set of training examples that consist of protein-protein complexes with known experimental binding affinities. These algorithms can analyze the features extracted from the complexes, such as geometric graph information, to identify relevant patterns and associations between the features and the binding affinities. By learning from these patterns, the algorithms can generalize and make predictions on unseen protein-protein complexes. There are several machine learning algorithms that can be used in combination with geometric graph features to predict binding affinity changes. These algorithms include random forests [49], support vector machines (SVM) [50], neural networks [51], and gradient boosting trees (GBT) [52]. Each algorithm has its strengths and weaknesses, and their performance can vary depending on the specific problem and dataset. Among these algorithms, gradient boosting trees (GBT) have gained significant popularity in recent years [39]. GBT is an ensemble method that builds a sequence of weak learners, typically decision trees, to correct the errors made by the previous learners. By combining these weak learners, GBT can effectively model complex relationships and improve prediction accuracy. One advantage of GBT is its robustness against overfitting, which is especially beneficial when dealing with a moderate number of features. Additionally, GBT models can provide interpretability, allowing us to gain insights into the factors contributing to the binding affinity changes. The implementation of the GBT algorithm in this study utilized the scikit-learn package (v 0.24.1). To optimize the performance of the GBT model for ensemble methods, specific hyperparameters were fine-tuned. The number of estimators was set to 40000, indicating the number of weak learners in the ensemble, while the learning rate was set to 0.001, determining the contribution of each weak learner to the final prediction. Given the large number of features involved in the prediction task, an efficient training process was achieved by limiting the maximum number of features considered to the square root of the descriptor length. This approach helped expedite the training process without compromising the overall performance of the GBT model. To ensure reliable performance evaluation, fifty runs were performed for each feature set, employing different random seeds. By averaging the results obtained from these runs, a more robust and representative performance measure was obtained. Despite the complexity of the prediction task and the involvement of numerous features, the selected parameter settings and multiple runs yielded satisfactory performance results. The GBT approach was chosen for its ability to effectively handle overfitting, exhibit good performance with moderately sized datasets, and provide interpretable models. These characteristics make GBT a suitable and reliable choice for this study, enabling accurate predictions of mutation-induced binding affinity changes in protein-protein complexes using the provided geometric graph features. This paper is available on under CC 4.0 license. Arxiv

The Methods We Used to Predict Mutation-Induced Binding Free Energy Changes

This paper is available on arxiv under CC 4.0 license. Authors: (1) Md Masud Rana, Department of Mathematics, University of Kentucky; (2) Duc Duy Nguyen, Department of Mathematics, University of Kentucky & ducnguyen@uky.edu. Table of Links Abstract & Introduction Datasets and Results Methods Conclusion, Data and Software Availability, Competing interests, Acknowledgments & References Abstract Protein-protein interactions (PPIs) are critical for various biological processes, and understanding their dynamics is essential for decoding molecular mechanisms and advancing fields such as cancer research and drug discovery. Mutations in PPIs can disrupt protein binding affinity and lead to functional changes and disease. Predicting the impact of mutations on binding affinity is valuable but experimentally challenging. Computational methods, including physics-based and machine learning-based approaches, have been developed to address this challenge. Machine learning-based methods, fueled by extensive PPI datasets such as Ab-Bind, PINT, SKEMPI, and others, have shown promise in predicting binding affinity changes. However, accurate predictions and generalization of these models across different datasets remain challenging. Geometric graph learning has emerged as a powerful approach, combining graph theory and machine learning, to capture structural features of biomolecules. We present GGL-PPI, a novel method that integrates geometric graph learning and machine learning to predict mutation-induced binding free energy changes. GGL-PPI leverages atom-level graph coloring and multi-scale weighted colored geometric subgraphs to extract informative features, demonstrating superior performance on three validation datasets, namely AB-Bind, SKEMPI 1.0, and SKEMPI 2.0 datasets. Evaluation on a blind test set highlights the unbiased predictions of GGLPPI for both direct and reverse mutations. The findings underscore the potential of GGL-PPI in accurately predicting binding free energy changes, contributing to our understanding of PPIs and aiding drug design efforts. — geometric graph, machine learning, protein-protein interactions, mutation, binding free energy changes Keywords 1 Introduction Protein-protein interactions (PPIs) play a fundamental role in numerous biological processes, including cell signaling, metabolic pathways, and immune responses [1;2;3]. Understanding PPIs and their dynamics is crucial for unraveling the intricate mechanisms underlying these processes and holds significant implications for various fields, such as cancer research, drug discovery, and personalized medicine [3;4]. The effects of mutations on PPIs have drawn substantial attention due to their potential impact on protein function and cellular behavior [5;6;7;8]. Missense mutations, which involve single amino acid substitutions, can disrupt the binding affinity between proteins and their partners [9;10]. Such alterations can lead to malfunctioning PPI networks, resulting in diseases, drug resistance, or other molecular disorders [11;12;13;14;15;16]. Therefore, accurate prediction of the impact of mutations on binding affinity holds significant importance in understanding disease mechanisms, facilitating therapeutic interventions, and enabling the design of innovative biopharmaceuticals. One of the key parameters used to assess the impact of mutations on PPIs is the binding free energy change (∆∆G). This thermodynamic parameter quantifies the difference in binding affinity between the wild-type and mutant protein complexes. Experimental determination of ∆∆G values, while accurate, can be tedious and costly. Consequently, there has been a surge in the development of computational methods to predict these energy changes. Broadly, these computation approaches fall into two main categories: physics-based and machine learning-based methods. The former, rooted in biophysical principles, delves into protein conformations and offers a rigorous approach [17;18;19]. However, they often demand significant computational resources and are not always scalable. On the other hand, machine learning-based methods have gained popularity due to their scalability and rapid prediction capabilities. Leveraging the wealth of data from PPI datasets such as ASEdb [20], PINT[21], ProTherm[22], SKEMPI [23;24], and others [25;26;27], machine learning models like mCSM28, BindProf 6 , iSEE29, MutaBind [7] , and several others [30;31;32;33] have been developed. These models have shown significant potential in predicting ∆∆Gs. However, challenges such as imbalanced training datasets, generalization across different PPI datasets, and the intricacy of capturing complex sequence-structure-function relationships remain obstacles [34;35;36;37]. This underscores the need for further research to enhance machine learning methodologies, ensuring accurate and efficient ∆∆G predictions. In recent years, geometric graph learning has emerged as a promising approach for analyzing complex biomolecular systems [38;39;40]. By representing proteins and their interactions as graphs, this methodology leverages the power of graph theory and machine learning to capture essential structural and spatial features of the biomolecular complexes. Specifically, the use of geometric subgraphs, which encode local interactions between atoms and residues, offers a rich representation. This not only sheds light on intricate molecular details but also provides insights into their impact on binding affinity [39]. This work presents a novel method, called GGL-PPI (Geometric Graph Learning for Protein-Protein Interactions), which combines the principles of geometric graph learning and machine learning to predict mutation-induced binding free energy changes. The workflow of GGL-PPI is depicted in Figure 1. Central to its methodology, GGL-PPI utilizes atom-level graph coloring and multi-scale weighted colored geometric subgraphs, enabling the extraction of informative features from protein structures and their interactions. These features serve as inputs to a gradient-boosting tree model, which facilitates precise and consistent predictions of binding free energy change upon mutations. When compared with existing models, GGL-PPI consistently outperforms state-of-the-art approaches across all datasets. Further addressing its generalizability, GGL-PPI was evaluated on a blind test set, Ssym dataset [36]. This evaluation was conducted using a homology-reduced balanced training set to avert data leakage, showcasing GGL-PPI’s robust performance and ability to produce unbiased predictions for both direct and reverse mutations. This paper is available on under CC 4.0 license. Arxiv

GGL-PPI: Geometric Graph Learning to Predict Mutation-Induced Binding Free Energy Changes

This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeffrey Ouyang-Zhang, UT Austin (2) Daniel J. Diaz, UT Austin (3) Adam R. Klivans, UT Austin (4) Philipp Krähenbühl, UT Austin Table of Links Abstract & Introduction Related Works Preliminary Methodology Results Limitations Conclusion, Acknowledgements, & References Experimental Setting & Additional Results 7 Conclusion We present a method that efficiently scales thermodynamic stability prediction from single mutations to higher-order mutations. Our key insight is that the effects of mutations on the same protein are correlated. Thus, for a target protein, it suffices to run a deep backbone once and decode the effect of all mutations simultaneously using a shallow decoder. With the AlphaFold model as our backbone, our method outperforms existing methods on a variety of single and multiple mutation benchmarks. Our method scales to millions of mutations with minimal computational overhead and runs in a fraction of the time it would take prior works. 8 Acknowledgements This work is supported by the NSF AI Institute for Foundations of Machine Learning (IFML). References [1] Randomization of genes by pcr mutagenesis. Genome research 2(1), 28–33 (1992) 2 [2] Adams, J.P., Brown, M.J., Diaz-Rodriguez, A., Lloyd, R.C., Roiban, G.D.: Biocatalysis: A Pharma Perspective. Advanced Synthesis and Catalysis 361(11), 2421–2432 (2019). https://doi.org/10.1002/adsc.201900424 1 [3] Ahdritz, G., Bouatta, N., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T.J., Berenberg, D., Fisk, I., Zanichelli, N., Zhang, B., Nowaczynski, A., Wang, B., Stepniewska-Dziubinska, M.M., Zhang, S., Ojewole, A., Guney, M.E., Biderman, S., Watkins, A.M., Ra, S., Lorenzo, P.R., Nivon, L., Weitzner, B., Ban, Y.E.A., Sorger, P.K., Mostaque, E., Zhang, Z., Bonneau, R., AlQuraishi, M.: Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv (2022). https://doi.org/10.1101/2022.11.20.517210 6 [4] Arnold, F.H.: Design by directed evolution. Accounts of chemical research 31(3), 125–131 (1998) 1, 2, 3 [5] Bell, E.L., Finnigan, W., France, S.P., Green, A.P., Hayes, M.A., Hepworth, L.J., Lovelock, S.L., Niikura, H., Osuna, S., Romero, E., Ryan, K.S., Turner, N.J., Flitsch, S.L.: Biocatalysis. Nature Reviews Methods Primers 1(1), 1–21 (2021). https://doi.org/10.1038/s43586-021-00044-z 1 [6] Benevenuta, S., Pancotti, C., Fariselli, P., Birolo, G., Sanavia, T.: An antisymmetric neural network to predict free energy changes in protein variants. Journal of Physics D: Applied Physics 54(24), 245403 [7] Benevenuta, S., Pancotti, C., Fariselli, P., Birolo, G., Sanavia, T.: An antisymmetric neural network to predict free energy changes in protein variants. Journal of Physics D: Applied Physics 54(24), 245403 (2021) 8 [8] Benevenuta, S., Birolo, G., Sanavia, T., Capriotti, E., Fariselli, P.: Challenges in predicting stabilizing variations: An exploration. Frontiers in Molecular Biosciences 9, 1075570 (2023) 5, 6, 17 [9] Brückner, A., Polge, C., Lentze, N., Auerbach, D., Schlattner, U.: Yeast two-hybrid, a powerful tool for systems biology. International journal of molecular sciences 10(6), 2763–2788 (2009) 1 [10] Cadet, F., Saavedra, E., Syren, P.O., Gontero, B.: Machine learning, epistasis, and protein engineering: From sequence-structure-function relationships to regulation of metabolic pathways. Frontiers in Molecular Biosciences 9, 1098289 (2022) 1 [11] Capriotti, E., Fariselli, P., Casadio, R.: I-mutant2. 0: predicting stability changes upon mutation from the protein sequence or structure. Nucleic acids research 33(suppl_2), W306–W310 (2005) 3, 8 [12] Chen, K., Arnold, F.H.: Tuning the activity of an enzyme for unusual environments: sequential random mutagenesis of subtilisin e for catalysis in dimethylformamide. Proceedings of the National Academy of Sciences 90(12), 5618–5622 (1993) 1, 3 [13] Chen, T., Gong, C., Diaz, D.J., Chen, X., Wells, J.T., Wang, Z., Ellington, A., Dimakis, A., Klivans, A., et al.: Hotprotein: A novel framework for protein thermostability prediction and editing. In: The Eleventh International Conference on Learning Representations (2022) 3 [14] Chen, Y., Lu, H., Zhang, N., Zhu, Z., Wang, S., Li, M.: Premps: Predicting the impact of missense mutations on protein stability. PLoS computational biology 16(12), e1008543 (2020) 3, 8 [15] Cheng, J., Randall, A., Baldi, P.: Prediction of protein stability changes for single-site mutations using support vector machines. Proteins: Structure, Function, and Bioinformatics 62(4), 1125–1132 (2006) 3, 8 [16] Dehghanpoor, R., Ricks, E., Hursh, K., Gunderson, S., Farhoodi, R., Haspel, N., Hutchinson, B., Jagodzinski, F.: Predicting the effect of single and multiple mutations on protein structural stability. Molecules 23(2), 251 (2018) 4, 7 [17] Dehouck, Y., Kwasigroch, J.M., Gilis, D., Rooman, M.: Popmusic 2.1: a web server for the estimation of protein stability changes upon mutation and sequence optimality. BMC bioinformatics 12(1), 1–12 (2011) 3, 8 [18] Diaz, D.J., Gong, C., Ouyang-Zhang, J., Loy, J.M., Wells, J.T., Yang, D., Ellington, A.J., Dimakis, A., Klivans, A.R.: Stability oracle: A structure-based graph-transformer for identifying stabilizing mutations (2023). https://doi.org/10.1101/2023.05.15.540857 3, 4, 7, 8, 16, 17, 18, 19 [19] Diaz, D.J., Kulikova, A.V., Ellington, A.D., Wilke, C.O.: Using machine learning to predict the effects and consequences of mutations in proteins. Current Opinion in Structural Biology 78, 102518 (2023) 3 [20] d’Oelsnitz, S., Diaz, D.J., Acosta, D.J., Schechter, M.W., Minus, M.B., Howard, J.R., Do, H., Loy, J., Alper, H., Ellington, A.D.: Synthetic microbial sensing and biosynthesis of amaryllidaceae alkaloids. bioRxiv pp. 2023–04 (2023) 3 [21] d’Oelsnitz, S., Kim, W., Burkholder, N.T., Javanmardi, K., Thyer, R., Zhang, Y., Alper, H.S., Ellington, A.D.: Using fungible biosensors to evolve improved alkaloid biosyntheses. Nature Chemical Biology 18(9), 981–989 (2022) 1 [22] Ellefson, J.W., Gollihar, J., Shroff, R., Shivram, H., Iyer, V.R., Ellington, A.D.: Synthetic evolutionary origin of a proofreading reverse transcriptase. Science 352(6293), 1590–1593 (2016) 1 [23] Feng, X., Sanchis, J., Reetz, M.T., Rabitz, H.: Enhancing the efficiency of directed evolution in focused enzyme libraries by the adaptive substituent reordering algorithm. Chemistry–A European Journal 18(18), 5646–5654 (2012) 3 [24] Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J.K., Brock, K., Gal, Y., Marks, D.S.: Disease variant prediction with deep generative models of evolutionary data. Nature 599(7883), 91–95 (2021) 9 [25] Gebauer, M., Skerra, A.: Engineered protein scaffolds as next-generation therapeutics. Annual Review of Pharmacology and Toxicology 60, 391–415 (2020). https://doi.org/10.1146/annurev-pharmtox-010818- 021118 1 [26] Gerasimavicius, L., Livesey, B.J., Marsh, J.A.: Correspondence between functional scores from deep mutational scans and predicted effects on protein stability. bioRxiv pp. 2023–02 (2023) 9 [27] Giver, L., Gershenson, A., Freskgard, P.O., Arnold, F.H.: Directed evolution of a thermostable esterase. Proceedings of the National Academy of Sciences 95(22), 12809–12813 (1998) 2, 3 [28] Hie, B.L., Shanker, V.R., Xu, D., Bruun, T.U., Weidenbacher, P.A., Tang, S., Wu, W., Pak, J.E., Kim, P.S.: Efficient evolution of human antibodies from general protein language models. Nature Biotechnology (2023) 3 [29] Jaroszewicz, W., Morcinek-Orłowska, J., Pierzynowska, K., Gaffke, L., W˛egrzyn, G.: Phage display and other peptide display technologies. FEMS Microbiology Reviews 46(2), fuab052 (2022) 1 [30] Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS) 20(4), 422–446 (2002) 6, 17 [31] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al.: Highly accurate protein structure prediction with alphafold. Nature 596(7873), 583–589 (2021) 2, 3, 4, 6, 9, 10 [32] Kellogg, E.H., Leaver-Fay, A., Baker, D.: Role of conformational sampling in computing mutation-induced changes in protein structure and stability. Proteins: Structure, Function, and Bioinformatics 79(3), 830–838 (2011) 3 [33] Klabunde, T., Petrassi, H.M., Oza, V.B., Raman, P., Kelly, J.W., Sacchettini, J.C.: Rational design of potent human transthyretin amyloid disease inhibitors. Nature structural biology 7(4), 312–321 (2000) 1, 2 [34] Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlar, J., Damborsky, J., Pluskal, T., Sivic, J., Mazurenko, S.: Machine learning-guided protein engineering. ACS Catalysis 13, 13863–13895 (2023) 3 [35] Laimer, J., Hofer, H., Fritz, M., Wegenkittl, S., Lackner, P.: Maestro-multi agent stability prediction upon point mutations. BMC bioinformatics 16(1), 1–13 (2015) 3, 4, 7, 8 [36] Li, B., Yang, Y.T., Capra, J.A., Gerstein, M.B.: Predicting changes in protein thermodynamic stability upon point mutation with deep 3d convolutional neural networks. PLoS computational biology 16(11), e1008291 (2020) 3, 4, 8 [37] Li, G., Panday, S.K., Alexov, E.: Saafec-seq: a sequence-based method for predicting the effect of single point mutations on protein thermodynamic stability. International journal of molecular sciences 22(2), 606 (2021) 3 [38] Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al.: Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv (2022) 3, 4, 6, 9, 10, 18 [39] Loell, K., Nanda, V.: Marginal protein stability drives subcellular proteome isoelectric point. Proceedings of the National Academy of Sciences 115(46), 11778–11783 (2018) 1 [40] Lu, H., Diaz, D.J., Czarnecki, N.J., Zhu, C., Kim, W., Shroff, R., Acosta, D.J., Alexander, B.R., Cole, H.O., Zhang, Y., et al.: Machine learning-aided engineering of hydrolases for pet depolymerization. Nature 604(7907), 662–667 (2022) 3 [41] Meghwanshi, G.K., Kaur, N., Verma, S., Dabi, N.K., Vashishtha, A., Charan, P.D., Purohit, P., Bhandari, H.S., Bhojak, N., Kumar, R.: Enzymes for pharmaceutical and therapeutic applications. Biotechnology and Applied Biochemistry 67(4), 586–601 (2020). https://doi.org/10.1002/bab.1919 1 [42] Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., Rives, A.: Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems 34, 29287–29303 (2021) 9, 18 [43] Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., Steinegger, M.: Colabfold: making protein folding accessible to all. Nature methods 19(6), 679–682 (2022) 6 [44] Montanucci, L., Capriotti, E., Frank, Y., Ben-Tal, N., Fariselli, P.: Ddgun: an untrained method for the prediction of protein stability changes upon single and multiple point variations. BMC bioinformatics 20, 1–10 (2019) 3, 4, 7, 8, 16, 18 [45] Montanucci, L., Savojardo, C., Martelli, P.L., Casadio, R., Fariselli, P.: On the biases in predictions of protein stability changes upon variations: the inps test case. Bioinformatics 35(14), 2525–2527 (2019) 16 [46] Musdal, Y., Govindarajan, S., Mannervik, B.: Exploring sequence-function space of a poplar glutathione transferase using designed information-rich gene variants. Protein Engineering, Design and Selection 30(8), 543–549 (2017) 3 [47] Nijkamp, E., Ruffolo, J., Weinstein, E.N., Naik, N., Madani, A.: Progen2: exploring the boundaries of protein language models. arXiv preprint arXiv:2206.13517 (2022) 9 [48] Nikam, R., Kulandaisamy, A., Harini, K., Sharma, D., Gromiha, M.M.: Prothermdb: thermodynamic database for proteins and mutants revisited after 15 years. Nucleic acids research 49(D1), D420–D424 (2021) 2, 16 [49] Notin, P., Dias, M., Frazer, J., Hurtado, J.M., Gomez, A.N., Marks, D., Gal, Y.: Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In: International Conference on Machine Learning. pp. 16990–17017. PMLR (2022) 2, 3, 9, 16, 18 [50] Notin, P.M., Van Niekerk, L., Kollasch, A.W., Ritter, D., Gal, Y., Marks, D.: Trancepteve: Combining family-specific and family-agnostic models of protein sequences for improved fitness prediction. bioRxiv pp. 2022–12 (2022) 9 [51] Nutschel, C., Fulton, A., Zimmermann, O., Schwaneberg, U., Jaeger, K.E., Gohlke, H.: Systematically scrutinizing the impact of substitution sites on thermostability and detergent tolerance for bacillus subtilis lipase a. Journal of chemical information and modeling 60(3), 1568–1584 (2020) 9 [52] Paik, I., Ngo, P.H., Shroff, R., Diaz, D.J., Maranhao, A.C., Walker, D.J., Bhadra, S., Ellington, A.D.: Improved bst dna polymerase variants derived via a machine learning approach. Biochemistry (2021) 3 [53] Pak, M.A., Markhieva, K.A., Novikova, M.S., Petrov, D.S., Vorobyev, I.S., Maksimova, E.S., Kondrashov, F.A., Ivankov, D.N.: Using alphafold to predict the impact of single mutations on protein stability and function. Plos one 18(3), e0282689 (2023) 3 [54] Pancotti, C., Benevenuta, S., Birolo, G., Alberini, V., Repetto, V., Sanavia, T., Capriotti, E., Fariselli, P.: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset. Briefings in Bioinformatics 23(2), bbab555 (2022) 2, 8, 16, 17 [55] Pancotti, C., Benevenuta, S., Repetto, V., Birolo, G., Capriotti, E., Sanavia, T., Fariselli, P.: A deep-learning sequence-based method to predict protein stability changes upon genetic variations. Genes 12(6), 911 (2021) 8 [56] Pires, D.E., Ascher, D.B., Blundell, T.L.: Duet: a server for predicting effects of mutations on protein stability using an integrated computational approach. Nucleic acids research 42(W1), W314–W319 (2014) 3, 8 [57] Pires, D.E., Ascher, D.B., Blundell, T.L.: mcsm: predicting the effects of mutations in proteins using graph-based signatures. Bioinformatics 30(3), 335–342 (2014) 8 [58] Pucci, F., Schwersensky, M., Rooman, M.: Artificial intelligence challenges for predicting the impact of mutations on protein stability. Current opinion in structural biology 72, 161–168 (2022) 17 [59] Qiu, Y., Wei, G.W.: Persistent spectral theory-guided protein engineering. Nature Computational Science pp. 1–15 (2023) 3, 6, 17 [60] Rao, R.M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., Rives, A.: Msa transformer. In: International Conference on Machine Learning. pp. 8844–8856. PMLR (2021) 3, 9, 10 [61] Riesselman, A.J., Ingraham, J.B., Marks, D.S.: Deep generative models of genetic variation capture the effects of mutations. Nature methods 15(10), 816–822 (2018) 9 [62] Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C.L., Ma, J., Fergus, R.: Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS (2019). https://doi.org/10.1101/622803 3, 7 [63] Rodrigues, C.H., Pires, D.E., Ascher, D.B.: Dynamut: predicting the impact of mutations on protein conformation, flexibility and stability. Nucleic acids research 46(W1), W350–W355 (2018) 3, 8 [64] Rodrigues, C.H., Pires, D.E., Ascher, D.B.: Dynamut2: Assessing changes in stability and flexibility upon single and multiple point missense mutations. Protein Science 30(1), 60–69 (2021) 3, 4, 7 [65] Sanavia, T., Birolo, G., Montanucci, L., Turina, P., Capriotti, E., Fariselli, P.: Limitations and challenges in protein stability prediction upon genome variations: towards future applications in precision medicine. Computational and Structural Biotechnology Journal 18, 1968–1979 (2020). https://doi.org/10.1016/j.csbj.2020.07.011, https://doi.org/10.1016/j.csbj.2020.07.011 6, 17 [66] Savojardo, C., Fariselli, P., Martelli, P.L., Casadio, R.: Inps-md: a web server to predict stability of protein variants from sequence and structure. Bioinformatics 32(16), 2542–2544 (2016) 3, 8 [67] Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F., Serrano, L.: The foldx web server: an online force field. Nucleic acids research 33(suppl_2), W382–W388 (2005) 3, 7, 8, 18 [68] Shroff, R., Cole, A.W., Diaz, D.J., Morrow, B.R., Donnell, I., Annapareddy, A., Gollihar, J., Ellington, A.D., Thyer, R.: Discovery of novel gain-of-function mutations guided by structure-based deep learning. ACS synthetic biology 9(11), 2927–2935 (2020) 3 [69] Starr, T.N., Thornton, J.W.: Epistasis in protein evolution. Protein science 25(7), 1204–1218 (2016) 1 [70] Steinegger, M., Soding, J.: Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology 35(11), 1026–1028 (2017) 6 [71] Stemmer, W.P.: Rapid evolution of a protein in vitro by dna shuffling. Nature 370(6488), 389–391 (1994) 2, 3 [72] Tsuboyama, K., Dauparas, J., Chen, J., Laine, E., Mohseni Behbahani, Y., Weinstein, J.J., Mangan, N.M., Ovchinnikov, S., Rocklin, G.J.: Mega-scale experimental analysis of protein folding stability in biology and protein design. bioRxiv pp. 2022–12 (2022) 1, 2, 6, 16 [73] Tung, J.W., Heydari, K., Tirouvanziam, R., Sahaf, B., Parks, D.R., Herzenberg, L.A., Herzenberg, L.A.: Modern flow cytometry: a practical approach. Clinics in laboratory medicine 27(3), 453–468 (2007) 1 [74] Umerenkov, D., Shashkova, T.I., Strashnov, P.V., Nikolaev, F., Sindeeva, M., Ivanisenko, N.V., Kardymon, O.L.: Prostata: Protein stability assessment using transformers. bioRxiv pp. 2022–12 (2022) 3, 4, 6, 7, 8, 18, 19 [75] Wittmann, B.J., Johnston, K.E., Wu, Z., Arnold, F.H.: Advances in machine learning for directed evolution. Current opinion in structural biology 69, 11–18 (2021) 3 [76] Wittmann, B.J., Yue, Y., Arnold, F.H.: Informed training set design enables efficient machine learningassisted directed protein evolution. Cell Systems 12(11), 1026–1045 (2021) 3 [77] Worth, C.L., Preissner, R., Blundell, T.L.: Sdm—a server for predicting effects of mutations on protein stability and malfunction. Nucleic acids research 39(suppl_2), W215–W222 (2011) 3, 8 [78] Wu, S., Snajdrova, R., Moore, J.C., Baldenius, K., Bornscheuer, U.T.: Biocatalysis: Enzymatic Synthesis for Industrial Applications. Angewandte Chemie - International Edition 60(1), 88–119 (2021). https://doi.org/10.1002/anie.202006648 1 [79] Yeung, N., Lin, Y.W., Gao, Y.G., Zhao, X., Russell, B.S., Lei, L., Miner, K.D., Robinson, H., Lu, Y.: Rational design of a structural and functional nitric oxide reductase. Nature 462(7276), 1079–1082 (2009) 1, 2 [80] Zhou, B., Lv, O., Yi, K., Xiong, X., Tan, P., Hong, L., Wang, Y.G.: Accurate and definite mutational effect prediction with lightweight equivariant graph neural networks. arXiv preprint arXiv:2304.08299 (2023) 3 [81] Zhou, Y., Pan, Q., Pires, D.E., Rodrigues, C.H., Ascher, D.B.: Ddmut: predicting effects of mutations on protein stability using deep learning. Nucleic Acids Research p. gkad472 (2023) 3

Predicting a Protein’s Stability
under a Million Mutations: Conclusion, Acknowledgement & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeffrey Ouyang-Zhang, UT Austin (2) Daniel J. Diaz, UT Austin (3) Adam R. Klivans, UT Austin (4) Philipp Krähenbühl, UT Austin Table of Links Abstract & Introduction Related Works Preliminary Methodology Results Limitations Conclusion, Acknowledgements, & References Experimental Setting & Additional Results A Experimental Setting We elaborate on our experimental protocol, including the datasets used for training and testing, the evaluation metrics, and the baselines that calibrate our models. A.1 Datasets A fundamental principle in machine learning is to partition data so that the model is not evaluated on the same data it was trained on. This is essential to assess the model’s ability to generalize beyond its training data (in our case, to new proteins). Protein sequences are regarded as the same protein when they have large sequence similarity (e.g. > 50%), even when they are not exactly the same string. The impact of training on protein sequences similar to those in the test set has received considerable attention [18, 45, 54]. We list the datasets we use below and report the maximum sequence similarity of our training set with each test set. This ensures a clear separation between our training and test sets. [72] is a large scale dataset containing mutant proteins with ∆∆G measurements. The dataset is generated by cDNA display proteolysis, a massively parallel technique that measures thermodynamic folding stability for millions of protein variants [72]. The cDNA proteins average 56.1 amino acids in length with a maximum length of 72 and a minimum length of 30 amino acids. The mean MSA depth is 7797 with a standard deviation of 6282. The maximum depth is 23525 and the minimum depth is 5. The mutations contain either one or two substitutions on a small protein. We follow the train-val split introduced in [18] and additionally filter out proteins in our training set that are similar to those in our evaluation datasets. Specifically, we train on 116 proteins with 213,000 total mutations, of which 97,000 are double mutants and 117,000 are single mutants. We hold out cDNA2, a validation set of 18 mini-proteins with 22,000 total double mutations. The proteins in our training set have at most 36% sequence similarity to those in cDNA2. A comprehensive analysis of the dataset, experimental assay, and filtering criteria are found in their paper [1]. cDNA proteolysis [48] contains experimentally measured thermodynamic data ∆∆G for protein mutants. We use PTMul, a split containing only higher-order mutations curated by DDGun [44]. There are 88 proteins with 846 total mutations in PTMul. The proteins in our training set have at most 35% sequence similarity to those in PTMul. ProTherm [54] is a manually curated dataset of mutant proteins and their ∆∆G measurements. S669 contains mutations with one substitution. There are 94 proteins with 669 total single mutations in S669. The proteins in our training set have at most 30% sequence similarity to those in S669. S669 [49] is a curated set of proteins in which mutations are assessed for some fitness phenotype. These fitness phenotypes vary drastically, from cell growth to expression to fluorescence. The fitness values are always normalized so a higher value indicates stronger fitness. For classification metrics, the fitness values are binarized by some threshold either defined manually or arbitrarily at the median value. We also consider a subset of the proteins in ProteinGym in which fitness correlates highly with thermodynamic stability. These phenotypes include thermostability, expression, etc. There are 87 proteins totaling over 1.6 million (potentially higher-order) mutations. The proteins in our training set have at most 45% sequence similarity to those in ProteinGym. ProteinGym [18] is an aggregate dataset with mutant proteins and their ∆∆G measurements. T2837 contains mutations with one substitution. There are 129 proteins with 2837 total single mutations in T2837. The proteins in our training set have at most 34% sequence similarity to those in T2837. T2837 A.2 Metrics We evaluate our model’s ability to assess mutations in ProTherm, S669, and ProteinGym. Additionally, we evaluate our model’s ability to detect stabilizing mutations in cDNA2. A.2.1 Mutation Assessment The model predicts the ∆∆G measurement under all mutations with experimental data. The models are evaluated by the following regression and classification metrics. The classification metrics are useful in applications where it is unnecessary to provide exact ∆∆G and a binary stabilizing or de-stabilizing label (or third label, neutral) is sufficient. Following the literature, we additionally evaluate reverse mutations (in contrast to the standard “forward” mutations) where the “to” and “from” amino acids are flipped (e.g. P1K to K1P). The model takes the mutant sequence as input and is expected to output the negation of the original ∆∆G value when mutating to the original native amino acid. evaluates the model’s ability to rank the mutations by their ∆∆G values. This metric considers only the relative ordering between mutations and disregards the predicted values. Spearman has its limitations because our datasets contain predominantly destabilizing mutations (∆∆G > 0). Spearman applied on these datasets overwhelmingly measures the model’s ability to rank destabilizing mutations which does not directly translate to identifying the most stabilizing mutations [8, 58]. Spearman correlation coefficient (rs) provides a measure of how closely the predicted measurements align with the true measurements. Root Mean Square Error (RMSE) is a popular classification metric that focuses on positives. The receiver operating characteristic curve is created by plotting precision against recall at various classification thresholds. Precision is calculated as the ratio of true positives to the sum of true positives and false positives. Recall is calculated as the ratio of true positives to the sum of true positives and false negatives. Area Under the receiver operating characteristic Curve (AUC) evaluates a model’s classification performance under imbalanced datasets. It takes into account true positive, true negative, false positive, and false negative predictions to measure the overall performance of a model. Matthew correlation coefficient (MCC) A.2.2 Mutation Detection The regression metrics for mutation assessment are not directly suitable for protein engineering where the goal is to find stabilizing mutations. In addition to the classification metrics, we evaluate our models using per-protein metrics which evaluate the model’s ability to detect a few stabilizing mutations from a large number of possible mutations. These metrics are applied to heavily annotated proteins (∼ 1000 ∆∆G measurements) with randomly sampled mutations. is the Spearman coefficient calculated on the subset of experimentally stabilizing mutations. This mimics the case in protein engineering where there are too many stabilizing mutations to test in vitro and the goal is to prioritize the most stabilizing ones. This metric is used in [18, 65]. Stabilizing Spearman correlation coefficient rs (nDCG) [30] quantifies the quality of a ranking among mutations by considering both the experimental ∆∆G and predicted rank of each mutation. Specifically, nDCG computes the sum of experimentally stabilizing ∆∆Gs weighted inversely by their ranks. This metric rewards a model that ranks the best mutations first. We truncate the ranking at K = 30 mutations to approximate the number of in-vitro experiments that can typically be run in parallel. This metric is found in [59]. Normalized Discounted Cumulative Gain measures the fraction of experimentally stabilizing mutations among the top K = 30 predicted mutations by the model. This simulates the success rate a protein engineer might expect when deploying this model. K is set to approximate the number of in-vitro experiments that can typically be run in parallel. Detection Precision (DetPr) A.3 Baselines Several baselines calibrate performance across datasets. A subset of the critical baselines is introduced briefly (see [54] for more). Mean is a simple baseline in which the mean measurement is predicted for all mutations. This baseline is not conditioned on the protein or mutation and instead uses only the dataset statistic. Multiple Sequence Alignment (MSA) leverages the statistics from a set of similar protein sequences to rank mutations. Similar protein sequences are fetched, aligned, and indexed at a mutation position. This yields an amino acid frequency count normalized into a probability distribution. The score is defined as the likelihood ratio between the mutated and native amino acid types. While calibrated for a given position and protein, it may not be well-calibrated across proteins. [38] also computes the likelihood ratio between a mutated and native amino acid type [42]. Unlike MSA, the probability distribution is decoded from ESM2 [38], a model pre-trained on masked amino acid prediction. Like MSA, ESM may not be well-calibrated across proteins. ESM [44] uses a linear combination of scores derived from evolutionary and structural features to compute ∆∆G. DDGun leverages an empirical force field to estimate the change in stability. It models the Van Der Waals interactions, solvation and hydrogen bonding interactions, changes in entropy, and other physical values. FoldX [67] is a sequence model for stability prediction built on the ESM2 model. It takes native and mutant protein sequences as input and produces a ∆∆G value. PROSTATA [74] [18] is a structure-based graph-transformer model for stability prediction. Stability Oracle [49] is a specialized autoregressive transformer that uses grouped attention to encourage specialization across heads. Tranception B Additional Results B.1 Higher-order Mutation Results on PTMul Breakdown ProTherm Multiple (PTMul) contains 846 mutations, of which 536 are double mutations and 310 mutations with more than two mutations. We breakdown the performance on PTMul into double mutations and higher-order mutations (greater than 2) in Table 6. While our model performs similarly against the additive baseline on double mutations, Mutate Everything outperforms the additive baseline on higher-order mutations. Table 6: outperforms existing methods, especially when there are multiple mutations. Results on PTMul splits containing double mutations and more than 2 mutations. Mutate Everything Table 7: . We report regression and classification metrics on the forward and reverse datasets. rs refers to Spearman correlation coefficient. Our training proteins have at most sequence similarity with those in T2837. is state-of-the-art on T2837 forward. Comparisons on Single Mutation Thermodynamic Stability prediction in T2837 34% Mutate Everything Table 8: TR refers to Thermodynamic Reversibility. rp is Pearson correlation coefficient. rs is Spearman correlation coefficient. Mutations that failed our data pipeline are excluded. Mutate Everything regression and classification metrics on T2837 and its reverse datasets. B.2 Single Mutation Results on T2837 We include our results on the newly proposed T2837 dataset in Table 7 [18]. Mutate Everything achieves a Spearman correlation of 0.65 compared to 0.62 of Stability Oracle, outperforming existing methods on the forward dataset. PROSTATA-IFML [18] fine-tunes PROSTATA [74] on the cDNA proteolysis dataset for more direct comparison. Mutate Everything performs competitively with other sequence models on the reverse dataset even without training on reverse mutations. We show our performance on additional metrics for T2837 in Table 8.

AMILO

amilo.co

-- employees

Since n.d.

COMPANY RANKING

AMILO

EVERGREEN INDEX #1471

Representing this company? Click here to claim and customize this page!

Amilo WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AMILO

amilo.co

-- employees

Since n.d.

COMPANY RANKING

AMILO

EVERGREEN INDEX #1471

RELATED COMPANIESRANK

STORIES ABOUT AMILO

Representing this company? Click here to claim and customize this page!

Amilo WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!