Starlearn

starlearn

How to Turn Your OpenAPI Specification Into an AI Chatbot With RAG

Authors:
(1) Antonio Riotto, Département de Physique Theorique, Universite de Geneve, 24 quai Ansermet, CH-1211 Geneve 4, Switzerland and Gravitational Wave Science Center (GWSC), Universite de Geneve, CH-1211 Geneva, Switzerland;
(2) Joe Silk, Institut d’Astrophysique, UMR 7095 CNRS, Sorbonne Universite, 98bis Bd Arago, 75014 Paris, France, Department of Physics and Astronomy, The Johns Hopkins University, Baltimore MD 21218, USA, and Beecroft Institute of Particle Astrophysics and Cosmology, Department of Physics, University of Oxford, Oxford OX1 3RH, UK. Table of Links Abstract and 1 Introduction 2 Some open questions 2.1 What is the abundance of PBHs? 2.2 What is the effect of PBH clustering? 2.3 What fraction of the currently observed GW events can be ascribed to PBHs? 2.4 Are PBHs the Dark Matter? 3 The PBH Roadmap 3.1 High redshift mergers 3.2 Sub-solar PBHs 3.3 Plugging the pair instability gap with PBH? 3.4 PBH eccentricity, 3.5 PBH spin and 3.6 Future gamma-ray telescopes 4 Conclusions and References 3.2 Sub-solar PBHs The detection of sub-solar BHs in a compact binary merger is regarded as one of the smoking gun signatures of PBHs. Indeed, BHs of astrophysical origin are not expected. The sub-solar mass range may be however populated by other compact objects like neutron stars, white dwarfs, or exotic compact objects [51] (e.g. boson stars [52]). GW signatures of any population of PBH below the astrophysical production provide a powerful statistical constraint on any solar or subsolar component, as evidenced by the sharp decline observed in 03 events. There could of course be rare events generated by solar or subsolar PBHs that might contribute to DM at the O(10%) level. One key ingredient is therefore to distinguish PBHs from other compact objects, e.g. through the measurements of the tidal disruption and tidal deformability Λ. A recent analysis has shown that for subsolar neutron-star binaries, the O4 and O5 projected sensitivities would allow measuring the effect of tidal disruption on the waveform in a large portion of the parameter space, also constraining the tidal deformability at O(10%) level, thus excluding a primordial origin of the binary. Viceversa, for subsolar PBH binaries, model-agnostic upper bounds on the tidal deformability can rule out neutron stars or more exotic competitors. Assuming events similar to the sub-threshold candidate SSM200308 reported in LVK O3b data are PBH binaries, O4 projected sensitivity would allow ruling out the presence of neutron-star tidal effects at ≈ 3σ C.L., thus strengthening the PBH hypothesis. Future experiments would lead to even stronger (> 5σ) conclusions on potential discoveries of this kind, see Fig. 2. PBHs of lunar mass, a regime where hints of unaccounted-for microlensing events have been reported by the OGLE survey of the galactic bulge [53], could be sufficiently numerous that nearby binaries might be detectable with future generation detectors [54]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Antonio Riotto, Département de Physique Theorique, Universite de Geneve, 24 quai Ansermet, CH-1211 Geneve 4, Switzerland and Gravitational Wave Science Center (GWSC), Universite de Geneve, CH-1211 Geneva, Switzerland; (2) Joe Silk, Institut d’Astrophysique, UMR 7095 CNRS, Sorbonne Universite, 98bis Bd Arago, 75014 Paris, France, Department of Physics and Astronomy, The Johns Hopkins University, Baltimore MD 21218, USA, and Beecroft Institute of Particle Astrophysics and Cosmology, Department of Physics, University of Oxford, Oxford OX1 3RH, UK. Authors: Authors: (1) Antonio Riotto, Département de Physique Theorique, Universite de Geneve, 24 quai Ansermet, CH-1211 Geneve 4, Switzerland and Gravitational Wave Science Center (GWSC), Universite de Geneve, CH-1211 Geneva, Switzerland; (2) Joe Silk, Institut d’Astrophysique, UMR 7095 CNRS, Sorbonne Universite, 98bis Bd Arago, 75014 Paris, France, Department of Physics and Astronomy, The Johns Hopkins University, Baltimore MD 21218, USA, and Beecroft Institute of Particle Astrophysics and Cosmology, Department of Physics, University of Oxford, Oxford OX1 3RH, UK. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Some open questions 2 Some open questions 2.1 What is the abundance of PBHs? 2.1 What is the abundance of PBHs? 2.2 What is the effect of PBH clustering? 2.2 What is the effect of PBH clustering? 2.3 What fraction of the currently observed GW events can be ascribed to PBHs? 2.3 What fraction of the currently observed GW events can be ascribed to PBHs? 2.4 Are PBHs the Dark Matter? 2.4 Are PBHs the Dark Matter? 3 The PBH Roadmap 3 The PBH Roadmap 3.1 High redshift mergers 3.1 High redshift mergers 3.2 Sub-solar PBHs 3.2 Sub-solar PBHs 3.3 Plugging the pair instability gap with PBH? 3.3 Plugging the pair instability gap with PBH? 3.4 PBH eccentricity, 3.5 PBH spin and 3.6 Future gamma-ray telescopes 3.4 PBH eccentricity, 3.5 PBH spin and 3.6 Future gamma-ray telescopes 4 Conclusions and References 4 Conclusions and References 3.2 Sub-solar PBHs The detection of sub-solar BHs in a compact binary merger is regarded as one of the smoking gun signatures of PBHs. Indeed, BHs of astrophysical origin are not expected. The sub-solar mass range may be however populated by other compact objects like neutron stars, white dwarfs, or exotic compact objects [51] (e.g. boson stars [52]). GW signatures of any population of PBH below the astrophysical production provide a powerful statistical constraint on any solar or subsolar component, as evidenced by the sharp decline observed in 03 events. There could of course be rare events generated by solar or subsolar PBHs that might contribute to DM at the O(10%) level. One key ingredient is therefore to distinguish PBHs from other compact objects, e.g. through the measurements of the tidal disruption and tidal deformability Λ. A recent analysis has shown that for subsolar neutron-star binaries, the O4 and O5 projected sensitivities would allow measuring the effect of tidal disruption on the waveform in a large portion of the parameter space, also constraining the tidal deformability at O(10%) level, thus excluding a primordial origin of the binary. Viceversa, for subsolar PBH binaries, model-agnostic upper bounds on the tidal deformability can rule out neutron stars or more exotic competitors. Assuming events similar to the sub-threshold candidate SSM200308 reported in LVK O3b data are PBH binaries, O4 projected sensitivity would allow ruling out the presence of neutron-star tidal effects at ≈ 3σ C.L., thus strengthening the PBH hypothesis. Future experiments would lead to even stronger (> 5σ) conclusions on potential discoveries of this kind, see Fig. 2. PBHs of lunar mass, a regime where hints of unaccounted-for microlensing events have been reported by the OGLE survey of the galactic bulge [53], could be sufficiently numerous that nearby binaries might be detectable with future generation detectors [54]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Sub-solar Primordial Black Holes

How to Add Full Text Search to Your Django App with django-watson

98% of Data Strategies Fail: Let's Fix It

The Snowflake Hack and Its Domino Effect

স্নোফ্লেক হ্যাক এবং এর ডমিনো প্রভাব

Der Snowflake-Hack und sein Dominoeffekt

El truco del copo de nieve y su efecto dominó

Le Snowflake Hack et son effet domino

स्नोफ्लेक हैक और इसका डोमिनो प्रभाव

スノーフレークハッキングとそのドミノ効果

눈송이 해킹과 도미노 효과

O truque do floco de neve e seu efeito dominó

Взлом Snowflake и его эффект домино

Kar Tanesi Hilesi ve Domino Etkisi

Hack bông tuyết và hiệu ứng Domino của nó

雪花黑客及其多米诺骨牌效应

Leveraging SwarmUI & Stable Diffusion 3 on Cloud Platforms: A Guide to Kaggle (No-Cost), Massed Comp

Authors:
(1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com};
(2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Dataset Design Evaluation Methodology Experiments and Analysis Related Work Conclusion, Reproducibility, and References A. Complete list of tools B. Scenario Prompt C. Unrealistic Queries D. Nuances comparing prior work 5 RELATED WORK In Section 1, we described our desired criteria for evaluating tool-using LLM-based assistants: using dialogue to specify intents requiring multi-step tool invocations, and actions rather than only retrieving information, for a fully automated evaluation not requiring human judgement over the outputs of the system under test. Table 5 summarizes how other work about evaluating tool-using LLMs compares along these factors. We describe the related work in greater detail below. Tool-augmented LLMs are also known as tool-augmented learning, tool LLMs, tool-learning, augmented language models (ALMs), or tool manipulation with LLMs (Xu et al., 2023; Mialon et al., 2023; Qin et al., 2023a). Development in this area consists of improving LLM performance in traditional tasks by giving them access to tools such as a calculator or search engine (Lu et al., 2023a; Yao et al., 2022b; Paranjape et al., 2023; Hao et al., 2023). It can also include applying LLMs to traditional automation tasks such as embodied robotics or browsing the web (Liu et al., 2023b; Deng et al., 2023; Yao et al., 2022a; Liang et al., 2023), dubbed “LLM-as-agent” by AgentBench (Liu et al., 2023a). Traditional tasks that tool-augmented LLMs have been applied to include question answering such as ScienceQA (Saikh et al., 2022) or HotPotQA (Yang et al., 2018), mathematical reasoning (Cobbe et al., 2021; Lu et al., 2023b; Qiao et al., 2023), multilingual translation and QA (Lewis et al., 2020; Scarton et al., 2019), open-domain QA (Zhu et al., 2021), and commonsense QA (Talmor et al., 2019) to name a few. These tasks are useful for demonstrating the benefits of augmenting LLMs with tool usage, but fail to fully distinguish how much LLMs rely on internal knowledge vs good usage of tools (Zhuang et al., 2023). They also fail to incorporate the use of tools that affect the external world since they are unnecessary for those tasks. Common agent benchmarks that have been applied to tool-augmented LLMs include WebShop (Yao et al., 2022a), Tabletop (Liang et al., 2023), Mind2Web (Deng et al., 2023), and ALFWorld (Shridhar et al., 2020). Additionally, AgentBench compiles Mind2Web, WebShop, and ALFWorld into a unified benchmark while adding additional agent environments such as interacting with a bash terminal, creating SQL commands to query a database, interacting with a knowledge graph, digital card game simulations, and lateral thinking puzzles (Liu et al., 2023a). ToolBench does something similar by compiling Tabletop and Webshop while introducing a variety of other tasks consisting of predicting a single API call. These benchmarks are useful for evaluating the effectiveness of tool-augmented LLMs in a variety of autonomous situations. However, none of them test tool-augmented LLMs in a conversational setting. Furthermore, tasks in these benchmarks consist of issuing a single utterance which an agent then tries to accomplish without any further human interaction. This is in contrast to ToolTalk, where a conversation will consist of multiple utterances with multiple intermediary tasks. Past works have also created datasets for evaluating tool-augmented LLM-based assistants. Examples include ToolLLM (Qin et al., 2023b), API-Bank (Li et al., 2023), TPTU (Ruan et al., 2023), Gorilla (Patil et al., 2023), RestGPT (Song et al., 2023), GPT4Tools (Yang et al., 2023), and ToolAlpaca (Tang et al., 2023) among others. Unfortunately, many of these datasets require manual inspection of the outputs of the assistant under test to perform a complete evaluation. A lot of them also have unrealistic queries, and do not reflect questions or intents humans are likely to say in real life.[4] Many of them are also simple, where the solution requires one or two tool calls (Li et al., 2023; Ruan et al., 2023; Yang et al., 2023; Tang et al., 2023). Except for Li et al. (2023), these consider users’ utterances in isolation rather than as part of a conversation or dialogue. There also exists a corpus of work on task-oriented dialogue systems. This area of research is focused on collecting realistic, task-oriented dialogue for the tasks of intent classification and slot filling (Larson & Leach, 2022). Some popular task-oriented dialogue datasets include MultiWoz (Budzianowski et al., 2018), Taskmaster and TicketTalk (Byrne et al., 2019; 2020), and STAR and STARv2 (Mosig et al., 2020; Zhao et al., 2022). The goals of creating realistic dialogue and evaluating on intent classification and slot filling have some overlap with ToolTalk. However, taskoriented dialogue datasets usually only predict a single intent per user utterance, do not simulate plugins or tools, and do not provide execution feedback for predicted tool calls. TicketTalk (Byrne et al., 2020) is notable in that it does provide a simulation of a movie booking API, however this API does not provide execution feedback and is not rigorously defined allowing for loose arguments like “here” or “now”. This paper is available on arxiv under CC 4.0 license. [4] We include a few examples from various papers in Appendix C. Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Authors: Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Abstract and Intro Dataset Design Dataset Design Evaluation Methodology Evaluation Methodology Experiments and Analysis Experiments and Analysis Related Work Related Work Conclusion, Reproducibility, and References Conclusion, Reproducibility, and References A. Complete list of tools A. Complete list of tools B. Scenario Prompt B. Scenario Prompt C. Unrealistic Queries C. Unrealistic Queries D. Nuances comparing prior work D. Nuances comparing prior work 5 RELATED WORK In Section 1, we described our desired criteria for evaluating tool-using LLM-based assistants: using dialogue to specify intents requiring multi-step tool invocations, and actions rather than only retrieving information, for a fully automated evaluation not requiring human judgement over the outputs of the system under test. Table 5 summarizes how other work about evaluating tool-using LLMs compares along these factors. We describe the related work in greater detail below. Tool-augmented LLMs are also known as tool-augmented learning, tool LLMs, tool-learning, augmented language models (ALMs), or tool manipulation with LLMs (Xu et al., 2023; Mialon et al., 2023; Qin et al., 2023a). Development in this area consists of improving LLM performance in traditional tasks by giving them access to tools such as a calculator or search engine (Lu et al., 2023a; Yao et al., 2022b; Paranjape et al., 2023; Hao et al., 2023). It can also include applying LLMs to traditional automation tasks such as embodied robotics or browsing the web (Liu et al., 2023b; Deng et al., 2023; Yao et al., 2022a; Liang et al., 2023), dubbed “LLM-as-agent” by AgentBench (Liu et al., 2023a). Traditional tasks that tool-augmented LLMs have been applied to include question answering such as ScienceQA (Saikh et al., 2022) or HotPotQA (Yang et al., 2018), mathematical reasoning (Cobbe et al., 2021; Lu et al., 2023b; Qiao et al., 2023), multilingual translation and QA (Lewis et al., 2020; Scarton et al., 2019), open-domain QA (Zhu et al., 2021), and commonsense QA (Talmor et al., 2019) to name a few. These tasks are useful for demonstrating the benefits of augmenting LLMs with tool usage, but fail to fully distinguish how much LLMs rely on internal knowledge vs good usage of tools (Zhuang et al., 2023). They also fail to incorporate the use of tools that affect the external world since they are unnecessary for those tasks. Common agent benchmarks that have been applied to tool-augmented LLMs include WebShop (Yao et al., 2022a), Tabletop (Liang et al., 2023), Mind2Web (Deng et al., 2023), and ALFWorld (Shridhar et al., 2020). Additionally, AgentBench compiles Mind2Web, WebShop, and ALFWorld into a unified benchmark while adding additional agent environments such as interacting with a bash terminal, creating SQL commands to query a database, interacting with a knowledge graph, digital card game simulations, and lateral thinking puzzles (Liu et al., 2023a). ToolBench does something similar by compiling Tabletop and Webshop while introducing a variety of other tasks consisting of predicting a single API call. These benchmarks are useful for evaluating the effectiveness of tool-augmented LLMs in a variety of autonomous situations. However, none of them test tool-augmented LLMs in a conversational setting. Furthermore, tasks in these benchmarks consist of issuing a single utterance which an agent then tries to accomplish without any further human interaction. This is in contrast to ToolTalk, where a conversation will consist of multiple utterances with multiple intermediary tasks. Past works have also created datasets for evaluating tool-augmented LLM-based assistants. Examples include ToolLLM (Qin et al., 2023b), API-Bank (Li et al., 2023), TPTU (Ruan et al., 2023), Gorilla (Patil et al., 2023), RestGPT (Song et al., 2023), GPT4Tools (Yang et al., 2023), and ToolAlpaca (Tang et al., 2023) among others. Unfortunately, many of these datasets require manual inspection of the outputs of the assistant under test to perform a complete evaluation. A lot of them also have unrealistic queries, and do not reflect questions or intents humans are likely to say in real life.[4] Many of them are also simple, where the solution requires one or two tool calls (Li et al., 2023; Ruan et al., 2023; Yang et al., 2023; Tang et al., 2023). Except for Li et al. (2023), these consider users’ utterances in isolation rather than as part of a conversation or dialogue. There also exists a corpus of work on task-oriented dialogue systems. This area of research is focused on collecting realistic, task-oriented dialogue for the tasks of intent classification and slot filling (Larson & Leach, 2022). Some popular task-oriented dialogue datasets include MultiWoz (Budzianowski et al., 2018), Taskmaster and TicketTalk (Byrne et al., 2019; 2020), and STAR and STARv2 (Mosig et al., 2020; Zhao et al., 2022). The goals of creating realistic dialogue and evaluating on intent classification and slot filling have some overlap with ToolTalk. However, taskoriented dialogue datasets usually only predict a single intent per user utterance, do not simulate plugins or tools, and do not provide execution feedback for predicted tool calls. TicketTalk (Byrne et al., 2020) is notable in that it does provide a simulation of a movie booking API, however this API does not provide execution feedback and is not rigorously defined allowing for loose arguments like “here” or “now”. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [4] We include a few examples from various papers in Appendix C.

Understanding Related Research on Tool-Augmented Learning

Going from a mediocre team to an elite team can seem daunting, especially when we need to figure out where to start. The DevOps Research and Assessment (DORA) can be our north star, helping us build the habits and practices we need to be elite. We'll look at DORA, the challenges of building habits, and how to form habits that stick. What Is DORA? The DevOps Research and Assessment (DORA) report focuses on four key areas to measure elite teams: Lead time: How long does it take from code committed to running in production
Deploy frequency: How often does our organization deploy to production
Change fail percentage: How often do we need to rollback, patch, hotfix, or fix forward after a change
Failed deployment recovery time: How long does it generally take to recover service The challenge here is going from knowing about those four areas that need to be worked on so that we go from being a mediocre team to an elite squad scoring a ten on the DORA quick check. The four measurements of an elite team are: Lead time is less than one hour


Deploy frequency is on-demand (multiple deploys per day)


The change fail percentage is ideally 0%


Failed deployment recovery time is less than one hour To see where tour team currently places, try the DORA quick check. This blog will focus on the tools, technologies, practices, culture, and philosophies that will help teams become elite. Improvements in our work take time since we need to break old habits that don't serve us well and build new ones. This blog will keep that in mind as we move through different topics. Each post will have an invitation to try out what we learn. If there are any questions or issues that need to be covered, make use of the comments section to submit the question. The Challenges of Forming New Habits We've all seen New Year's Resolutions and possibly have had some of our own. According to the Forbes report in 2024, 53% of those surveyed had given up on their New Year's Resolutions by the end of March. For many people, sticking to a new resolution or goal is challenging. There is now a day known as Ditch New Year's Resolution Day, which during 2024 fell on January 17th. While the media paints a picture of a significant stumbling block, statistics show it's less troublesome than perceived. We can see the reality of the quit rate in the chart below: Source Why Do People Fail in Their New Resolutions/Habits? In an article by James Clear, he outlines five reasons our new habits might fail: Trying to change more than one thing at a time


Setting big goals without breaking them down into smaller steps


Seeking a result, not the ritual


We have not changed the environment in which we perform our good habits, so we quickly revert to our bad habits


Assuming small changes don't add up However, there are ways to set ourselves up for success. In the book Atomic Habits, James talks about three key lessons: Small habits make a big difference.
Forget about setting goals. Instead, focus on the system.
Build identity-based habits. The Impact of Habits Building identity-based habits means, for example, instead of saying I go running, we say we're runners, or instead of saying we paint, we say we're painters. There is a subtle difference in the wording, but if we start to think we're runners, that can lead to thinking, what does a runner do? It could mean changing the food we eat, how often we train, when we train, and the goals we set for ourselves. If we focus on getting 1% better every day in one year, we'll end up being 37 times better than when we started. It means that anything we set our mind to can profoundly impact us. It may not be noticeable at first, but as the changes compound over time, we will notice that we're a different person than when we started. Like a cliff being eroded over time by the waves, at the start, we overlook the impact each wave has, but over time, we see how the cliff face changes in its shape. How Do I Go About Creating and Maintaining Habits? In the Habits Cheatsheet, James Clear talks about the four laws to create a good habit: The 1st law (Cue): Make it obvious
The 2nd law (Craving): Make it attractive
The 3rd law (Response): Make it easy
The 4th law (Reward): Make it satisfying As an example, let's say we've decided that we want to drink more water every day. We could make it obvious by placing a water bottle on our desk so that we can't miss it. Having a nice bottle or putting some lemon or mint in the water would be attractive. To make it easy, we fill the water bottle the night before so we don't have to think about it in the morning. Then, to make it satisfying, we could have a calendar above our desk where we put a sticker to mark each day that we've managed to empty our water bottle. How Do I Beat My Old Bad Habits? Again, in the Habits Cheatsheet, there are the inverse laws for avoiding our bad habits: Inversion of the 1st law (Cue): Make it invisible
Inversion of the 2nd law (Craving): Make it unattractive
Inversion of the 3rd law (Response): Make it difficult
Inversion of the 4th law (Reward): Make it unsatisfying Let's say we want to watch less TV. We could make it invisible by placing the TV out of view, moving it to another room, or inside a cupboard. We might unsubscribe to our streaming accounts and TV subscriptions to make it unattractive. To make it difficult, we place the TV's power cord or remote control in another room. Finally, to make it unsatisfying, we could have an accountability partner that we check in with, with whom we will have to confess that we ended up watching TV. The Next Steps As we reflect on our DORA scores, we have an opportunity to identify areas where we can improve by forming new habits. The DORA quick check provides guidance on key areas we can work on to improve our score. In our team, we can decide what to work on to raise our score, moving us from developers to software craftspeople. Then, we can form habits to help us make lasting changes. For example, if we identify a need to reduce lead time, we may realize our review process is slowing us down. When we explore it, we recognize we need to set aside time to review the code. We could take a habit-stacking approach. For example, when we open our laptops in the morning (our current habit), we first review all of the pull requests that are pending review for us. Another could be first to review pull requests after our lunch break. To make it attractive, we might realize that the pull requests lack enough information, so we add a pull request template so all the information is there. Then, to make it easy, we could have a simple way for people to launch the application and see the change. To make it satisfying, we could recognize a person each week who does a great job of meaningful reviews and helps speed up the delivery of features. Conclusion We've explored the DORA report and how it measures teams' ability to deliver software. We've examined how habits are the foundation of meeting our goals/resolutions, looking at good and bad habits. Our challenge is to move from software developers to software craftspeople, using our daily habits to help us transform. References State of DevOps Report: https://cloud.google.com/devops/state-of-devops
DORA quick check assessment: https://dora.dev/quickcheck/
Forbes article on New Year's Resolutions Statistics: https://www.forbes.com/health/mind/new-years-resolutions-statistics/
5 Common Mistakes That Cause New Habits to Fail: https://jamesclear.com/habits-fail
Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits
Summary of the Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits-summary
Continuous Improvement - How It Works and How to Master It: https://jamesclear.com/continuous-improvement
Habits cheatsheets: https://jamesclear.com/atomic-habits/cheatsheet
Habit stacking: https://jamesclear.com/habit-stacking The lead image was generated using Microsoft Designer. Going from a mediocre team to an elite team can seem daunting, especially when we need to figure out where to start. The DevOps Research and Assessment (DORA) can be our north star, helping us build the habits and practices we need to be elite. We'll look at DORA, the challenges of building habits, and how to form habits that stick. What Is DORA? The DevOps Research and Assessment (DORA) report focuses on four key areas to measure elite teams: report report Lead time: How long does it take from code committed to running in production Deploy frequency: How often does our organization deploy to production Change fail percentage: How often do we need to rollback, patch, hotfix, or fix forward after a change Failed deployment recovery time: How long does it generally take to recover service Lead time: How long does it take from code committed to running in production Lead time: Deploy frequency: How often does our organization deploy to production Deploy frequency: Change fail percentage: How often do we need to rollback, patch, hotfix, or fix forward after a change Change fail percentage: Failed deployment recovery time: How long does it generally take to recover service Failed deployment recovery time: The challenge here is going from knowing about those four areas that need to be worked on so that we go from being a mediocre team to an elite squad scoring a ten on the DORA quick check. The four measurements of an elite team are: The four measurements of an elite team are: Lead time is less than one hour Deploy frequency is on-demand (multiple deploys per day) The change fail percentage is ideally 0% Failed deployment recovery time is less than one hour Lead time is less than one hour Lead time is less than one hour Deploy frequency is on-demand (multiple deploys per day) Deploy frequency is on-demand (multiple deploys per day) The change fail percentage is ideally 0% The change fail percentage is ideally 0% Failed deployment recovery time is less than one hour Failed deployment recovery time is less than one hour To see where tour team currently places, try the DORA quick check . DORA quick check DORA quick check This blog will focus on the tools, technologies, practices, culture, and philosophies that will help teams become elite. Improvements in our work take time since we need to break old habits that don't serve us well and build new ones. This blog will keep that in mind as we move through different topics. Each post will have an invitation to try out what we learn. If there are any questions or issues that need to be covered, make use of the comments section to submit the question. The Challenges of Forming New Habits We've all seen New Year's Resolutions and possibly have had some of our own. According to the Forbes report in 2024, 53% of those surveyed had given up on their New Year's Resolutions by the end of March. For many people, sticking to a new resolution or goal is challenging. There is now a day known as Ditch New Year's Resolution Day, which during 2024 fell on January 17th. While the media paints a picture of a significant stumbling block, statistics show it's less troublesome than perceived. We can see the reality of the quit rate in the chart below: Source Source Why Do People Fail in Their New Resolutions/Habits? In an article by James Clear, he outlines five reasons our new habits might fail: article article Trying to change more than one thing at a time Setting big goals without breaking them down into smaller steps Seeking a result, not the ritual We have not changed the environment in which we perform our good habits, so we quickly revert to our bad habits Assuming small changes don't add up Trying to change more than one thing at a time Trying to change more than one thing at a time Setting big goals without breaking them down into smaller steps Setting big goals without breaking them down into smaller steps Seeking a result, not the ritual Seeking a result, not the ritual We have not changed the environment in which we perform our good habits, so we quickly revert to our bad habits We have not changed the environment in which we perform our good habits, so we quickly revert to our bad habits Assuming small changes don't add up Assuming small changes don't add up However, there are ways to set ourselves up for success. In the book Atomic Habits , James talks about three key lessons : Atomic Habits three key lessons three key lessons Small habits make a big difference. Forget about setting goals. Instead, focus on the system. Build identity-based habits. Small habits make a big difference. Forget about setting goals. Instead, focus on the system. Build identity-based habits. The Impact of Habits Building identity-based habits means, for example, instead of saying I go running, we say we're runners, or instead of saying we paint, we say we're painters. There is a subtle difference in the wording, but if we start to think we're runners, that can lead to thinking, what does a runner do? It could mean changing the food we eat, how often we train, when we train, and the goals we set for ourselves. If we focus on getting 1% better every day in one year, we'll end up being 37 times better than when we started. It means that anything we set our mind to can profoundly impact us. It may not be noticeable at first, but as the changes compound over time, we will notice that we're a different person than when we started. Like a cliff being eroded over time by the waves, at the start, we overlook the impact each wave has, but over time, we see how the cliff face changes in its shape. How Do I Go About Creating and Maintaining Habits? In the Habits Cheatsheet , James Clear talks about the four laws to create a good habit: Habits Cheatsheet The 1st law (Cue): Make it obvious The 2nd law (Craving): Make it attractive The 3rd law (Response): Make it easy The 4th law (Reward): Make it satisfying The 1st law ( Cue ): Make it obvious Cue The 2nd law ( Craving ): Make it attractive Craving The 3rd law ( Response ): Make it easy Response The 4th law ( Reward ): Make it satisfying Reward As an example, let's say we've decided that we want to drink more water every day. We could make it obvious by placing a water bottle on our desk so that we can't miss it. Having a nice bottle or putting some lemon or mint in the water would be attractive. To make it easy, we fill the water bottle the night before so we don't have to think about it in the morning. Then, to make it satisfying, we could have a calendar above our desk where we put a sticker to mark each day that we've managed to empty our water bottle. How Do I Beat My Old Bad Habits? Again, in the Habits Cheatsheet, there are the inverse laws for avoiding our bad habits: Inversion of the 1st law (Cue): Make it invisible Inversion of the 2nd law (Craving): Make it unattractive Inversion of the 3rd law (Response): Make it difficult Inversion of the 4th law (Reward): Make it unsatisfying Inversion of the 1st law ( Cue ): Make it invisible Cue Inversion of the 2nd law ( Craving ): Make it unattractive Craving Inversion of the 3rd law ( Response ): Make it difficult Response Inversion of the 4th law ( Reward ): Make it unsatisfying Reward Let's say we want to watch less TV. We could make it invisible by placing the TV out of view, moving it to another room, or inside a cupboard. We might unsubscribe to our streaming accounts and TV subscriptions to make it unattractive. To make it difficult, we place the TV's power cord or remote control in another room. Finally, to make it unsatisfying, we could have an accountability partner that we check in with, with whom we will have to confess that we ended up watching TV. The Next Steps As we reflect on our DORA scores, we have an opportunity to identify areas where we can improve by forming new habits. The DORA quick check provides guidance on key areas we can work on to improve our score. In our team, we can decide what to work on to raise our score, moving us from developers to software craftspeople. Then, we can form habits to help us make lasting changes. For example, if we identify a need to reduce lead time, we may realize our review process is slowing us down. When we explore it, we recognize we need to set aside time to review the code. We could take a habit-stacking approach. For example, when we open our laptops in the morning (our current habit), we first review all of the pull requests that are pending review for us. Another could be first to review pull requests after our lunch break. habit-stacking pull requests To make it attractive, we might realize that the pull requests lack enough information, so we add a pull request template so all the information is there. Then, to make it easy, we could have a simple way for people to launch the application and see the change. To make it satisfying, we could recognize a person each week who does a great job of meaningful reviews and helps speed up the delivery of features. Conclusion We've explored the DORA report and how it measures teams' ability to deliver software. We've examined how habits are the foundation of meeting our goals/resolutions, looking at good and bad habits. Our challenge is to move from software developers to software craftspeople, using our daily habits to help us transform. References State of DevOps Report: https://cloud.google.com/devops/state-of-devops DORA quick check assessment: https://dora.dev/quickcheck/ Forbes article on New Year's Resolutions Statistics: https://www.forbes.com/health/mind/new-years-resolutions-statistics/ 5 Common Mistakes That Cause New Habits to Fail: https://jamesclear.com/habits-fail Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits Summary of the Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits-summary Continuous Improvement - How It Works and How to Master It: https://jamesclear.com/continuous-improvement Habits cheatsheets: https://jamesclear.com/atomic-habits/cheatsheet Habit stacking: https://jamesclear.com/habit-stacking State of DevOps Report: https://cloud.google.com/devops/state-of-devops https://cloud.google.com/devops/state-of-devops https://cloud.google.com/devops/state-of-devops DORA quick check assessment: https://dora.dev/quickcheck/ https://dora.dev/quickcheck/ https://dora.dev/quickcheck/ Forbes article on New Year's Resolutions Statistics: https://www.forbes.com/health/mind/new-years-resolutions-statistics/ https://www.forbes.com/health/mind/new-years-resolutions-statistics/ https://www.forbes.com/health/mind/new-years-resolutions-statistics/ 5 Common Mistakes That Cause New Habits to Fail: https://jamesclear.com/habits-fail https://jamesclear.com/habits-fail https://jamesclear.com/habits-fail Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits https://jamesclear.com/atomic-habits https://jamesclear.com/atomic-habits Summary of the Atomic Habits book by James Clear: https://jamesclear.com/atomic-habits-summary https://jamesclear.com/atomic-habits-summary https://jamesclear.com/atomic-habits-summary Continuous Improvement - How It Works and How to Master It: https://jamesclear.com/continuous-improvement https://jamesclear.com/continuous-improvement https://jamesclear.com/continuous-improvement Habits cheatsheets: https://jamesclear.com/atomic-habits/cheatsheet https://jamesclear.com/atomic-habits/cheatsheet https://jamesclear.com/atomic-habits/cheatsheet Habit stacking: https://jamesclear.com/habit-stacking https://jamesclear.com/habit-stacking https://jamesclear.com/habit-stacking The lead image was generated using Microsoft Designer. The lead image was generated using Microsoft Designer . Microsoft Designer Microsoft Designer

Elite Teams Are Built on Strong Habits

Understanding Barrier-to-Exit in Amazon's Recommender System

অ্যামাজনের সুপারিশকারী সিস্টেমে বাধা-থেকে-প্রস্থান বোঝা

Verständnis der Ausstiegsbarrieren im Empfehlungssystem von Amazon

Comprender la barrera de salida en el sistema de recomendación de Amazon

Comprendre les barrières à la sortie dans le système de recommandation d'Amazon

अमेज़न की अनुशंसा प्रणाली में बैरियर-टू-एग्जिट को समझना

Amazon のレコメンデーション システムにおける退出障壁を理解する

Amazon 추천 시스템의 종료 장벽 이해

Compreendendo a barreira de saída no sistema de recomendação da Amazon

Понимание барьера выхода в рекомендательной системе Amazon

Amazon'un Tavsiye Sisteminde Çıkış Engelini Anlamak

Tìm hiểu về Rào cản thoát trong Hệ thống đề xuất của Amazon

了解亚马逊推荐系统的退出壁垒

220 Stories To Learn About Ux Design

Stories, News & Mentions about STARLEARN

Starlearn’s Stories on HackerNoon