However, the risks and concerns have often been downplayed, with a lack of authenticity from some when it comes to the challenges around a lack of regulation, data protection, models hallucinating and being unable to ‘unlearn errors’, as well as a certain amount of trivializing when it comes to happen to the skills base in the market.

PagerDuty CTO - ‘Generative AI focus should be about minimizing collateral damage’

C athie Wood gained fame as an investor for her unwavering belief in companies at the forefront of technological innovation. Her ARK Invest focuses on finding disruptive technologies that can change the world, such as artificial intelligence (AI).

3 Cathie Wood Tech Stocks to Buy Hand Over Fist

UiPath, Unity, and PagerDuty aren't household names, but each company's mastery of its respective tech specialty helps win customers and grow revenue. As a result, these three growth stocks have earned a spot in Cathie Wood's ARK Invest funds and can generate attractive returns for you as well.

Detailed price information for Uipath Inc Cl A (PATH-N) from The Globe and Mail including charting and trades.

PagerDuty focuses on IT infrastructure monitoring to minimize disruption of an organization's crucial systems. Its platform uses AI to identify when various issues are related to the same root cause,

Detailed price information for Unity Software Inc (U-N) from The Globe and Mail including charting and trades.

Grab our FREE report, featuring an exclusive CEO interview. Ready for the name and trading symbol? Pagerduty Inc (NYSE: PD)’s stock price has increased by 4.87 compared to its previous closing price of 19.72. However, the company has seen a -1.01% ...

Pagerduty Inc (PD) Stock: A Look at the Analyst Recommendations

The Chaos Engineering Tools Market size is expected to grow from USD 1.9 billion in 2023 to USD 2.9 billion by 2028 at a Compound Annual Growth Rate (CAGR) of 8.8% during the forecast period, according to a new report by MarketsandMarkets™.

Chaos Engineering Tools Market worth $2.9 billion by 2028 - Exclusive Report by MarketsandMarkets™

As organisations prioritise system reliability and resilience in an ever-evolving digital landscape, the future of the Chaos Engineering

Shares of IT incident response platform PagerDuty (NYSE:PD) fell 9.48% in the pre-market session after it announced plans to raise $350 million in convertible senior notes via a private placement to qualified institutional buyers. PagerDuty intends to use ...

Why PagerDuty (PD) Stock Is Falling Today

PagerDuty

pagerduty

How we replaced manual review of PagerDuty alerts with an automatic weekly summary dashboard. I’m a software engineer at on the & Infrastructure team. We support a myriad of services for both external clients and internal employees: we enable the product, growth, and teams with tools like Redshift for large analytics queries and the for real-time investigation across the many services powering Plaid. Plaid Data Science engineering ELK stack Supporting these analytics & monitoring services is mission-critical: if they go down dozens of workflows across the company break; it’s important that we catch any degradation of service before it becomes it an issue. Over the course of 2018 we’ve added dozens of new features to the services we support: new datasets, faster analytics capabilities, and more self-service data portals. We need to find any issues before they are realized by our users: We use to send real-time alerts when there is a degradation in service performance. Our team has an on-call rotation: between four engineers we rotate being the directly responsible individual for handling any alerts of service performance generated through PagerDuty. The week you’re on call, you use the PagerDuty app to alert of any realtime alerts, and the PagerDuty slack integration to send other messages to the slack channel. PagerDuty DSI-Urgent DSI-Notify PagerDuty slackbot — friend to all. Towards the end of the year we realized that while we had done well to add new PagerDuty alerts to support SLAs for our services, we weren’t as disciplined with triaging and fixing the underlying long-term issues. In order to step back from treating the symptoms and work towards overall service quality, we decided to add an “on-call summary & handoff” digest to our weekly team meetings. This helped facilitate discussion about why we were being paged, iterate on a service to improve reliability, and change alert structure to reduce false alarms. Trialing a formalized handoff document, with categories of alerts to facilitate trend analysis and some comments on specific alerts for specific details. This worked well as a pilot — we were able to better surface common issues, more easily identify patterns, and decouple the on-call cadence of addressing real-time issues with the need for making longer term substantive changes. It was not, however, our long-term vision to add manual work to the on-call rotation. PagerDuty offers a few views into analytics, but we wanted finer grain control of our data and to be able to join it to other analytics data sources. We set out to use the to pull the data that would help inform patterns of incidents and supplant the manual tracking of our handoff document. PagerDuty V2 API We use for a wide-range of recurring data ETL, and wrote a one-off Airflow DAG that downloads a history of the data from PagerDuty, writes it into S3, and then copies from S3 to Redshift. You can see a sanitized version of the python script in . Apache Airflow this gist Airflow DAG for ingesting PagerDuty data We focused on two tables: and , which gives us a history of which team got notified about what alert on what date. PagerDuty offers a one month history of incidents through the API. Because a user can modify an incident well after the original date of the alert, we re-harvest the entire 30-day incident history and update any modified data. By contrast, log_entries is an append-only log, so we more simply take new data at a regular cadence and copy it into redshift. incidents log_entries Finally, with the data in Redshift we can create an automated version of our handoff document! Here’s an example of how we query the final schema to pull insights out of the incident and notification history: -- categorize alerts, and count by category + date-- takes a rolling five day historyselectdate_trunc('day', (created_at)::timestamp)::date, casewhen incidents.summary like '%airflow heartbeat%'then 'Airflow Heartbeat'when incidents.summary like '%mysql_dump%'then 'MysqlDump'when incidents.summary like '%TaskInstance: dump_%'then 'MongoDump'when incidents.summary like '%ES%not green%'then 'ES not green'when incidents.summary like '%queue is building%'then 'Logstash Queue'when incidents.summary like '%kinesis%behind%'then 'Kinesis Behind'when incidents.summary like '%TaskInstance%'then 'Airflow - Other'else-- remove numbers to more easily group alertsregexp_replace(incidents.summary, '[0-9]', '')end as summary, count(1)frompagerduty.incidentswhereabs(datediff('day', getdate(), created_at)) < 5and (escalation_policy_summary='DW-DSI')group by1, 2order by3 desc Which leads us to the grand conclusion: a Periscope dashboard where we can look at the number of alerts, the breakdown by category and service, and track trends over time. The final dashboard in Periscope, summarizing alerts over the past 7 days. Our handoff doc is a little cleaner now: rather than having to manually type out reports about individual alerts, we can summarize the week from our dashboard and jump straight to the collective and individual actions we want to take to build resilient and reliable data infrastructure. New handoff document, which focuses on concrete Action Items This new process has helped us prioritize & fix noisy alerts and persistent incidents. As a bonus, it helps our users surface and escalate lingering issues that have been masked by the cadence on on-call alerts, focusing on other important issues: The good case: our internal users attacking the real issues.

PagerDuty Analytics with Python & Periscope

We’re excited to announce that Bugsnag now supports error monitoring and stability management for Unreal Engine mobile games. Unreal Engine Unreal Engine is a game engine developed by Epic Games, first showcased in the 1998 first-person shooter game Unreal. Initially developed for PC first-person shooters, Unreal Engine has since been used in a variety of genres of three-dimensional (3D) games. It has also been widely adopted by the film and television industry. Written in C++, the Unreal Engine features a high degree of portability and supports a range of desktop, mobile, console, and virtual reality platforms. Bugsnag already provides stability monitoring to help engineering teams with a range of platforms and frameworks. mobile and desktop We're excited about extending our support for mobile platforms to include Unreal Engine. Developers will now have the actionable insights and analytics they need to increase the stability of their games and deliver a streamlined experience to their fans. Manage the Stability of Unreal Engine Games With Bugsnag Gain real-time visibility and actionable insights into stability. Development teams are releasing new versions and game modes more and more frequently to engage and retain users. With Bugsnag, you can understand the stability of every release and see how rapidly new releases are being adopted by users. You can compare release health to decide if they’re healthy enough to promote to production, or if they need to be pulled back. Setting your allows you to use the real-time stability scores to make data-driven decisions about when to build features in your application vs. focus on bug fixing. target stability and critical stability Essentially, if your game’s stability is below critical stability, the development team needs to focus on bug fixing and if it’s above target stability, they can focus on building new features. Additionally, the helps you understand stability issues over time. You can pinpoint error spikes, investigate specific incidents, determine if a release has introduced new bugs, and more. interactive timeline analysis Remember: Not all bugs are worth fixing. Prioritize fixing the bugs that matter. Understand which bugs are occurring more frequently and impacting the most users. View errors grouped by root cause and sorted by user impact so you can make data-driven decisions about which errors need to be addressed first. Prioritize Fixing the Bugs That Matter Not all bugs are created equal and you may want to prioritize fixing errors that are impacting . superfans or your revenue generating customers You can easily choose which meta data to collect for each error, such as the user’s spending tier, and then that are affecting paying gamers. segment errors Errors impacting business-critical app functions, such as the user sign-up or in-app purchase flows, are also more important to address quickly to minimize further impact to your businesses’ bottom line. Using error segmentation in Bugsnag, you can see which errors are impacting a certain part of the codebase, such as the one supporting game token purchases, so you can prioritize fixing them right away. Customize Error Views and Alerts Using bookmarks, which are a set of saved error segmentation filters, error views can be configured in the for each team member working on building and maintaining the game. Project Overview Dashboard An error view includes a feed of errors and a trend graph to show how errors are impacting the stability of the segment. You can set up error views for VIP users or superfans, in-app purchase flows, and even ad networks. Gaining visibility into errors originating from ad SDKs can help you decide if you need to turn off an erratic SDK that is negatively impacting the gaming experience. You can use the same bookmarks for configuring error alerts, so that you are only notified of errors occurring in segments that matter to you. This way you can improve the signal-to-noise ratio and focus the team on addressing errors that will have the greatest impact on improving app stability and user experience. You can: Fix errors with speed and precision to minimize further impact to user experience. Get to the root cause of errors quickly with comprehensive diagnostics. Comprehensive Diagnostics and Error Reproduction Reproducing and fixing errors is easier when you can . understand exactly what caused the crash See comprehensive diagnostic information for every error, including , user interaction breadcrumbs, device and user information. symbolicated stack traces Error reports also include Unreal-specific metadata, such as the MapUrl of the last loaded level, any custom User Activity you set within the app code, and the name of the current GameState class. You can also see changes to these values and the events that occurred in the run-up to the error as breadcrumbs. Integrations Streamline debugging workflows with powerful integrations. There are pre-built with the tools your team already uses, like Slack, PagerDuty and with Jira, Pivotal, and more. integrations 2-way issue tracker integration Automation can help streamline your debugging workflows and so you can quickly roll out fixes and minimize further impact to user experience. drive code ownership Get Started With Bugsnag Bugsnag’s new notifier library for Unreal Engine initially supports Android and iOS games. Simply create a new Unreal Engine project and follow the quick start guide in to start monitoring your application. our documentation If you’re new to Bugsnag, start a or to see the capabilities of our in action. 14-day free trial request a demo stability management platform

As a Gaming Developer, Which Error-Monitoring System Should I Use? 

The world is a much more connected place than it was 20 years ago. With the change came new expectations. Downtimes can cripple businesses. There is an expectation of privacy and security. If we post on Facebook and the post gets lost due to any kind of issue, people will not tolerate it. Data integrity and durability are more critical than ever. As an organization, we must consider multiple technological tenets—security, availability, latency, durability, integrity, reliability, and privacy. We don’t get to choose from this list; we just sort it by importance based on business needs. Business continuity Disaster can come in many flavors, ranging from natural disasters to hardware failures, misconfiguration, and cyberattacks. The problem with cyberattacks is that we risk having our data stolen and wonder if the rest of the data is trustworthy. Maybe the attack was a sleight of hand, and the primary objective was to introduce malicious data that would haunt the organization forever. After the security incident, we need to recover the data and restore the service to the level our customers expect. One of the issues we face is reconstructing the original picture and configuration. We need to understand how all the pieces work together and their interdependency. Data restoration is more than just restoring a highly scalable web service snapshot. Objective At this point, our primary objective should be to ensure that our services are available and reliable by whatever means necessary. In the context of dependency management, we need to have a comprehensive data and dependency catalog, including the relationships between the entities. This provides us with the necessary insight into what exists. We also need regular operations testing and robust disaster recovery through automation and manual runbooks. We need to cultivate a culture of disaster recovery fire drills and identify fallbacks. Maturity Model Like an individual who enters college goes through freshman, sophomore, junior, and senior years before graduating, businesses need to follow a slow, progressive approach. It’s impractical for a company or organization to go from no disaster recovery to a fully-fledged, perfectly fleshed-out plan. The road to success is incremental, and each step builds on the previous step. Maturity models examine an organization’s current state from any specific perspective, such as disaster recovery, and help the organization reach the desired state or capability. As the organization progresses toward the end goal, it goes through various checkpoints, usually called Levels. Most Maturity Models have four levels, starting from Level 1 up to Level 4. Why are we talking about maturity models? One step in such a disaster recovery maturity model requires that the organization understand dependencies among various entities and data objects. Recovery usually requires restoring the data and bringing back critical systems in a specific incremental manner, Implementation Based on the business needs, understanding dependency and its relationship can be somewhere in the middle levels of the maturity models. In a 4-level system, it is most likely in Level 2 or Level 3. The other approach is to split the Dependency Management into the same number of milestones as the number of levels. Suppose we are using a 4-level maturity model. In that case, we can split up dependency management to span all 4 Levels, starting with the most straightforward steps and going all the way to the most complicated, expensive, and involved steps. Going forward, we will only discuss the dependency management aspect of this maturity model. Level 1 - Discovering and Understanding Dependencies For many organizations, this can be the most daunting task. If dependency management was never considered necessary, there might be no institutional knowledge. We will start by discussing technical dependencies and then discuss process dependencies. Regarding technical dependencies, we need to identify the direct dependencies first. If the code is written in JavaScript and runs on a cloud service, then the direct dependencies might be JavaScript, NodeJS, AWS Lambda, etc. If we use npm for package management, the packages listed in the dependency file are direct dependencies. Next, we have transitive dependencies, which are pulled automatically from direct dependencies. In practice, this forms a directed acyclic graph. Another aspect of technical dependencies is the cloud architecture. Let’s take an example of AWS—we might use EC2, S3, API Gateway, CloudFront, ELB, etc. If the cloud resources were deployed using tools like CloudFormation or CDK, then it would be trivial to determine the dependencies. If the resources were created manually, the dependency diagram must be made manually by reading the source code and drawing conclusions. Process dependency falls under the non-technical category and defines how different stakeholders work together to move forward. It can also outline the release process or runbooks to invoke during an incident. Level 2 - Setup Continuous Dependencies Validation Dependencies can change since there are too many cooks in the kitchen. The organization's continuous integration tool should enumerate and track the direct code and library dependencies. Every time the code is pushed, a snapshot of current pinned dependencies should be saved to recreate the specific circumstances or restore to a known good state. Like the Continuous integration concept, the non-technical aspects should have some automation. Alerts should be created on metric, and using tools like PagerDuty, appropriate people should be notified of issues, and proper escalation channels should be created. It should be tested occasionally to avoid the known problem called Dogs Not Barking - a situation when you wonder if you are not getting any alerts because nothing is wrong or the alerting logic is broken. Some dependencies can be validated by using the Scream test. Under proper notification and preparation, turn off specific dependencies and see what fails. You may find a Load Bearing Mac Mini. Collect all of this configuration and manage it through some kind of provisioning tool like Chef and Terraform Level 3 - Fixing dependencies by adding, updating, or removing Once we have all the dependencies figured out, take some time and pat yourself on the back. Now, back on the issue, the work isn’t done. During the quest, you realize that some dependencies are causing more problems than they solve. One approach would be to update the dependencies. For example, if we are using old versions of OpenSSL, updating them to newer versions can solve many of the warnings raised by the build system and even help us pass any security compliance. Sometimes, the specific versions of two separate packages are incompatible. The solution is to find versions that can work together. This involves changing the version and might involve downgrading one or more packages. Sometimes, just removing a package or dependency solves the problem. It could have been a dead dependency with no further need but causing warnings and spamming the application logs. Under some circumstances, we might have to add a dependency to bring harmony. Suppose a transitive dependency is causing the problem, but we cannot update the direct dependency. In that case, explicitly adding the fixed version of the transitive dependency as a direct dependency can be an acceptable solution. Level 4 - Manual and Automated Disaster Recovery Drills This is the highest form of maturity in disaster recovery. We need runbooks that outline the roles and responsibilities of individuals involved. The runbook should also define the interdependencies and criticality of each step. The disaster recovery process should be practiced regularly by simulating a disaster and identifying possible sources of friction that get introduced when the plan is set in motion. The gold standard should be automated disaster recovery under human supervision. Automating the disaster recovery workflow ensures that humans do not make mistakes that go unnoticed. Automating has an additional advantage in that we can now run stress tests and simulation disasters by turning off certain network parts in a controlled manner to validate whether our automated disaster recovery is performing as expected. Conclusion Dependency management plays a critical role in disaster recovery, which is essential after a security incident. By following a structured and methodological approach, organizations can build their understanding progressively and control the dependencies. Each maturity model level builds on the previous one and strengthens the organization's security posture. While it can be easy to write off dependency management as a superfluous technical exercise, it is primarily about creating a comprehensive framework that brings together computer systems, people, and processes to work harmoniously. The systematic approach ensures that when disaster strikes, even in the form of a security breach, we can safely recover the data and be confident in the integrity of the said data. The world is a much more connected place than it was 20 years ago. With the change came new expectations. Downtimes can cripple businesses. There is an expectation of privacy and security. If we post on Facebook and the post gets lost due to any kind of issue, people will not tolerate it. Data integrity and durability are more critical than ever. As an organization, we must consider multiple technological tenets—security, availability, latency, durability, integrity, reliability, and privacy. We don’t get to choose from this list; we just sort it by importance based on business needs. Business continuity Disaster can come in many flavors, ranging from natural disasters to hardware failures, misconfiguration, and cyberattacks. The problem with cyberattacks is that we risk having our data stolen and wonder if the rest of the data is trustworthy. Maybe the attack was a sleight of hand, and the primary objective was to introduce malicious data that would haunt the organization forever. After the security incident, we need to recover the data and restore the service to the level our customers expect. One of the issues we face is reconstructing the original picture and configuration. We need to understand how all the pieces work together and their interdependency. Data restoration is more than just restoring a highly scalable web service snapshot. Objective At this point, our primary objective should be to ensure that our services are available and reliable by whatever means necessary. In the context of dependency management, we need to have a comprehensive data and dependency catalog, including the relationships between the entities. This provides us with the necessary insight into what exists. We also need regular operations testing and robust disaster recovery through automation and manual runbooks. We need to cultivate a culture of disaster recovery fire drills and identify fallbacks. Maturity Model Like an individual who enters college goes through freshman, sophomore, junior, and senior years before graduating, businesses need to follow a slow, progressive approach. It’s impractical for a company or organization to go from no disaster recovery to a fully-fledged, perfectly fleshed-out plan. The road to success is incremental, and each step builds on the previous step. Maturity models examine an organization’s current state from any specific perspective, such as disaster recovery, and help the organization reach the desired state or capability. As the organization progresses toward the end goal, it goes through various checkpoints, usually called Levels. Most Maturity Models have four levels, starting from Level 1 up to Level 4. Why are we talking about maturity models? One step in such a disaster recovery maturity model requires that the organization understand dependencies among various entities and data objects. Recovery usually requires restoring the data and bringing back critical systems in a specific incremental manner, Implementation Based on the business needs, understanding dependency and its relationship can be somewhere in the middle levels of the maturity models. In a 4-level system, it is most likely in Level 2 or Level 3. The other approach is to split the Dependency Management into the same number of milestones as the number of levels. Suppose we are using a 4-level maturity model. In that case, we can split up dependency management to span all 4 Levels, starting with the most straightforward steps and going all the way to the most complicated, expensive, and involved steps. Going forward, we will only discuss the dependency management aspect of this maturity model. Level 1 - Discovering and Understanding Dependencies For many organizations, this can be the most daunting task. If dependency management was never considered necessary, there might be no institutional knowledge. We will start by discussing technical dependencies and then discuss process dependencies. Regarding technical dependencies, we need to identify the direct dependencies first. If the code is written in JavaScript and runs on a cloud service, then the direct dependencies might be JavaScript, NodeJS, AWS Lambda, etc. If we use npm for package management, the packages listed in the dependency file are direct dependencies. Next, we have transitive dependencies, which are pulled automatically from direct dependencies. In practice, this forms a directed acyclic graph. Another aspect of technical dependencies is the cloud architecture. Let’s take an example of AWS—we might use EC2, S3, API Gateway, CloudFront, ELB, etc. If the cloud resources were deployed using tools like CloudFormation or CDK, then it would be trivial to determine the dependencies. If the resources were created manually, the dependency diagram must be made manually by reading the source code and drawing conclusions. Process dependency falls under the non-technical category and defines how different stakeholders work together to move forward. It can also outline the release process or runbooks to invoke during an incident. Level 2 - Setup Continuous Dependencies Validation Dependencies can change since there are too many cooks in the kitchen. The organization's continuous integration tool should enumerate and track the direct code and library dependencies. Every time the code is pushed, a snapshot of current pinned dependencies should be saved to recreate the specific circumstances or restore to a known good state. Like the Continuous integration concept, the non-technical aspects should have some automation. Alerts should be created on metric, and using tools like PagerDuty, appropriate people should be notified of issues, and proper escalation channels should be created. It should be tested occasionally to avoid the known problem called Dogs Not Barking - a situation when you wonder if you are not getting any alerts because nothing is wrong or the alerting logic is broken. Some dependencies can be validated by using the Scream test . Under proper notification and preparation, turn off specific dependencies and see what fails. You may find a Load Bearing Mac Mini . Collect all of this configuration and manage it through some kind of provisioning tool like Chef and Terraform Scream test Scream test Load Bearing Mac Mini Load Bearing Mac Mini Level 3 - Fixing dependencies by adding, updating, or removing Once we have all the dependencies figured out, take some time and pat yourself on the back. Now, back on the issue, the work isn’t done. During the quest, you realize that some dependencies are causing more problems than they solve. One approach would be to update the dependencies. For example, if we are using old versions of OpenSSL, updating them to newer versions can solve many of the warnings raised by the build system and even help us pass any security compliance. Sometimes, the specific versions of two separate packages are incompatible. The solution is to find versions that can work together. This involves changing the version and might involve downgrading one or more packages. Sometimes, just removing a package or dependency solves the problem. It could have been a dead dependency with no further need but causing warnings and spamming the application logs. Under some circumstances, we might have to add a dependency to bring harmony. Suppose a transitive dependency is causing the problem, but we cannot update the direct dependency. In that case, explicitly adding the fixed version of the transitive dependency as a direct dependency can be an acceptable solution. Level 4 - Manual and Automated Disaster Recovery Drills This is the highest form of maturity in disaster recovery. We need runbooks that outline the roles and responsibilities of individuals involved. The runbook should also define the interdependencies and criticality of each step. The disaster recovery process should be practiced regularly by simulating a disaster and identifying possible sources of friction that get introduced when the plan is set in motion. The gold standard should be automated disaster recovery under human supervision. Automating the disaster recovery workflow ensures that humans do not make mistakes that go unnoticed. Automating has an additional advantage in that we can now run stress tests and simulation disasters by turning off certain network parts in a controlled manner to validate whether our automated disaster recovery is performing as expected. Conclusion Dependency management plays a critical role in disaster recovery, which is essential after a security incident. By following a structured and methodological approach, organizations can build their understanding progressively and control the dependencies. Each maturity model level builds on the previous one and strengthens the organization's security posture. While it can be easy to write off dependency management as a superfluous technical exercise, it is primarily about creating a comprehensive framework that brings together computer systems, people, and processes to work harmoniously. The systematic approach ensures that when disaster strikes, even in the form of a security breach, we can safely recover the data and be confident in the integrity of the said data.

Dependency Management is Critical for Disaster Recovery After a Security Incident

Top APM Tools: 2024 Edition

It's one o'clock in the morning. My phone starts going crazy. I groan and look over. Sure enough, it's another PagerDuty call. I am on-call this week, and it's the third time in as many days that I have been woken up for the same reason. Hundreds of our customers have complained in unison, for three days in a row, that they have purchased event tickets from us and arrived at the venue only to find that their tickets have already been redeemed. They are being denied entry, and they are understandably very angry. Initial Investigation Showed Some Very Odd Behavior. Customers had purchased tickets, and then a matter of minutes later, their tickets would be redeemed. This was far in advance of the event itself, so it was clearly unexpected. And it wasn't spread evenly across our customer base; the issue was heavily focused within some select groups of university students. We already knew that we had a fundamental issue with our redemption system; it was possible to scan the QR code in the ticket order confirmation email yourself like the door staff would do at the venue, which would redeem the ticket. Unfortunately, the bigger fix for this was still a few weeks away. Even so, why would large numbers of customers within specific university circles redeem or accidentally redeem their own tickets before the event?! None of our early theories made sense, and the issue kept happening. We Went Digging. Eventually, We Found the Issue. The automated virus scanner on specific university .edu email systems were hitting all links in our confirmation emails to check them for viruses, including programmatically scanning all QR codes. Therefore this one process, completely invisible to my team, was automatically redeeming our customers' event tickets before the customer even received the email. Never, Ever Update Persisted Data on Page-Load. By making the ticket QR code hit a webpage that immediately redeems the ticket on page-load, it meant that anything visiting that link, whether human or machine, would inadvertently update our database to mark the ticket as redeemed. There are thousands of unseen automated visitors to our websites that we don't think about most of the time. Here are some examples: Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com, and there are prominent links on that page to/blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Email scanners. Very common in schools, governments, large corporations, and more. Internet archive machines The simple lesson here is to avoid things happening automatically on page-loads as much as possible. Besides the issues that we experienced, unexpected side-effects like this are bad UX. If the action is "Redeem", make it happen when you click a button that says "Redeem". I hope that this was useful - subscribe now to get my Dev Diary in your inbox! It's one o'clock in the morning. My phone starts going crazy. I groan and look over. Sure enough, it's another PagerDuty call. I am on-call this week, and it's the third time in as many days that I have been woken up for the same reason. Hundreds of our customers have complained in unison, for three days in a row, that they have purchased event tickets from us and arrived at the venue only to find that their tickets have already been redeemed . They are being denied entry, and they are understandably very angry. tickets have already been redeemed very angry. Initial Investigation Showed Some Very Odd Behavior. Customers had purchased tickets, and then a matter of minutes later, their tickets would be redeemed. This was far in advance of the event itself, so it was clearly unexpected. And it wasn't spread evenly across our customer base; the issue was heavily focused within some select groups of university students. We already knew that we had a fundamental issue with our redemption system; it was possible to scan the QR code in the ticket order confirmation email yourself like the door staff would do at the venue, which would redeem the ticket. Unfortunately, the bigger fix for this was still a few weeks away. Even so, why would large numbers of customers within specific university circles redeem or accidentally redeem their own tickets before the event?! None of our early theories made sense, and the issue kept happening. We Went Digging. Eventually, We Found the Issue. The automated virus scanner on specific university .edu email systems were hitting all links in our confirmation emails to check them for viruses, including programmatically scanning all QR codes . automated virus scanner .edu programmatically scanning all QR codes Therefore this one process, completely invisible to my team, was automatically redeeming our customers' event tickets before the customer even received the email. before the customer even received the email. Never, Ever Update Persisted Data on Page-Load. By making the ticket QR code hit a webpage that immediately redeems the ticket on page-load, it meant that anything visiting that link, whether human or machine, would inadvertently update our database to mark the ticket as redeemed. anything There are thousands of unseen automated visitors to our websites that we don't think about most of the time. Here are some examples: Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com, and there are prominent links on that page to/blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com , and there are prominent links on that page to /blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Browsers prefetching resources. https://myblog.com /blog-post-1 /blog-post-2 just being on the homepage Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Google crawlers. Email scanners. Very common in schools, governments, large corporations, and more. Email scanners. Very common in schools, governments, large corporations, and more. Email scanners. Internet archive machines Internet archive machines Internet archive machines The simple lesson here is to avoid things happening automatically on page-loads as much as possible. Besides the issues that we experienced, unexpected side-effects like this are bad UX. If the action is "Redeem", make it happen when you click a button that says "Redeem". I hope that this was useful - subscribe now to get my Dev Diary in your inbox!

Dev Diary #3: The Sneaky Issue That Was Silently Ruining Our Database

Day 1 and Day 2 Operations for Platform Engineers

DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon.


Our product (firewall) will continue working; clients just won't be able to access their accounts.


We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data.
Make secondary backups of critical data in the cloud.
Backup your Kubernetes master.
Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations.
Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything The fire will be put out soon. The fire will be put out soon. The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything We have backups; we'll recover everything We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't store backups in the same data center as your data. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). jobs@wallarm.com

How We Migrated to the Cloud... of Ashes (Wallarm OVH Recovery in 2021)

A Dozen (or so) Learnings From 15 Years of Software Incident Management

সফ্টওয়্যার ঘটনা ব্যবস্থাপনার 15 বছর থেকে এক ডজন (বা তাই) শিক্ষা

Ein Dutzend (oder so) Erkenntnisse aus 15 Jahren Software-Vorfallmanagement

Una docena (más o menos) de aprendizajes de 15 años de gestión de incidentes de software

Une douzaine (ou deux) d'enseignements tirés de 15 ans de gestion des incidents logiciels

सॉफ्टवेयर घटना प्रबंधन के 15 वर्षों से एक दर्जन (या लगभग) सीखें

15 年間のソフトウェア インシデント管理から学んだ 12 の教訓

15년간의 소프트웨어 사고 관리 경험을 통해 얻은 12가지 정도의 교훈

Uma dúzia (ou mais) de aprendizados de 15 anos de gerenciamento de incidentes de software

Дюжина (или около того) уроков за 15 лет управления инцидентами с программным обеспечением

15 Yıllık Yazılım Olay Yönetiminden Bir Düzine (ya da O Kadar) Ders

Hàng chục (hoặc hơn) bài học từ 15 năm quản lý sự cố phần mềm

从 15 年的软件事故管理中学到的十几个经验

This Article Is Worth Your Attention if You Are passionate about writing good quality software and want to enhance the stability of your app with tests. Are tired of unexpected bugs popping up in your production systems. Need help understanding what automated tests are and how to approach them. Why Do We Need Automated Tests? As engineers, we want to , but with each new feature we create, we inevitably increase the size and complexity of our apps. build things that work As the product grows, it becomes more and more time-consuming to (e.g., with your hands) test every functionality affected by your changes. manually The absence of automated tests leads to us either spending too much time and slowing our shipping speed down or spending too little to save the velocity, resulting in new bugs in the backlog along with the late-night calls from PagerDuty. On the contrary, . So, computers can be programmed to do the same repeatedly let's delegate testing to computers! Types of Tests The suggests . Let's dive deep into every kind and understand why we need each. Testing pyramid idea three main types of tests: unit, integration, and end-to-end Unit Tests A is a small piece of logic you test in (without relying on other components). unit isolation They finish within seconds. allows them to run them at any point in time, locally and on CI, without spinning up the dependent services/making API and database calls. Unit tests are fast. Isolation A function that accepts two numbers and sums them together. We want to call it with different arguments and assert that the returned value is correct. Unit test example: // Function "sum" is the unit
const sum = (x, y) => x + y

test('sums numbers', () => {
  // Call the function, record the result
  const result = sum(1, 2);

  // Assert the result
  expect(result).toBe(3)
})

test('sums numbers', () => {
  // Call the function, record the result
  const result = sum(5, 10);

  // Assert the result
  expect(result).toBe(15)
}) A more interesting example is the React component that renders some text after the API request is finished. We need to mock the API module to return the necessary values for our tests, render the component, and assert the rendered HTML has the content we need. // "MyComponent" is the unit
const MyComponent = () => {
  const { isLoading } = apiModule.useSomeApiCall();

  return isLoading ? <div>Loading...</div> : <div>Hello world</div>
}

test('renders loading spinner when loading', () => {
  // Mocking the API module, so that it returns the value we need
  jest.mock(apiModule).mockReturnValue(() => ({
    useSomeApiCall: jest.fn(() => ({
      // Return "isLoading: true" for this test case
      isLoading: true
    }))
  }))

  // Execute the unit (render the component)
  const result = render(<MyComponent />)

  // Assert the result
  result.findByText('Loading...').toBeInTheDocument()
})

test('renders text content when not loading', () => {
  // Mocking the API module
  jest.mock(apiModule).mockReturnValue(() => ({
    useSomeApiCall: jest.fn(() => ({
      // Return "isLoading: false" for this test case
      isLoading: false
    }))
  }))

  // Execute the unit (render the component)
  const result = render(<MyComponent />)

  // Assert the result
  result.findByText('Hello world').toBeInTheDocument()
}) Integration Tests When your interacts with other , we call it an . These tests are slower than unit tests, but they test how the parts of your app connect. unit units (dependencies) integration A service that creates users in a database. This requires a DB instance ( ) to be available when the tests are executed. We will test that the service can create and retrieve a user from the DB. Integration test example: dependency import db from 'db'

// We will be testing "createUser" and "getUser"
const createUser = name => db.createUser(name) // creates a user
const getUser = name => db.getUserOrNull(name) // retrieves a user or null

test("creates and retrieves users", () => {
  // Try to get a user that doesn't exist, assert Null is returned
  const nonExistingUser = getUser("i don't exist")
  expect(nonExistingUser).toBe(null);

  // Create a user
  const userName = "test-user"
  createUser(userName);

  // Get the user that was just created, assert it's not Null
  const user = getUser(userName);
  expect(user).to.not.be(null)
}) End-to-End Tests It's an test when we test the , where all its dependencies are available. Those tests best simulate actual user behavior and allow you to catch in your app, but they are the type of tests. end-to-end fully deployed app all possible issues slowest Whenever you want to run end-to-end tests, you must provision all the infrastructure and make sure 3rd party providers are available in your environment. You want to have them for the features of your app. only mission-critical Login flow. We want to go to the app, fill in the login details, submit it, and see the welcome message. Let's take a look at an end-to-end test example: test('user can log in', () => {
  // Visit the login page
  page.goto('https://example.com/login');

  // Fill in the login form
  page.fill('#username', 'john');
  page.fill('#password', 'some-password');

  // Click the login button
  page.click('#login-button');

  // Assert the welcome message is visible
  page.assertTextVisible('Welcome, John!')
}) How Do You Choose What Kind of Test to Write? Remember that , and tests. end-to-end tests are slower than integration integration tests are slower than unit If the feature you are working on is mission-critical, consider writing at least one test (such as checking how the Login functionality works when developing the Authentication flow). end-to-end Besides mission-critical flows, we want to test as many edge cases and various states of the feature as possible. allow us to test how the parts of the app work together. Integration tests Endpoints should perform the operations, produce the expected result, and not throw any unexpected errors. Having integration tests for endpoints and client components is a good idea. Client components should display the correct content and respond to user interactions with how you expect them to respond. And finally, when should we choose ? All the small functions that can be tested in isolation, such as that sums the numbers, that renders tag, are great candidates for unit tests. Units are perfect if you follow the approach. unit tests sum Button <button> Test Driven Development What's Next? (but start small) Write some tests! that suits your project/language. Each language has a popular library for testing, such as / for JavaScript, / for end-to-end (uses JavaScript as well), for Java, etc. Install a testing framework Jest Vitest Cypress Playwright JUnit Find a small function in your project and write a test for it. unit Write an test for some component/service-database interaction. integration Choose a critical scenario that can be quickly tested, such as a simple login flow, and write an test for that. end-to-end Do the things above once to understand how it works. Then, do it again during some feature/bug work. Then share it with your colleagues so that you all write tests, save time, and sleep better at night! Useful Resources: My personal blog by The Practical Test Pyramid Ham Vocke by Test Driven Development Martin Fowler

Stable Software: The Power of Automated Tests

স্থিতিশীল সফ্টওয়্যার: স্বয়ংক্রিয় পরীক্ষার শক্তি সম্পর্কে জানুন

Stabile Software: Erfahren Sie mehr über die Leistungsfähigkeit automatisierter Tests

Software estable: conozca el poder de las pruebas automatizadas

Logiciel stable : découvrez la puissance des tests automatisés

स्थिर सॉफ़्टवेयर: स्वचालित परीक्षणों की शक्ति के बारे में जानें

安定したソフトウェア: 自動テストの威力について学ぶ

안정적인 소프트웨어: 자동화된 테스트의 힘에 대해 알아보세요

Software estável: aprenda sobre o poder dos testes automatizados

Стабильное программное обеспечение: узнайте о возможностях автоматизированных тестов

Kararlı Yazılım: Otomatik Testlerin Gücünü Öğrenin

Phần mềm ổn định: Tìm hiểu về sức mạnh của kiểm thử tự động

稳定的软件：了解自动化测试的力量

Looks like it’s part 2 of my series on Data Quality! In a , I explored Airbnb’s strategy for enhancing data quality through incentives. They implemented a single score and clear scoring criteria to establish a common understanding between data producers and consumers, fostering a genuine sense of ownership. previous post https://hackernoon.com/data-quality-score-one-score-to-rule-them-all?embedable=true Now, Lyft is taking a distinct approach, not attempting the same thing differently, but rather focusing on different aspects of data quality. And, Lyft’s strategy complements Airbnb’s efforts. While I regard Airbnb’s DQ score (or any similar score) as an effective means to consolidate various attempts at elevating data quality, Lyft is tackling this challenge from a different angle. The Airbnb’s DQ score serves as a valuable tool for providing a concrete visualization of data quality. In essence, any initiative to enhance data quality should manifest a discernible impact on this score. On the other hand, Lyft presents one possible initiative to proactively enhance quality by testing and validating data against specific quality criteria. https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6?source=post_page-----415baad6c95b--------------------------------&embedable=true Fundamentally, it’s a different point in the data quality lifecycle. Introducing a mechanism to improve quality necessitates the ability to measure it initially. So, while ’s focus lies on and data quality, relying on the producer’s interest to enhance this quality and “look good,” Lyft takes a different approach. Lyft places emphasis on actively and data quality, providing both producers and consumers with the means to effectively improve and control the quality. Airbnb measuring observing testing validating Collectively, these approaches provide a comprehensive strategy to address and enhance throughout its . data quality lifecycle For this reason, I was particularly interested in taking a closer look at Lyft’s approach. Another factor that intrigued me is testing, more specifically, contract testing, which has been used for many years now in basic software engineering with the emergence of microservice architecture. Data contracts, however, are something more recent in the domain of data engineering and are seen as the pinnacle or one of the ultimate steps to put in place on the path to building high-quality data pipelines. This is why I wanted to examine Lyft’s approach in more detail and explore some potential parallels. As mentioned, the approaches taken by Airbnb and Lyft are complementary and aim to achieve the same goal: improving data quality. has developed the DQ score, which focuses on measuring and enhancing 4 distinct : Airbnb aspects of data quality DQ Score has guiding principles, including full coverage, automation, actionability, multi-dimensionality, and evolvability. It has dimensions like Accuracy, Reliability, Stewardship, and Usability. Lyft’s Verity is a platform designed to enhance data quality across 5 dimensions Defines data quality as the measure of how well data can be used as intended, covering aspects like . semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness It’s easy to draw parallels between the 5 data quality aspects improved by Lyft’s Verity and the 4 data quality dimensions measured by the Airbnb’s DQ score. For example, aspects like Timeliness would certainly contribute to the DQ score’s Reliability, while Accuracy would be dependent on semantic correctness, completeness, and uniqueness of the data. The Usability score, on the other hand, is influenced by the consistency of the data among other factors. Lyft’s Verity focuses on defining related to . It follows a test-first and validation approach, whereas Airbnb’s unified DQ score emphasizes evaluating data quality through various dimensions. checks semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness If we wanted to incorporate the DQ score into this last schema, it would be on the side of the Alert/Debug steps. Airbnb’s DQ score uses different signals to assess data quality across the . Accuracy, Reliability, Stewardship, and Usability aspects We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). As discussed earlier, there are likely overlaps between Airbnb’s DQ score and Verity. While Airbnb focuses on pushing data quality to the right, emphasizing measurement and scoring, Lyft’s Verity takes a proactive approach by shifting the check definition configurations, testing, and validation processes to the left, emphasizing proactive improvement of data quality. Now, my primary interest lies to the left, in the check definition configurations, testing, and validation aspects. How Lyft integrates data quality testing into its processes? Let’s first examine the execution paths. Currently, Lyft’s Verity is primarily focused on ensuring the quality of data stored in its Hive data warehouse. However, there are plans to expand its capabilities to support other data sources in the future. Note that while they refer to Hive as a data workhouse, they actually utilize it as a hybrid data storage solution, operating as a data warehouse for structured, processed, and cleansed data (silver layer) while also serving as a data lake for raw event data (bronze layer). Poor data quality in Hive caused tainted experimentation metrics, inaccurate machine learning features, and flawed executive dashboards. Verity’s checks can be integrated into an Airflow DAG to ensure that only high-quality raw data is processed and stored in Hive as derived data. Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed inside or . Airflow Flyte … The VerityAirflowOperator can be used in a blocking fashion to halt a DAG upon a check failure, preventing bad data from ever reaching production. This utilizes the “ ” pattern: we create data in a staged schema, verify the data with a blocking operator, then promote it to production if it passes quality checks. Stage-Check-Exchange Checks can also be performed manually or scheduled automatically to verify both raw and derived data. Verity Scheduled Checks are isolated from any data orchestration engine, so they still run even if Airflow or Flyte are completely down. This remedies a common problem of checks not alerting because the Airflow Task never ran. So, there are essentially 3 primary ways to trigger a check: as part of an Airflow DAG, manually, or scheduled through the Verity platform/UI. I don’t believe that current checks can be integrated into real-time streaming pipelines (such as Flink + Kafka) to validate for example data as it enters Hive (or even earlier). Implementing this type of real-time check would enable prompt detection of discrepancies, leading to reduced storage and processing costs and overall data quality enhancement. Well, to be thorough, Verity checks are managed through an API server, which could be used to trigger checks programmatically through some APIs. Verity API Server — This service handles all the external APIs regarding running checks as well as persisting and retrieving their results. The API Server does not execute any checks, but rather writes a message to our Check Queue, which utilizes (SQS). SimpleQueueService So, maybe, you could potentially trigger these jobs in a more real-time fashion, such as from a streaming job or even, as a long stretch, by integrating with Hive change data capture (CDC). However, when executed outside of Airflow, these jobs would not be able to block the data processing job; instead, they would generate asynchronous alerts pushed to the Check Queue. Some consumers would prefer to have data processing delayed when a check fails, while others would rather proceed and receive an alert. Now, let’s look at these data quality tests. Here’s an example that checks whether is never null. This is accomplished through a combination of query and condition components. rider_events.session_id core rider events session_id is not null: # check name
  metadata:
    id: 90bde4fa-148b-4f06-bd5f-f15b3d2ad759
    ownership_slack: #dispatch-service-dev
    tags: [rides, core-data, high-priority]
  query:
    type: dsl
    data_source_id: hive.core.rider_events
    filters:
      - session_id = null
  condition:
    type: fixed_threshold
    max: 0
  notifier_group:
    pagerduty_policy: dispatch-service
    email: dispatch-service@lyft.pagerduty.com Verity is primarily focused on defining and enforcing data quality checks rather than defining complete data schemas. Schema validation is not a novel concept. Several methods exist for defining event data schemas in event-based systems, such as JSON Schema, Protocol Buffers, Avro, or storage formats like Parquet. The optimal choice depends on your tech stack, usage, and specific requirements. While data schemas are valuable for defining the overall structure of data objects or table rows, they fall short in capturing more sophisticated validation checks specific to consumers, such as data distribution, business rules, SLAs, and thresholds. Data contracts go beyond schema validation, which focuses on identifying . I personally find that JSON Schema offers here a more suitable and readable option, effectively separating these structural and syntactical validation capabilities from serialization or storage concerns. syntactical errors However, to address and enhance data accuracy, having an effective means of defining data checks allows for defining the other facet of data contracts. semantic errors This is where Verity DSL comes into play. Beyond semantical validation, data contracts offer another crucial aspect that deserves attention. From a syntactical standpoint, validation checks remain consistent regardless of the consumer or producer involved. The set of validation rules is not tied to any specific consumer or producer and can be defined once and for all as a single schema. However, the Verity data contract DSL offers a finer granularity defining small independent rules, which is particularly well-suited for this context: the semantic meaning and usage of the data vary depending on the specific consumer. Additionally, not all consumers need to utilize all of the properties of an object. Their expectations differ. This doesn’t imply they are contradictory (which would certainly be an issue), but rather complementary and distinct ones. Therefore, allowing all consumers to establish unique rules that, when combined collaboratively, can provide a comprehensive understanding of the semantic significance of data quality is quite crucial. And it’s this aspect that particularly resonates with me. Bear with me, this may seem like a stretch, but from my perspective, it’s worth mentioning. :) collaborative Data exchange enables different teams (producers and consumers) to collaborate effectively. Establishing a shared understanding of these data exchanges is paramount, just like APIs in traditional software development. In microservice architectures, a known as consumer-driven contracts (CDC) has emerged, where consumers define the expected behavior of APIs provided by producers. Producers are responsible for verifying these contracts before releasing new versions. collaborative testing approach I think data contracts share a similar collaborative spirit. Even though data validation is performed on real data, rather than at release time, and does not block releases, it’s based on cooperation and encourages teamwork between data producers and consumers. I strongly believe that this collaborative approach is key to improving data quality and should be further integrated into the process. Well, I’m a big fan of drawing parallels… Notice actually that this collaborative aspect is something mentioned also as part of Lyft’s verity charter. The VerityUI provides a streamlined data discovery experience via the Verity Homepage. Our full-text search on the Check Definition Metadata lets users see all the checks currently being enforced and their Check Results. This has useful aggregations like owning team, table name, and tags. I’m not entirely clear on how data contract issues are shared between consumers and producers through the Verity platform UI, but I definitely recognize the significance of collaboration through the dashboards: The producer of a data product interface can confidently ensure that they’re not inadvertently causing downstream breakages that they hadn’t anticipated. The consumer of the interface can rest assured that their reliance on the interface is not and will not be compromised. While Verity is a remarkable tool for defining data quality checks, it’s unfortunately not open-source. Fortunately, there’s another open-source data quality framework called Soda Core that provides similar functionality. Soda Core is a free and open-source command-line tool and Python library that enables data engineers to test data quality. It utilizes user-defined input to generate SQL queries that run checks on datasets in a data source to find invalid, missing or unexpected data. When checks fail, they surface the data that you defined as “bad” in the check. https://medium.com/@seckindinc/soda-core-the-simplest-open-source-data-reliability-tool-f35ae5b576d9?source=post_page-----415baad6c95b--------------------------------&embedable=true During a scan of a dataset, Soda Core evaluates the predefined checks to identify invalid, missing, or unexpected data. Here’s the equivalent check using Soda.core syntax for the Verity DSL test that was previously defined. name: rider_events_session_id_check
source: hive
query: SELECT * FROM rider_events WHERE session_id IS NULL;
raise_alert: true
threshold: 10
action: slack
message: "There are more than 10 rows that are null for the 'session_id' property in the 'rider_events' table. Please investigate this issue." soda run --check checks/rider_events_session_id_check.yaml Soda Core is a powerful tool for ensuring the quality of your data. It can help you to identify and fix data problems early, before they can cause issues for your business. It’s worth noting that Soda Core can also facilitate data quality checks for streaming data by seamlessly integrating with Spark DataFrames. While Verity’s data quality checks for Hive are applied to static datasets, the checks for streaming data need to be more lightweight and efficient. Data would typically be processed in small batches of events, with very low latency, making them suitable for real-time checks and specific use cases like anomaly detection. Finally, let’s mention that there are other data validation tools available, such as DeeQu, Great Expectations, … As we wrap up, I hope you have a clearer understanding of the steps you can take to enhance your data quality journey. We’ve seen two distinct approaches to data quality improvement, each with its own strengths and methodologies. One focuses on increasing visibility and observability, motivating data producers to raise the quality bar. The other prioritizes elevating the quality standard through a testing and validation-first approach. Both are complementary. Verity is not merely a domain-specific language (DSL) for defining data checks; it’s a centralized platform that empowers data practitioners to collaborate effectively. This platform helps producers and consumers align on data quality expectations, including format, structure, and accuracy. Verity’s data contract management capabilities could (is?) further enhanced by integrating with a broader set of features, such as metadata management and discovery, to address more sophisticated data quality needs. Similar to Airbnb’s DQ score, Lyft’s Verity fosters a collaborative feedback loop between data producers and consumers. By incentivizing and empowering each team to take ownership of data quality, Verity cultivates a supportive environment where data quality continuously improves. , Found this article useful? Follow me on Linkedin , and ! Please 👏 this article to share it! Hackernoon Medium Also published . here

Stories, News & Mentions about PAGERDUTY

Pagerduty’s Stories on HackerNoon

Pagerduty's News & Mentions