However, the risks and concerns have often been downplayed, with a lack of authenticity from some when it comes to the challenges around a lack of regulation, data protection, models hallucinating and being unable to ‘unlearn errors’, as well as a certain amount of trivializing when it comes to happen to the skills base in the market.

PagerDuty CTO - ‘Generative AI focus should be about minimizing collateral damage’

C athie Wood gained fame as an investor for her unwavering belief in companies at the forefront of technological innovation. Her ARK Invest focuses on finding disruptive technologies that can change the world, such as artificial intelligence (AI).

3 Cathie Wood Tech Stocks to Buy Hand Over Fist

UiPath, Unity, and PagerDuty aren't household names, but each company's mastery of its respective tech specialty helps win customers and grow revenue. As a result, these three growth stocks have earned a spot in Cathie Wood's ARK Invest funds and can generate attractive returns for you as well.

Detailed price information for Uipath Inc Cl A (PATH-N) from The Globe and Mail including charting and trades.

PagerDuty focuses on IT infrastructure monitoring to minimize disruption of an organization's crucial systems. Its platform uses AI to identify when various issues are related to the same root cause,

Detailed price information for Unity Software Inc (U-N) from The Globe and Mail including charting and trades.

Grab our FREE report, featuring an exclusive CEO interview. Ready for the name and trading symbol? Pagerduty Inc (NYSE: PD)’s stock price has increased by 4.87 compared to its previous closing price of 19.72. However, the company has seen a -1.01% ...

Pagerduty Inc (PD) Stock: A Look at the Analyst Recommendations

The Chaos Engineering Tools Market size is expected to grow from USD 1.9 billion in 2023 to USD 2.9 billion by 2028 at a Compound Annual Growth Rate (CAGR) of 8.8% during the forecast period, according to a new report by MarketsandMarkets™.

Chaos Engineering Tools Market worth $2.9 billion by 2028 - Exclusive Report by MarketsandMarkets™

As organisations prioritise system reliability and resilience in an ever-evolving digital landscape, the future of the Chaos Engineering

Shares of IT incident response platform PagerDuty (NYSE:PD) fell 9.48% in the pre-market session after it announced plans to raise $350 million in convertible senior notes via a private placement to qualified institutional buyers. PagerDuty intends to use ...

Why PagerDuty (PD) Stock Is Falling Today

PagerDuty

pagerduty

How we replaced manual review of PagerDuty alerts with an automatic weekly summary dashboard. I’m a software engineer at on the & Infrastructure team. We support a myriad of services for both external clients and internal employees: we enable the product, growth, and teams with tools like Redshift for large analytics queries and the for real-time investigation across the many services powering Plaid. Plaid Data Science engineering ELK stack Supporting these analytics & monitoring services is mission-critical: if they go down dozens of workflows across the company break; it’s important that we catch any degradation of service before it becomes it an issue. Over the course of 2018 we’ve added dozens of new features to the services we support: new datasets, faster analytics capabilities, and more self-service data portals. We need to find any issues before they are realized by our users: We use to send real-time alerts when there is a degradation in service performance. Our team has an on-call rotation: between four engineers we rotate being the directly responsible individual for handling any alerts of service performance generated through PagerDuty. The week you’re on call, you use the PagerDuty app to alert of any realtime alerts, and the PagerDuty slack integration to send other messages to the slack channel. PagerDuty DSI-Urgent DSI-Notify PagerDuty slackbot — friend to all. Towards the end of the year we realized that while we had done well to add new PagerDuty alerts to support SLAs for our services, we weren’t as disciplined with triaging and fixing the underlying long-term issues. In order to step back from treating the symptoms and work towards overall service quality, we decided to add an “on-call summary & handoff” digest to our weekly team meetings. This helped facilitate discussion about why we were being paged, iterate on a service to improve reliability, and change alert structure to reduce false alarms. Trialing a formalized handoff document, with categories of alerts to facilitate trend analysis and some comments on specific alerts for specific details. This worked well as a pilot — we were able to better surface common issues, more easily identify patterns, and decouple the on-call cadence of addressing real-time issues with the need for making longer term substantive changes. It was not, however, our long-term vision to add manual work to the on-call rotation. PagerDuty offers a few views into analytics, but we wanted finer grain control of our data and to be able to join it to other analytics data sources. We set out to use the to pull the data that would help inform patterns of incidents and supplant the manual tracking of our handoff document. PagerDuty V2 API We use for a wide-range of recurring data ETL, and wrote a one-off Airflow DAG that downloads a history of the data from PagerDuty, writes it into S3, and then copies from S3 to Redshift. You can see a sanitized version of the python script in . Apache Airflow this gist Airflow DAG for ingesting PagerDuty data We focused on two tables: and , which gives us a history of which team got notified about what alert on what date. PagerDuty offers a one month history of incidents through the API. Because a user can modify an incident well after the original date of the alert, we re-harvest the entire 30-day incident history and update any modified data. By contrast, log_entries is an append-only log, so we more simply take new data at a regular cadence and copy it into redshift. incidents log_entries Finally, with the data in Redshift we can create an automated version of our handoff document! Here’s an example of how we query the final schema to pull insights out of the incident and notification history: -- categorize alerts, and count by category + date-- takes a rolling five day historyselectdate_trunc('day', (created_at)::timestamp)::date, casewhen incidents.summary like '%airflow heartbeat%'then 'Airflow Heartbeat'when incidents.summary like '%mysql_dump%'then 'MysqlDump'when incidents.summary like '%TaskInstance: dump_%'then 'MongoDump'when incidents.summary like '%ES%not green%'then 'ES not green'when incidents.summary like '%queue is building%'then 'Logstash Queue'when incidents.summary like '%kinesis%behind%'then 'Kinesis Behind'when incidents.summary like '%TaskInstance%'then 'Airflow - Other'else-- remove numbers to more easily group alertsregexp_replace(incidents.summary, '[0-9]', '')end as summary, count(1)frompagerduty.incidentswhereabs(datediff('day', getdate(), created_at)) < 5and (escalation_policy_summary='DW-DSI')group by1, 2order by3 desc Which leads us to the grand conclusion: a Periscope dashboard where we can look at the number of alerts, the breakdown by category and service, and track trends over time. The final dashboard in Periscope, summarizing alerts over the past 7 days. Our handoff doc is a little cleaner now: rather than having to manually type out reports about individual alerts, we can summarize the week from our dashboard and jump straight to the collective and individual actions we want to take to build resilient and reliable data infrastructure. New handoff document, which focuses on concrete Action Items This new process has helped us prioritize & fix noisy alerts and persistent incidents. As a bonus, it helps our users surface and escalate lingering issues that have been masked by the cadence on on-call alerts, focusing on other important issues: The good case: our internal users attacking the real issues.

PagerDuty Analytics with Python & Periscope

We’re excited to announce that Bugsnag now supports error monitoring and stability management for Unreal Engine mobile games. Unreal Engine Unreal Engine is a game engine developed by Epic Games, first showcased in the 1998 first-person shooter game Unreal. Initially developed for PC first-person shooters, Unreal Engine has since been used in a variety of genres of three-dimensional (3D) games. It has also been widely adopted by the film and television industry. Written in C++, the Unreal Engine features a high degree of portability and supports a range of desktop, mobile, console, and virtual reality platforms. Bugsnag already provides stability monitoring to help engineering teams with a range of platforms and frameworks. mobile and desktop We're excited about extending our support for mobile platforms to include Unreal Engine. Developers will now have the actionable insights and analytics they need to increase the stability of their games and deliver a streamlined experience to their fans. Manage the Stability of Unreal Engine Games With Bugsnag Gain real-time visibility and actionable insights into stability. Development teams are releasing new versions and game modes more and more frequently to engage and retain users. With Bugsnag, you can understand the stability of every release and see how rapidly new releases are being adopted by users. You can compare release health to decide if they’re healthy enough to promote to production, or if they need to be pulled back. Setting your allows you to use the real-time stability scores to make data-driven decisions about when to build features in your application vs. focus on bug fixing. target stability and critical stability Essentially, if your game’s stability is below critical stability, the development team needs to focus on bug fixing and if it’s above target stability, they can focus on building new features. Additionally, the helps you understand stability issues over time. You can pinpoint error spikes, investigate specific incidents, determine if a release has introduced new bugs, and more. interactive timeline analysis Remember: Not all bugs are worth fixing. Prioritize fixing the bugs that matter. Understand which bugs are occurring more frequently and impacting the most users. View errors grouped by root cause and sorted by user impact so you can make data-driven decisions about which errors need to be addressed first. Prioritize Fixing the Bugs That Matter Not all bugs are created equal and you may want to prioritize fixing errors that are impacting . superfans or your revenue generating customers You can easily choose which meta data to collect for each error, such as the user’s spending tier, and then that are affecting paying gamers. segment errors Errors impacting business-critical app functions, such as the user sign-up or in-app purchase flows, are also more important to address quickly to minimize further impact to your businesses’ bottom line. Using error segmentation in Bugsnag, you can see which errors are impacting a certain part of the codebase, such as the one supporting game token purchases, so you can prioritize fixing them right away. Customize Error Views and Alerts Using bookmarks, which are a set of saved error segmentation filters, error views can be configured in the for each team member working on building and maintaining the game. Project Overview Dashboard An error view includes a feed of errors and a trend graph to show how errors are impacting the stability of the segment. You can set up error views for VIP users or superfans, in-app purchase flows, and even ad networks. Gaining visibility into errors originating from ad SDKs can help you decide if you need to turn off an erratic SDK that is negatively impacting the gaming experience. You can use the same bookmarks for configuring error alerts, so that you are only notified of errors occurring in segments that matter to you. This way you can improve the signal-to-noise ratio and focus the team on addressing errors that will have the greatest impact on improving app stability and user experience. You can: Fix errors with speed and precision to minimize further impact to user experience. Get to the root cause of errors quickly with comprehensive diagnostics. Comprehensive Diagnostics and Error Reproduction Reproducing and fixing errors is easier when you can . understand exactly what caused the crash See comprehensive diagnostic information for every error, including , user interaction breadcrumbs, device and user information. symbolicated stack traces Error reports also include Unreal-specific metadata, such as the MapUrl of the last loaded level, any custom User Activity you set within the app code, and the name of the current GameState class. You can also see changes to these values and the events that occurred in the run-up to the error as breadcrumbs. Integrations Streamline debugging workflows with powerful integrations. There are pre-built with the tools your team already uses, like Slack, PagerDuty and with Jira, Pivotal, and more. integrations 2-way issue tracker integration Automation can help streamline your debugging workflows and so you can quickly roll out fixes and minimize further impact to user experience. drive code ownership Get Started With Bugsnag Bugsnag’s new notifier library for Unreal Engine initially supports Android and iOS games. Simply create a new Unreal Engine project and follow the quick start guide in to start monitoring your application. our documentation If you’re new to Bugsnag, start a or to see the capabilities of our in action. 14-day free trial request a demo stability management platform

As a Gaming Developer, Which Error-Monitoring System Should I Use? 

Top APM Tools: 2024 Edition

It's one o'clock in the morning. My phone starts going crazy. I groan and look over. Sure enough, it's another PagerDuty call. I am on-call this week, and it's the third time in as many days that I have been woken up for the same reason. Hundreds of our customers have complained in unison, for three days in a row, that they have purchased event tickets from us and arrived at the venue only to find that their tickets have already been redeemed. They are being denied entry, and they are understandably very angry. Initial Investigation Showed Some Very Odd Behavior. Customers had purchased tickets, and then a matter of minutes later, their tickets would be redeemed. This was far in advance of the event itself, so it was clearly unexpected. And it wasn't spread evenly across our customer base; the issue was heavily focused within some select groups of university students. We already knew that we had a fundamental issue with our redemption system; it was possible to scan the QR code in the ticket order confirmation email yourself like the door staff would do at the venue, which would redeem the ticket. Unfortunately, the bigger fix for this was still a few weeks away. Even so, why would large numbers of customers within specific university circles redeem or accidentally redeem their own tickets before the event?! None of our early theories made sense, and the issue kept happening. We Went Digging. Eventually, We Found the Issue. The automated virus scanner on specific university .edu email systems were hitting all links in our confirmation emails to check them for viruses, including programmatically scanning all QR codes. Therefore this one process, completely invisible to my team, was automatically redeeming our customers' event tickets before the customer even received the email. Never, Ever Update Persisted Data on Page-Load. By making the ticket QR code hit a webpage that immediately redeems the ticket on page-load, it meant that anything visiting that link, whether human or machine, would inadvertently update our database to mark the ticket as redeemed. There are thousands of unseen automated visitors to our websites that we don't think about most of the time. Here are some examples: Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com, and there are prominent links on that page to/blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Email scanners. Very common in schools, governments, large corporations, and more. Internet archive machines The simple lesson here is to avoid things happening automatically on page-loads as much as possible. Besides the issues that we experienced, unexpected side-effects like this are bad UX. If the action is "Redeem", make it happen when you click a button that says "Redeem". I hope that this was useful - subscribe now to get my Dev Diary in your inbox! It's one o'clock in the morning. My phone starts going crazy. I groan and look over. Sure enough, it's another PagerDuty call. I am on-call this week, and it's the third time in as many days that I have been woken up for the same reason. Hundreds of our customers have complained in unison, for three days in a row, that they have purchased event tickets from us and arrived at the venue only to find that their tickets have already been redeemed . They are being denied entry, and they are understandably very angry. tickets have already been redeemed very angry. Initial Investigation Showed Some Very Odd Behavior. Customers had purchased tickets, and then a matter of minutes later, their tickets would be redeemed. This was far in advance of the event itself, so it was clearly unexpected. And it wasn't spread evenly across our customer base; the issue was heavily focused within some select groups of university students. We already knew that we had a fundamental issue with our redemption system; it was possible to scan the QR code in the ticket order confirmation email yourself like the door staff would do at the venue, which would redeem the ticket. Unfortunately, the bigger fix for this was still a few weeks away. Even so, why would large numbers of customers within specific university circles redeem or accidentally redeem their own tickets before the event?! None of our early theories made sense, and the issue kept happening. We Went Digging. Eventually, We Found the Issue. The automated virus scanner on specific university .edu email systems were hitting all links in our confirmation emails to check them for viruses, including programmatically scanning all QR codes . automated virus scanner .edu programmatically scanning all QR codes Therefore this one process, completely invisible to my team, was automatically redeeming our customers' event tickets before the customer even received the email. before the customer even received the email. Never, Ever Update Persisted Data on Page-Load. By making the ticket QR code hit a webpage that immediately redeems the ticket on page-load, it meant that anything visiting that link, whether human or machine, would inadvertently update our database to mark the ticket as redeemed. anything There are thousands of unseen automated visitors to our websites that we don't think about most of the time. Here are some examples: Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com, and there are prominent links on that page to/blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Browsers prefetching resources. This is a big one. Let's say you visit https://myblog.com , and there are prominent links on that page to /blog-post-1 and /blog-post-2 . It is very likely that your browser will already be prefetching those pages in the background. If those pages mutate data on page-load, for example, to track visits, just being on the homepage is enough to trigger it, even though you might never click on the links. Browsers prefetching resources. https://myblog.com /blog-post-1 /blog-post-2 just being on the homepage Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Google crawlers. Another big one. We recently saw a large number of errors that came from Google crawlers visiting pages and then triggering unauthenticated calls to APIs. Google crawlers. Email scanners. Very common in schools, governments, large corporations, and more. Email scanners. Very common in schools, governments, large corporations, and more. Email scanners. Internet archive machines Internet archive machines Internet archive machines The simple lesson here is to avoid things happening automatically on page-loads as much as possible. Besides the issues that we experienced, unexpected side-effects like this are bad UX. If the action is "Redeem", make it happen when you click a button that says "Redeem". I hope that this was useful - subscribe now to get my Dev Diary in your inbox!

Dev Diary #3: The Sneaky Issue That Was Silently Ruining Our Database

Day 1 and Day 2 Operations for Platform Engineers

DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon.


Our product (firewall) will continue working; clients just won't be able to access their accounts.


We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data.
Make secondary backups of critical data in the cloud.
Backup your Kubernetes master.
Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations.
Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! DISCLAIMER: This story happened in 2021, but I finally found some time and emotional power to make it written. Please enjoy your reading! Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. Hi Ivan! I have some good news and bad news, as they say. It looks like we'll finish our migration to the cloud today. It's been almost a year, and I've finally gathered the courage to write down the whole story, without embellishing or omitting any details. I hope this truth will help readers do better and learn from our mistakes. 7 PM in California is when Europe is still asleep, and our work here is done, allowing us some rest. That wasn't the case on March 10th. Honestly, I barely remember what March looked like after the 10th. And April too. Time seemed to speed up that spring and only slowed down by summer. We migrated to a cloud of ash. The cloud migration project is now complete! We migrated to a cloud of ash. The cloud migration project is now complete! On March 10th (00:47 in France, March 9th evening in California), the OVH data center in Strasbourg burned down. Technically two data centers, if you could call them container units separate buildings. Before this, I only knew Strasbourg as a border city in France with a human rights court. That day, I also learned that our servers were there—and that they were burning. At that point, I had these myths in my mind: The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything The fire will be put out soon. The fire will be put out soon. The fire will be put out soon. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. Our product (firewall) will continue working; clients just won't be able to access their accounts. We have backups; we'll recover everything We have backups; we'll recover everything We have backups; we'll recover everything None of these statements turned out to be entirely true. But to make it interesting and understandable, let's briefly introduce Wallarm: We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. We sell an API and application firewall that functions as an Ingress/Nginx/Envoy module, analyzes traffic, and stores the results (malicious requests) in the cloud. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. Clients access the cloud to review UX and retrieve alerts via API to PagerDuty, Slack, etc. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. We have three clouds; the one in Europe that burned at OVH was the oldest infrastructure-wise. Now, let's address the myths one by one Myth #1: The Fire Will Be Put Out Soon Myth #1: The Fire Will Be Put Out Soon The first message said that a fire had started in one of the rooms of SBG2, nothing alarming. But within a couple of hours, it was clear things were bad, and the data center was lost—burned down completely. The fire also spread to the neighboring container, SBG-1, and half of it burned down as well. Here's a tweet from OVH's CEO Octave Klaba: "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." "We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend activating your Disaster Recovery Plan." Interestingly, I had lunch with Octave in San Francisco back in the summer of 2019, where he shared their big plans for expansion and going public. I should have given him our branded fire extinguisher then... Myth #2: Our Product Will Continue Working Myth #2: Our Product Will Continue Working This was almost true, but not for all clients. We sell software that operates within client infrastructure and is independent of our clouds, one of which had burned down in France. However, this isn't entirely accurate because each new pod in the kube, upon startup, registers in the cloud via API. Without this registration, it doesn't start. Although we had tested scenarios where the cloud disconnects during an already running service, startup was not covered. We were in a situation where, after some time, all Kubernetes clients in Europe would have stopped working. More so, their traffic would have halted until our module was removed from the ingress. Our software operates inline. This problem was solved with a placeholder that outputs a JSON stating that all registrations for pods are accepted without checks. Later, we refactored this part to function properly, now the ingress starts even if the cloud is unavailable. Myth #3: We'll Restore Everything from Backups Myth #3: We'll Restore Everything from Backups In short, the backups were in the same data center. When we opted for dedicated servers, we wanted them close for better internal connectivity. Or perhaps we just weren't attentive to where our servers were placed. Either way, backups definitely should have been stored separately and in the cloud. And that's what we do now. We had to restore using logs, old and partial backups from people's machines, by decompiling client rules. Let me elaborate on our rules, called LOM (Local Training Set)—yes, we coined that ourselves. These rules don't reach the client in the same form as they are stored in the cloud; they are compiled into something like a decision tree for faster firewall operation. We needed to create a decompiler for these rules, which we did. We asked clients for their compiled LOM files, decompiled them, and uploaded them back to our cloud. We couldn't just restart client training from scratch as the product operates in blocking mode, with settings to prevent false positives and structures for client application APIs. This project was successfully executed with minimal losses, and we were able to lift the rules back up. Client Support Client Support Moving forward—I must say, we DID NOT LOSE A SINGLE CLIENT due to this incident. Despite everything initially going against us, we received not only words of support but real assistance in the form of LOM files, as mentioned earlier. Most importantly, we felt genuine understanding and love from our clients, which is hard to quantify or describe. I want to thank everyone again; it was very heartwarming. I wrote to the CEO of Google Cloud, and he replied. We decided to use Google Cloud in Europe to set up a new cloud there. We didn't have a manager in France, only in the USA. Google isn't the easiest company for communication, and they don't process tickets very quickly. So, I decided to reach out directly to the top on a whim and sent an email to Thomas Kurian, CEO of Google Cloud, with the subject "URGENT: GCP Account Manager is missing in the middle crisis." He responded in just 3 minutes and resolved the issue with the manager. Everything had to be done quickly, and, surprisingly, it worked. Memorable Souvenirs and Awards Memorable Souvenirs and Awards After the incident, we awarded bonuses to all participants (firefighters), which were categorized based on their contributions to the resolution of the consequences and sleepless nights. To add a bit of humor, we also released a series of 'This is fine' meme dogs, each representing a category of "firefighter" who received it. Hard Lessons Learned Hard Lessons Learned To be short, just take this piece of advice and share it with your team now. Use it as a checklist; just believe me, it’s mandatory. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't store backups in the same data center as your data. Don't store backups in the same data center as your data. Make secondary backups of critical data in the cloud. Make secondary backups of critical data in the cloud. Backup your Kubernetes master. Backup your Kubernetes master. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Conduct annual "drills" to simulate which parts of your system will fail under various circumstances, especially if you have many software components in different locations. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Don't be afraid to share accurate and honest status updates with clients; they will help and support you. Conclusions Conclusions We live in a world where a fire in a data center caused by a single uninterrupted power supply can burn down two buildings. Design, fire suppression, and isolation offer no guarantees. Rely only on yourself and your team. And mitigate risks with trusted providers (hint, we're now tested). Thanks to our clients and our teams: support for handling all communication and nerves, DevOps for their super adequate actions and swift setup of a new European cloud, development for the decompiler of rules and all programmer efforts, detection, everyone I forgot to mention, and of course, Viktor for coordinating the process and the disaster recovery plan that was ready before the fire. I'm out of words to express my gratitude to everyone involved in this project. Thank you! If you're interested, consider joining us at Wallarm; we're hiring globally and can help with relocation/nomad. We're cool. Write to us at jobs@wallarm.com (especially looking for strong technical product managers (TPMs), DevOps, and developers in Go/C). jobs@wallarm.com

How We Migrated to the Cloud... of Ashes (Wallarm OVH Recovery in 2021)

A Dozen (or so) Learnings From 15 Years of Software Incident Management

সফ্টওয়্যার ঘটনা ব্যবস্থাপনার 15 বছর থেকে এক ডজন (বা তাই) শিক্ষা

Ein Dutzend (oder so) Erkenntnisse aus 15 Jahren Software-Vorfallmanagement

Una docena (más o menos) de aprendizajes de 15 años de gestión de incidentes de software

Une douzaine (ou deux) d'enseignements tirés de 15 ans de gestion des incidents logiciels

सॉफ्टवेयर घटना प्रबंधन के 15 वर्षों से एक दर्जन (या लगभग) सीखें

15 年間のソフトウェア インシデント管理から学んだ 12 の教訓

15년간의 소프트웨어 사고 관리 경험을 통해 얻은 12가지 정도의 교훈

Uma dúzia (ou mais) de aprendizados de 15 anos de gerenciamento de incidentes de software

Дюжина (или около того) уроков за 15 лет управления инцидентами с программным обеспечением

15 Yıllık Yazılım Olay Yönetiminden Bir Düzine (ya da O Kadar) Ders

Hàng chục (hoặc hơn) bài học từ 15 năm quản lý sự cố phần mềm

从 15 年的软件事故管理中学到的十几个经验

This Article Is Worth Your Attention if You Are passionate about writing good quality software and want to enhance the stability of your app with tests. Are tired of unexpected bugs popping up in your production systems. Need help understanding what automated tests are and how to approach them. Why Do We Need Automated Tests? As engineers, we want to , but with each new feature we create, we inevitably increase the size and complexity of our apps. build things that work As the product grows, it becomes more and more time-consuming to (e.g., with your hands) test every functionality affected by your changes. manually The absence of automated tests leads to us either spending too much time and slowing our shipping speed down or spending too little to save the velocity, resulting in new bugs in the backlog along with the late-night calls from PagerDuty. On the contrary, . So, computers can be programmed to do the same repeatedly let's delegate testing to computers! Types of Tests The suggests . Let's dive deep into every kind and understand why we need each. Testing pyramid idea three main types of tests: unit, integration, and end-to-end Unit Tests A is a small piece of logic you test in (without relying on other components). unit isolation They finish within seconds. allows them to run them at any point in time, locally and on CI, without spinning up the dependent services/making API and database calls. Unit tests are fast. Isolation A function that accepts two numbers and sums them together. We want to call it with different arguments and assert that the returned value is correct. Unit test example: // Function "sum" is the unit
const sum = (x, y) => x + y

test('sums numbers', () => {
  // Call the function, record the result
  const result = sum(1, 2);

  // Assert the result
  expect(result).toBe(3)
})

test('sums numbers', () => {
  // Call the function, record the result
  const result = sum(5, 10);

  // Assert the result
  expect(result).toBe(15)
}) A more interesting example is the React component that renders some text after the API request is finished. We need to mock the API module to return the necessary values for our tests, render the component, and assert the rendered HTML has the content we need. // "MyComponent" is the unit
const MyComponent = () => {
  const { isLoading } = apiModule.useSomeApiCall();

  return isLoading ? <div>Loading...</div> : <div>Hello world</div>
}

test('renders loading spinner when loading', () => {
  // Mocking the API module, so that it returns the value we need
  jest.mock(apiModule).mockReturnValue(() => ({
    useSomeApiCall: jest.fn(() => ({
      // Return "isLoading: true" for this test case
      isLoading: true
    }))
  }))

  // Execute the unit (render the component)
  const result = render(<MyComponent />)

  // Assert the result
  result.findByText('Loading...').toBeInTheDocument()
})

test('renders text content when not loading', () => {
  // Mocking the API module
  jest.mock(apiModule).mockReturnValue(() => ({
    useSomeApiCall: jest.fn(() => ({
      // Return "isLoading: false" for this test case
      isLoading: false
    }))
  }))

  // Execute the unit (render the component)
  const result = render(<MyComponent />)

  // Assert the result
  result.findByText('Hello world').toBeInTheDocument()
}) Integration Tests When your interacts with other , we call it an . These tests are slower than unit tests, but they test how the parts of your app connect. unit units (dependencies) integration A service that creates users in a database. This requires a DB instance ( ) to be available when the tests are executed. We will test that the service can create and retrieve a user from the DB. Integration test example: dependency import db from 'db'

// We will be testing "createUser" and "getUser"
const createUser = name => db.createUser(name) // creates a user
const getUser = name => db.getUserOrNull(name) // retrieves a user or null

test("creates and retrieves users", () => {
  // Try to get a user that doesn't exist, assert Null is returned
  const nonExistingUser = getUser("i don't exist")
  expect(nonExistingUser).toBe(null);

  // Create a user
  const userName = "test-user"
  createUser(userName);

  // Get the user that was just created, assert it's not Null
  const user = getUser(userName);
  expect(user).to.not.be(null)
}) End-to-End Tests It's an test when we test the , where all its dependencies are available. Those tests best simulate actual user behavior and allow you to catch in your app, but they are the type of tests. end-to-end fully deployed app all possible issues slowest Whenever you want to run end-to-end tests, you must provision all the infrastructure and make sure 3rd party providers are available in your environment. You want to have them for the features of your app. only mission-critical Login flow. We want to go to the app, fill in the login details, submit it, and see the welcome message. Let's take a look at an end-to-end test example: test('user can log in', () => {
  // Visit the login page
  page.goto('https://example.com/login');

  // Fill in the login form
  page.fill('#username', 'john');
  page.fill('#password', 'some-password');

  // Click the login button
  page.click('#login-button');

  // Assert the welcome message is visible
  page.assertTextVisible('Welcome, John!')
}) How Do You Choose What Kind of Test to Write? Remember that , and tests. end-to-end tests are slower than integration integration tests are slower than unit If the feature you are working on is mission-critical, consider writing at least one test (such as checking how the Login functionality works when developing the Authentication flow). end-to-end Besides mission-critical flows, we want to test as many edge cases and various states of the feature as possible. allow us to test how the parts of the app work together. Integration tests Endpoints should perform the operations, produce the expected result, and not throw any unexpected errors. Having integration tests for endpoints and client components is a good idea. Client components should display the correct content and respond to user interactions with how you expect them to respond. And finally, when should we choose ? All the small functions that can be tested in isolation, such as that sums the numbers, that renders tag, are great candidates for unit tests. Units are perfect if you follow the approach. unit tests sum Button <button> Test Driven Development What's Next? (but start small) Write some tests! that suits your project/language. Each language has a popular library for testing, such as / for JavaScript, / for end-to-end (uses JavaScript as well), for Java, etc. Install a testing framework Jest Vitest Cypress Playwright JUnit Find a small function in your project and write a test for it. unit Write an test for some component/service-database interaction. integration Choose a critical scenario that can be quickly tested, such as a simple login flow, and write an test for that. end-to-end Do the things above once to understand how it works. Then, do it again during some feature/bug work. Then share it with your colleagues so that you all write tests, save time, and sleep better at night! Useful Resources: My personal blog by The Practical Test Pyramid Ham Vocke by Test Driven Development Martin Fowler

Stable Software: The Power of Automated Tests

স্থিতিশীল সফ্টওয়্যার: স্বয়ংক্রিয় পরীক্ষার শক্তি সম্পর্কে জানুন

Stabile Software: Erfahren Sie mehr über die Leistungsfähigkeit automatisierter Tests

Software estable: conozca el poder de las pruebas automatizadas

Logiciel stable : découvrez la puissance des tests automatisés

स्थिर सॉफ़्टवेयर: स्वचालित परीक्षणों की शक्ति के बारे में जानें

安定したソフトウェア: 自動テストの威力について学ぶ

안정적인 소프트웨어: 자동화된 테스트의 힘에 대해 알아보세요

Software estável: aprenda sobre o poder dos testes automatizados

Стабильное программное обеспечение: узнайте о возможностях автоматизированных тестов

Kararlı Yazılım: Otomatik Testlerin Gücünü Öğrenin

Phần mềm ổn định: Tìm hiểu về sức mạnh của kiểm thử tự động

稳定的软件：了解自动化测试的力量

Looks like it’s part 2 of my series on Data Quality! In a , I explored Airbnb’s strategy for enhancing data quality through incentives. They implemented a single score and clear scoring criteria to establish a common understanding between data producers and consumers, fostering a genuine sense of ownership. previous post https://hackernoon.com/data-quality-score-one-score-to-rule-them-all?embedable=true Now, Lyft is taking a distinct approach, not attempting the same thing differently, but rather focusing on different aspects of data quality. And, Lyft’s strategy complements Airbnb’s efforts. While I regard Airbnb’s DQ score (or any similar score) as an effective means to consolidate various attempts at elevating data quality, Lyft is tackling this challenge from a different angle. The Airbnb’s DQ score serves as a valuable tool for providing a concrete visualization of data quality. In essence, any initiative to enhance data quality should manifest a discernible impact on this score. On the other hand, Lyft presents one possible initiative to proactively enhance quality by testing and validating data against specific quality criteria. https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6?source=post_page-----415baad6c95b--------------------------------&embedable=true Fundamentally, it’s a different point in the data quality lifecycle. Introducing a mechanism to improve quality necessitates the ability to measure it initially. So, while ’s focus lies on and data quality, relying on the producer’s interest to enhance this quality and “look good,” Lyft takes a different approach. Lyft places emphasis on actively and data quality, providing both producers and consumers with the means to effectively improve and control the quality. Airbnb measuring observing testing validating Collectively, these approaches provide a comprehensive strategy to address and enhance throughout its . data quality lifecycle For this reason, I was particularly interested in taking a closer look at Lyft’s approach. Another factor that intrigued me is testing, more specifically, contract testing, which has been used for many years now in basic software engineering with the emergence of microservice architecture. Data contracts, however, are something more recent in the domain of data engineering and are seen as the pinnacle or one of the ultimate steps to put in place on the path to building high-quality data pipelines. This is why I wanted to examine Lyft’s approach in more detail and explore some potential parallels. As mentioned, the approaches taken by Airbnb and Lyft are complementary and aim to achieve the same goal: improving data quality. has developed the DQ score, which focuses on measuring and enhancing 4 distinct : Airbnb aspects of data quality DQ Score has guiding principles, including full coverage, automation, actionability, multi-dimensionality, and evolvability. It has dimensions like Accuracy, Reliability, Stewardship, and Usability. Lyft’s Verity is a platform designed to enhance data quality across 5 dimensions Defines data quality as the measure of how well data can be used as intended, covering aspects like . semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness It’s easy to draw parallels between the 5 data quality aspects improved by Lyft’s Verity and the 4 data quality dimensions measured by the Airbnb’s DQ score. For example, aspects like Timeliness would certainly contribute to the DQ score’s Reliability, while Accuracy would be dependent on semantic correctness, completeness, and uniqueness of the data. The Usability score, on the other hand, is influenced by the consistency of the data among other factors. Lyft’s Verity focuses on defining related to . It follows a test-first and validation approach, whereas Airbnb’s unified DQ score emphasizes evaluating data quality through various dimensions. checks semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness If we wanted to incorporate the DQ score into this last schema, it would be on the side of the Alert/Debug steps. Airbnb’s DQ score uses different signals to assess data quality across the . Accuracy, Reliability, Stewardship, and Usability aspects We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). As discussed earlier, there are likely overlaps between Airbnb’s DQ score and Verity. While Airbnb focuses on pushing data quality to the right, emphasizing measurement and scoring, Lyft’s Verity takes a proactive approach by shifting the check definition configurations, testing, and validation processes to the left, emphasizing proactive improvement of data quality. Now, my primary interest lies to the left, in the check definition configurations, testing, and validation aspects. How Lyft integrates data quality testing into its processes? Let’s first examine the execution paths. Currently, Lyft’s Verity is primarily focused on ensuring the quality of data stored in its Hive data warehouse. However, there are plans to expand its capabilities to support other data sources in the future. Note that while they refer to Hive as a data workhouse, they actually utilize it as a hybrid data storage solution, operating as a data warehouse for structured, processed, and cleansed data (silver layer) while also serving as a data lake for raw event data (bronze layer). Poor data quality in Hive caused tainted experimentation metrics, inaccurate machine learning features, and flawed executive dashboards. Verity’s checks can be integrated into an Airflow DAG to ensure that only high-quality raw data is processed and stored in Hive as derived data. Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed inside or . Airflow Flyte … The VerityAirflowOperator can be used in a blocking fashion to halt a DAG upon a check failure, preventing bad data from ever reaching production. This utilizes the “ ” pattern: we create data in a staged schema, verify the data with a blocking operator, then promote it to production if it passes quality checks. Stage-Check-Exchange Checks can also be performed manually or scheduled automatically to verify both raw and derived data. Verity Scheduled Checks are isolated from any data orchestration engine, so they still run even if Airflow or Flyte are completely down. This remedies a common problem of checks not alerting because the Airflow Task never ran. So, there are essentially 3 primary ways to trigger a check: as part of an Airflow DAG, manually, or scheduled through the Verity platform/UI. I don’t believe that current checks can be integrated into real-time streaming pipelines (such as Flink + Kafka) to validate for example data as it enters Hive (or even earlier). Implementing this type of real-time check would enable prompt detection of discrepancies, leading to reduced storage and processing costs and overall data quality enhancement. Well, to be thorough, Verity checks are managed through an API server, which could be used to trigger checks programmatically through some APIs. Verity API Server — This service handles all the external APIs regarding running checks as well as persisting and retrieving their results. The API Server does not execute any checks, but rather writes a message to our Check Queue, which utilizes (SQS). SimpleQueueService So, maybe, you could potentially trigger these jobs in a more real-time fashion, such as from a streaming job or even, as a long stretch, by integrating with Hive change data capture (CDC). However, when executed outside of Airflow, these jobs would not be able to block the data processing job; instead, they would generate asynchronous alerts pushed to the Check Queue. Some consumers would prefer to have data processing delayed when a check fails, while others would rather proceed and receive an alert. Now, let’s look at these data quality tests. Here’s an example that checks whether is never null. This is accomplished through a combination of query and condition components. rider_events.session_id core rider events session_id is not null: # check name
  metadata:
    id: 90bde4fa-148b-4f06-bd5f-f15b3d2ad759
    ownership_slack: #dispatch-service-dev
    tags: [rides, core-data, high-priority]
  query:
    type: dsl
    data_source_id: hive.core.rider_events
    filters:
      - session_id = null
  condition:
    type: fixed_threshold
    max: 0
  notifier_group:
    pagerduty_policy: dispatch-service
    email: dispatch-service@lyft.pagerduty.com Verity is primarily focused on defining and enforcing data quality checks rather than defining complete data schemas. Schema validation is not a novel concept. Several methods exist for defining event data schemas in event-based systems, such as JSON Schema, Protocol Buffers, Avro, or storage formats like Parquet. The optimal choice depends on your tech stack, usage, and specific requirements. While data schemas are valuable for defining the overall structure of data objects or table rows, they fall short in capturing more sophisticated validation checks specific to consumers, such as data distribution, business rules, SLAs, and thresholds. Data contracts go beyond schema validation, which focuses on identifying . I personally find that JSON Schema offers here a more suitable and readable option, effectively separating these structural and syntactical validation capabilities from serialization or storage concerns. syntactical errors However, to address and enhance data accuracy, having an effective means of defining data checks allows for defining the other facet of data contracts. semantic errors This is where Verity DSL comes into play. Beyond semantical validation, data contracts offer another crucial aspect that deserves attention. From a syntactical standpoint, validation checks remain consistent regardless of the consumer or producer involved. The set of validation rules is not tied to any specific consumer or producer and can be defined once and for all as a single schema. However, the Verity data contract DSL offers a finer granularity defining small independent rules, which is particularly well-suited for this context: the semantic meaning and usage of the data vary depending on the specific consumer. Additionally, not all consumers need to utilize all of the properties of an object. Their expectations differ. This doesn’t imply they are contradictory (which would certainly be an issue), but rather complementary and distinct ones. Therefore, allowing all consumers to establish unique rules that, when combined collaboratively, can provide a comprehensive understanding of the semantic significance of data quality is quite crucial. And it’s this aspect that particularly resonates with me. Bear with me, this may seem like a stretch, but from my perspective, it’s worth mentioning. :) collaborative Data exchange enables different teams (producers and consumers) to collaborate effectively. Establishing a shared understanding of these data exchanges is paramount, just like APIs in traditional software development. In microservice architectures, a known as consumer-driven contracts (CDC) has emerged, where consumers define the expected behavior of APIs provided by producers. Producers are responsible for verifying these contracts before releasing new versions. collaborative testing approach I think data contracts share a similar collaborative spirit. Even though data validation is performed on real data, rather than at release time, and does not block releases, it’s based on cooperation and encourages teamwork between data producers and consumers. I strongly believe that this collaborative approach is key to improving data quality and should be further integrated into the process. Well, I’m a big fan of drawing parallels… Notice actually that this collaborative aspect is something mentioned also as part of Lyft’s verity charter. The VerityUI provides a streamlined data discovery experience via the Verity Homepage. Our full-text search on the Check Definition Metadata lets users see all the checks currently being enforced and their Check Results. This has useful aggregations like owning team, table name, and tags. I’m not entirely clear on how data contract issues are shared between consumers and producers through the Verity platform UI, but I definitely recognize the significance of collaboration through the dashboards: The producer of a data product interface can confidently ensure that they’re not inadvertently causing downstream breakages that they hadn’t anticipated. The consumer of the interface can rest assured that their reliance on the interface is not and will not be compromised. While Verity is a remarkable tool for defining data quality checks, it’s unfortunately not open-source. Fortunately, there’s another open-source data quality framework called Soda Core that provides similar functionality. Soda Core is a free and open-source command-line tool and Python library that enables data engineers to test data quality. It utilizes user-defined input to generate SQL queries that run checks on datasets in a data source to find invalid, missing or unexpected data. When checks fail, they surface the data that you defined as “bad” in the check. https://medium.com/@seckindinc/soda-core-the-simplest-open-source-data-reliability-tool-f35ae5b576d9?source=post_page-----415baad6c95b--------------------------------&embedable=true During a scan of a dataset, Soda Core evaluates the predefined checks to identify invalid, missing, or unexpected data. Here’s the equivalent check using Soda.core syntax for the Verity DSL test that was previously defined. name: rider_events_session_id_check
source: hive
query: SELECT * FROM rider_events WHERE session_id IS NULL;
raise_alert: true
threshold: 10
action: slack
message: "There are more than 10 rows that are null for the 'session_id' property in the 'rider_events' table. Please investigate this issue." soda run --check checks/rider_events_session_id_check.yaml Soda Core is a powerful tool for ensuring the quality of your data. It can help you to identify and fix data problems early, before they can cause issues for your business. It’s worth noting that Soda Core can also facilitate data quality checks for streaming data by seamlessly integrating with Spark DataFrames. While Verity’s data quality checks for Hive are applied to static datasets, the checks for streaming data need to be more lightweight and efficient. Data would typically be processed in small batches of events, with very low latency, making them suitable for real-time checks and specific use cases like anomaly detection. Finally, let’s mention that there are other data validation tools available, such as DeeQu, Great Expectations, … As we wrap up, I hope you have a clearer understanding of the steps you can take to enhance your data quality journey. We’ve seen two distinct approaches to data quality improvement, each with its own strengths and methodologies. One focuses on increasing visibility and observability, motivating data producers to raise the quality bar. The other prioritizes elevating the quality standard through a testing and validation-first approach. Both are complementary. Verity is not merely a domain-specific language (DSL) for defining data checks; it’s a centralized platform that empowers data practitioners to collaborate effectively. This platform helps producers and consumers align on data quality expectations, including format, structure, and accuracy. Verity’s data contract management capabilities could (is?) further enhanced by integrating with a broader set of features, such as metadata management and discovery, to address more sophisticated data quality needs. Similar to Airbnb’s DQ score, Lyft’s Verity fosters a collaborative feedback loop between data producers and consumers. By incentivizing and empowering each team to take ownership of data quality, Verity cultivates a supportive environment where data quality continuously improves. , Found this article useful? Follow me on Linkedin , and ! Please 👏 this article to share it! Hackernoon Medium Also published . here

Advancing Data Quality: Exploring Data Contracts with Lyft

ডেটা গুণমান উন্নত করা: Lyft-এর সাথে ডেটা চুক্তিগুলি অন্বেষণ করা

Verbesserung der Datenqualität: Erkundung von Datenverträgen mit Lyft

Mejora de la calidad de los datos: exploración de los contratos de datos con Lyft

Améliorer la qualité des données : explorer les contrats de données avec Lyft

डेटा गुणवत्ता को आगे बढ़ाना: Lyft के साथ डेटा अनुबंध की खोज करना

データ品質の向上: Lyft とのデータ契約の検討

데이터 품질 향상: Lyft를 통한 데이터 계약 탐색

Avançando na qualidade dos dados: explorando contratos de dados com Lyft

Повышение качества данных: изучение контрактов на передачу данных с Lyft

Veri Kalitesini Geliştirme: Lyft ile Veri Sözleşmelerini Keşfetmek

Nâng cao chất lượng dữ liệu: Khám phá hợp đồng dữ liệu với Lyft

提高数据质量：与 Lyft 探索数据合同

Last month, when I was back in my hometown & on-call simultaneously, I got an alert from OpsGenie at 4:30 am. The alert was loud enough to wake everyone up. Everyone was curious about who was calling this early, so I explained to them that it was a bug in my alerting me. The immediate question was why an error in the code had to give me a call when it was occurring. code Now, to explain it, the simplest analogy I could think of was of a firefighter. Similar to how when a fire happens, the fire alarm alerts the fireman. Similarly, in the case of software engineering, the bug is the fire, the call is the fire alarm & I am the firefighter. In recent months, I have been trying to set up the on-call process in my team & this blog post, we will be breaking down and understanding what the on-call process looks like and what are some of the things to keep in mind when setting up the process. Firstly, let's understand why a developer needs to be on-call. This is something I read about the practice that is followed at PagerDuty, ” Since developers usually want the flexibility to deploy to production whenever they want, they, being the subject matter experts on the code, should be the ones taking ownership of any alerts. It's a practice that is followed where SREs/DevOps Engineers are usually responsible for the infrastructure (hardware, databases) while the Developers are responsible for everything built on top of it. This responsibility division is usually followed in most of the organizations but may vary depending on the team size & the organization structure. "you build it, you own it. Now that we have understood why a developer needs to be on-call let's break down the on-call process. Setting up Alerts This is the most essential step in starting your on-call process. The main goal of this step is to identify the most crucial services, which, if down, can disrupt the user experience or indicate some fault in the infrastructure, which may not be impacting the user directly but has some indirect effect on the overall functioning of the product. One of the important things to keep in mind is to avoid alert fatigue, i.e., making sure the alerts are fewer by removing unactionable/unwanted alerts. This is something that gets better over time by experimentation. One of the ways to check if the experimentation is working or not is to see the trend of alerts over time. Multiple apps might be used, like New Relic, Sentry, and HoneyBadger, but it is always recommended to redirect all the alerts from different platforms to one alert management tool (OpsGenie/PagerDuty). Setting up the OnCall Schedule When it comes to on-call schedules, there's no one-size-fits-all. The schedule varies from team to team. For teams that are spread across geographies, teams usually adhere to a process where people have to be on call only during the day & as the day sets, team members from the other end of the geography take charge of on-call. In cases where the team is in the same geography, the team members are on-call for both day & night but might vary from team to team. follow-the-sun One of the things to note is to encourage people on-call to swap their schedules with other teammates if they are available in case of emergency/travel plans so that they can try to maintain their regular schedule as much as possible. When doing so, it is important to update the alert management system to make sure all alerts are redirected to the right person. Remember that on-call can sometimes be tiring for the engineer & hence it is recommended to relax or swap hours in between. Escalation Policy There are times when the on-call engineer might not be able to acknowledge an alert due to various reasons like being AFK or internet issues. In such cases, having an escalation policy helps. This is what a sample escalation policy looks like. At the 0th Minute, that is when the alert occurs, the on-call engineer is notified. If the alert stays unacknowledged till the 4th minute after the alert occurred, the next on-call engineer will be alerted, and so on. The escalation policy can again be modified from team to team based on the requirements. Now that we have understood how to set the on-call process let's understand the process during an outage. During an Outage When the Alarm Rings: The Act of Acknowledgment and Assessment At the heart of on-call, acknowledging an alert is the first move. The on-call engineer becomes the first responder, swiftly assessing the severity and impact of the alert. The key here is to stay calm. It is recommended to document the severity criteria (P0 - entire site down, P1 - some parts not functioning) so that the on-call engineer can map the alert to severity & respond. This is called Triaging. Inform and Mobilize: The Symphony of Stakeholder Communication With the facts in hand, it's time to inform the right stakeholders. Communication is key, and orchestrating it effectively ensures everyone is on the same page. This usually involves keeping the customer-facing & engineering teams in the loop with the progress of the outage. Try keeping the appropriate channels of communication updated. It can also be in the form of JIRA tickets with relevant data, graphs, and CLI outputs. Seeking a Lifeline: Collaboration in War Rooms For complex issues, enter the war room—a collaborative space where different engineers unite to tackle the problem head-on. Seeking help is not a sign of weakness but a strategic move towards resolution. This also involves getting direction on the outage from the subject matter experts. Mitigate, Don't Muddle: The Art of Temporary Fixes In the heat of the moment, the goal is clear: mitigate the damage and restore functionality. Avoid fixing what's in production; instead, return to the last known stable state. Remember to never fix code when an outage is going on. Root Cause Quest: Don't Dilly-Dally, But Don't Rush Once the mitigation is done, follow the structured approach that involves finding the root cause first, fixing it to the core & ensuring a more stable foundation for the future. Testing, Deployment, and the Healing Process Don't bypass testing and deployment practices in the aftermath. Learn from the incident, implement preventive measures, and fortify your defenses against future disruptions. Closing the Curtains on Chaos: When the Root Cause is Resolved An outage can be considered resolved when all the alerts are back to normal & we see that the product is behaving expectedly for all the customers. It is important to note that an outage is considered resolved not after mitigation but once the permanent fix is in production. The Post-Incident Analysis: A Blameless Review As a last step, it's time for Root Cause Analysis (RCA) with a blameless mindset. Answer the quintessential who, what, where, when, and why to extract meaningful insights. It is in this step that we can document what went good/bad & what can be improved further with the next steps. Metrics to look at Now that we know how to respond to an alert, some of the metrics that can help us understand how the on-call process metrics of the team are MTTA (mean time to acknowledge) is the average time it takes from when an alert is triggered to when work begins on the issue. This metric is useful for tracking your team’s responsiveness and your alert system’s effectiveness. MTTR (mean time to resolve) is the average time it takes to fully resolve a failure. This includes not only the time spent detecting the failure, diagnosing the problem, and repairing the issue but also the time spent ensuring that the failure won’t happen again. I hope this article helps you in understanding the on-call process better. Also published . here

Stories, News & Mentions about PAGERDUTY

Pagerduty’s Stories on HackerNoon

Pagerduty's News & Mentions