&quot;incident.io is tied to our ability to provide a really great experience for our customers; it&#x27;s core to what our company is about. We have very demanding customers at the intersection of enterprises and developers. So, making it easy for us to track follow-ups and do all of the right things ...

incident.io — Incident management that brings calm to chaos

Incident.io is a Slack-integrated incident management tool used to announce, manage, and resolve all incidents in a single channel.

incident.io - Crunchbase Company Profile & Funding

Elegantly designed and with powerful workflow automations that power more efficient response processes, teams use incident.io to supercharge their response flow across the entire organization. Trusted by hundreds of tech-led companies such as Etsy, monday.com and Skyscanner, incident.io empowers ...

incident.io | LinkedIn

Search company, investor... ... incident.io is a slack-powered incident management platform. It automates workflow and integrates existing tools. It enables companies to declare, collaborate, communicate around, and learn from events that disturb their normal course of business such as critical ...

incident.io - Products, Competitors, Financials, Employees, ...

While at Monzo, Evans had built open source tooling to help incidents run through resolution pipelines more efficiently, which sparked the idea for Incident.io. “We noticed that other companies were either struggling with manual process, or investing precious engineering time repeatedly building ...

Incident.io raises $28.7M for Slack-based incident response

Compare the best business software and services based on user ratings and social data. Reviews for CRM, ERP, CAD, PDM, HR, and Marketing software.

incident.io Reviews 2023: Details, Pricing, & Features - G2

Information on valuation, funding, cap tables, investors, and executives for incident.io. Use the PitchBook Platform to explore the full profile.

incident.io Company Profile: Valuation & Investors | PitchBook

incident.io is a Slack-powered incident management tool, which helps you to create, respond to and learn from your incidents.

incident.io 2023 Pricing, Features, Reviews & Alternatives | GetApp

Incident.io has raised a $4.7 million funding round led by Index Ventures and Point Nine. When your company is facing an outage or a data breach,

Incident.io helps your team communicate better during outages | ...

When processes and tools add ... around the world, from high growth startups like Linear and Render to billion-dollar companies such as GoCardless and Loom.With new funding, incident.io will be building out its team in London, growing its user base and investing heavily in ...

incident.io raises $4.7 million to bring world-class incident ...

incident.io: The best way for your whole organisation to respond, review and learn from incidents.

incident.io | Index Ventures

Global technology companies use incident.io to empower teams to run incidents from start to finish, by getting the right people involved in real-time on Slack. Our platform integrates with tools you already use, with intuitive workflows and guiding actions to support your team through tough ...

incident.io: Revenue, Competitors, Alternatives

incident.io Help Center

incident.io Help Center

A “two-way street” communications ... young company’s growth. “We ask customers about their problems, and how we can help them,” Seymour says. “On the other hand, they’ll tell us when they’ve spotted a bug – and we can build a strong relationship with the customer by fixing it straight away.” · Meeting the engineering team’s high standards for performance and quality demanded CI/CD from incident.io’s earliest ...

incident.io raises product quality with CircleCI - CircleCI

incident.io

My first -incident happened in my second week here, when I screwed up the process for requesting extra Slack permissions, which made it impossible to install our app for a few minutes. This was a bit embarrassing, but also simple to resolve for someone more familiar with the process, and declaring an incident meant we got there in just a few minutes. incident.io Declaring the first incident when you start a new job can be intimidating, but it really shouldn't be. Let's look at some common fears, and work out how to address them. "Won't it be loads of work?" Most organizations have some kind of incident response procedure that will a list of things you have to do, like working out how many customers this affected, deciding how to make things right with them, and putting in proactive measures to avoid this happening again. That can seem like a daunting prospect. You might ask Is this issue ? really worth all that A safe default should be "yes". If it turns out this issue wasn't so bad after all, you should be able to shut down the response pretty quickly. But if it escalates further, you'll be glad you already have the process rolling. You can find out more about how to automate your incident process in our previous article. If it does end up being a lot of work, that isn't necessarily a bad thing. As an individual, you're likely to learn a lot from tackling your first big incident. As a team or company, you've addressed a serious issue proactively. "Will this get me or my team in trouble?" If the answer is "yes" you should probably be looking for another job. As Chris discussed before, having more incidents is not a bad thing --- it's not in the long-term interests of any organisation to brush small incidents under the rug, because you can never tell which might turn into huge problems later on. The same logic applies here too! Most teams that don't have any incidents are either not taking any risks and slowing down delivery, or hiding their problems. Neither of those is a sign of a healthy team. That's not to say that managers should set a target number of incidents per team per quarter, but it does mean that managers should be looking at outlier teams that have very incidents, as well as those that have more than their fair share. few Are they afraid of getting blamed? Are they spending so long making sure everything they deliver is perfectly robust that they forget about their customers? Did they struggle to respond to an earlier incident effectively and need extra help learning how it's done well? "I've never done this before" There will always be a first time, and it's probably better if the first incident you run isn't a critical "everything is down" one. A major incident is stressful enough without having to learn about your organization's response processes at the same time. Game days, where you run a pretend incident in a non-production environment, are great for learning your incident response process (and honing your debugging skills!), but you have to apply that knowledge to something real sooner or later. Even with blameless post-mortems and very well-run incident response processes, I've seen teams who decide that downtime on a key product isn't an incident because it hadn't been down for and probably no one had noticed. that long When joining a team, especially as a more experienced engineer, part of the experience you bring is about the different cultures you've worked in, including how you respond when things go wrong. This might feel uncomfortable, shifting the culture of a team always does, but the payoff is absolutely worth it. Also published here.

Don't Hesitate to Raise an Incident

No dude en plantear un incidente

Nowadays, easy access to data is table-stakes for high-performing companies. Easy access doesn't come for free, though: it requires investment and a careful selection of tools. For young companies like us, the question is how much? And when do you make that investment? Having grown to ten people, several without engineering backgrounds but with strong data needs, we decided 2022 was going to be that time. Two weeks of experimentation and a couple of company workshops later, we're super happy with what we've built. If you're a company with similar needs and care about data access for all, follow this guide and we guarantee you'll get a great result. Architecture overview With data warehouse solutions ( , , ) going mainstream, modern data stacks are becoming increasingly boring - great news if you're starting from scratch! BigQuery Snowflake Redshift In general, you'll want to: Extract data from various tools (your CRM, accounting platform, etc) into a data warehouse Perform data cleaning and transformation on the source datasets Find a visualization tool that can use the warehouse for exploration and insight Finding complementary tools is difficult, and we spoke to data engineers from several companies to find what was working for them (and what wasn't). We arrived at the following stack: On the left, you'll see data sources (only a sample, we have many more than this), tools where we generate and work with data. One of them is Product, the Postgres database that powers incident.io, while and are SaaS we use to run our business. Intercom Close and are both ETL tools designed to transport data from various sources into destinations. We use both to load data into BigQuery, our data warehouse. Fivetran Stitch Once in BigQuery, we clean and transform the data using a tool called . This allows us to turn raw source data into an ideal, 'data warehouse native', format that can be easily used for analysis. dbt Finally, provides a visualization and analytics layer, running queries against BigQuery. Metabase That's it, the entire data stack. We think these tools are all we'll need to grow incident.io into a much larger company, and given one of our goals (always, for everything!) is simplicity, we're happy the list is so small. Let's take a more detailed look at how this works. Extraction: Fivetran and Stitch Modern companies produce a huge amount of data across an increasing number of tools. While each tool provides a custom view of its data, I'm a that combining datasets unlocks more value than the sum of their parts. We do this by consolidating our data into a single warehouse, which means we can query across all our tools and build insights that combine data in interesting ways. strong believer It used to be that companies would write their own hacky scripts to perform this extraction - I've had terrible incidents caused by ETL database triggers in the past, and even generic ETL tools myself. built a few From experience then, believe me when I say you don't want to build these. Thankfully, ETL products like and run and maintain these extraction processes for you. Fivetran Stitch In both Fivetran and Stitch, the process is the same: Pick connector to access the source data (your Postgres database, or CRM) Configure it with authentication and choose which data to sync Have the data pushed into the warehouse, usually under a dataset-specific to the source Here's a view of the connectors we have configured in Fivetran: It's worth noting the variety of sources. If you're building a SaaS product, one of the most crucial data sources will be whatever database powers your product. has a monolithic application powered by Postgres, in which we store tables like: incident.io , a row for each organization using the product organisations , each user for an organization users , every incident managed in the product incidents This data literally our product, and your equivalent is likely the most high-value data asset to push into your warehouse. is Your database will limit how you can replicate that data, but Fivetran provides connectors that cover most circumstances - as an example, our , so Fivetran falls back to , instead of reading WAL. Heroku Postgres lacks logical replication support periodically scanning tables and using xmin as a cursor Fivetran will automatically manage the warehouse schema on your behalf, creating tables within the destination dataset for each table in the source. As an example, here's a view of the BigQuery datasets from our production database: Other data sources like your CRM will be powered by API access to that tool. For those connectors, Fivetran will paginate through API resources using a cursor that is set each run, taking those responses and pushing them into the destination. Expect this data to be of low quality, or awkward to use without some additional effort. Translating API resources to a data warehouse format needs some human effort, and that's where our transformation step comes in. Why two tools? Until now, you were probably wondering why we have two ETL tools. Fivetran looks great, can't we just use that? The answer is we'd like to, as Fivetran has been exceptional where we've been able to use it, and the pricing is amazing - I think our bill is about $100/month for all these connectors, which is a bargain for the value we get from it. We're limited by connector availability, though. Stitch is a similar product to Fivetran but is built on the open-source , a huge collection of data taps (sources) and targets (destinations). Singer project With Singer as the underlying technology, Stitch has a more comprehensive offering of connectors than Fivetran. Of the two tools, only Stitch has support for (our CRM) which meant we're forced to run both products, if only for a single connector. Close Inconvenience aside, I don't think this is a bad strategy. The goal of ETL tools is to get your data into the warehouse, and provided that happens in a reliable fashion, it shouldn't matter too much how it got there. For that reason, our policy is “prefer Fivetran, but Stitch otherwise”. It works for us! Transformation: dbt Now we have our data in the warehouse, we need to transform it into an appropriate form for querying and analysis. That means turning raw source data into recognizable business concepts, along with tidying artifacts of the ETL process (removing deleted rows, etc). There is an obviously correct choice for tooling at this stage, which is . Having built a huge community of avid data engineers, dbt is the solution for working with data warehouses, and should be consistent with the rest of the industry. dbt While the tool choice is obvious, how to use dbt is going to be more controversial. There's a load of on dbt best practices, but as you can see from my , there's enough ambiguity to tie you up. great resources Slack questions We've arrived at a structure that minimises the number of dbt models (and their schema files, which is the real maintenance burden) and categorises tables by the quality and intended usage. What follows is taken from our “Intro to data” tutorial: What data should I use? Not all data is created equal, and it's useful to understand the different types of model you can query from Metabase, so you can make best use of it. Firstly, the data we expect people to be querying is what we produce from dbt. All tables produced by dbt are located in the dataset in Metabase. All (dbt) This dataset contains many tables, and each table will be either: Dim ⇒ dimension table Fct ⇒ fact table Stg ⇒ staging table In general, prefer to use (in this order): Dimension (dim) and fact (fct) tables Normally described as 'marts', these tables represent key business data and have been engineered for ease of use when querying. An example of a is . In general, dimension tables have a single row per entity they describe (ie, an organisation) and have a large number of columns that can be used to filter and group those entities. dimension table dim_organisations Our table has columns like: dim_organisations and , identifying the organisation id name , whether they have completed setup in_setup_flow , if there is an active Stripe subscription is_paying , the ID of the Lead in Close (CRM) lead_id Note that most of the hard work to enrich an organisation with data from other sources (such as Close) has already happened, making it easy to filter on any of the available dimensions without complex joins. are complementary to dimension tables, and can be seen as a list of things that happened (also referred to as an 'event stream'). Where dimension tables have one row per entity, you'll find many rows (events) for each of those entities in a fact table, and you'll be expected to join against the relevant dimension table to perform filtering. Fact tables As an example, we have an table. This contains a row for each day an organisation has existed, along with a number of measurements like total Slack users, number of incidents, count of users who have created incidents, etc. fct_organisation_statistics These are the highest quality data tables we offer, and you should prefer to use them when they exist. Staging (stg) tables Source data, which is what Fivetran and Stitch generate, is unstable and often awkward to use. As an example, the source Close data has no concept of opportunities, as those exist within the lead data model and aren't modelled as first-class objects. Instead of having people query the awkward source tables, we choose to build from the source data which: staging tables Renames columns to be consistent (all true/false columns start with or ) is_ has_ Casts columns to the right type (timestamp strings are parsed as timestamps) Extracts deeply nested data into a suitable form (custom fields become columns) And whatever else is needed to turn the source data into the 'ideal' data for querying. Going back to our Close example, we have the following tables: which cleans the source lead data and makes custom fields easy to consume stg_close__leads which is built entirely from the nested opportunity data in the leads source, presented as you would want to query it stg_close__opportunities Fallback to staging data when there isn't a dimension or fact table that can better serve your use case. Working with dbt Those last few sections are from our data tutorial, and is how we teach choosing the right data tables. It's normal that this mirrors how we structure our dbt repository: dbt/models
├── marts
│   └── core
│       ├── core.yml
│       ├── dim_incidents.sql
│       ├── dim_organisations.sql
│       ├── dim_users.sql
│       ├── fct_organisation_statistics.sql
│       └── fct_user_incident_graph.sql
└── staging
    ├── close
    │   ├── README.md
    │   ├── src_close.yml
    │   ├── stg_close.yml
    │   ├── stg_close__activities.sql
    │   ├── stg_close__leads.sql
    │   └── stg_close__opportunities.sql
    └── product
        ├── README.md
        ├── src_product.yml
        ├── stg_product.yml
        ├── stg_product__actions.sql
        ├── stg_product__announcement_rules.sql
        ├── ...
        └── stg_product__workflows.sql While I was speed-running dbt tutorials, I would have loved to see this structure, to lend some confidence we were headed in the right direction. With that caveat that I'm only an amateur dbt'ist, it's worth noting some decisions that went into this structure: You'll see mentions of in dbt literature, described as the first transformation after source and appearing before staging. We've opted not to create these, and to go straight for staging tables - this avoids maintaining another layer of dbt schemas, and lots of our data (especially Product) is close to staging format at source anyway. 'base' tables We don't produce marts from staging models unless we join them across schemas (as with the dimension tables) or perform complex transformations on them (fact tables). Again, this is to reduce the number of dbt schemas. We only have core marts right now. Until we add more marts, it makes little sense implementing a more complex structure (such as grouping marts under business units), better to adapt as we grow. In terms of developing against dbt, each engineer has a separate BigQuery dataset (ie. ) that they target in local development for testing. Running a full build ( ) is quick right now, making local development enjoyable. dbt_lawrence dbt build While this setup makes trade-offs appropriate for incident.io's context, I think anyone setting up a data stack could adopt this with little issue, and it's easy to tweak things once you're more familiar with the tooling. Visualization: Metabase Now our data is cleaned, transformed and shaped into an ideal format for analysis, it's time we pick a visualisation tool. This choice is crucial, as whatever we pick will be how people interact with our data. One of our key goals for this stack was for non-technically trained staff to benefit, and past experience with tools like Looker suggested that was possible only with significant help from a BI team, which we wanted to avoid. We picked as the most intuitive of the options, with visualisations that can be built without any SQL knowledge. Metabase Whether a tool is intuitive is subjective, but Metabase certainly feels that way to me. As an example, let's try and chart the number of incident timeline items created per month and see what that process looks like: Things to notice: No SQL was required, even though we built the visualization from scratch (not from an existing question). Joining the Stg Product Timeline Items table with Stg Product Organisations was really easy, in part because Metabase can automatically determine join keys (more on this later). The interface was super clean, much closer to a wizard or existing spreadsheet tools than the traditional data toolchain. All of this helps immensely with onboarding, especially for non-technical staff. Syncing dbt schemas to Metabase Metabase can really help lower the barrier to accessing your data, but it's limited by how well it knows the schema. One example is visible in the previous screen recording, where joining Stg Product Timeline Items with Stg Product Organisations prefilled the join key (Organisation ID). Metabase can only default this if it knows Organisation ID on Stg Product Timeline Items is a foreign key of Stg Product Organisations, and which field it references. As it happens, our dbt schemas already know this information, through column tests. Here's the schema for the field of Stg Product Timeline Items: organisation_id ---
models:
  - name: stg_product__timeline_items
    columns:
      - name: organisation_id
        description: "Organisation ID"
        tests:
          - not_null
          - relationships:
              to: ref('dim_organisations')
              field: organisation_id Whenever we have a column that specifies a relationship test, we can infer a foreign-key relationship against the parent table. So how do we get this into Metabase? There's a tool called that can infer Metabase semantic type information from the dbt schema and push it into Metabase- we run this whenever complete a dbt build, helping sync Metabase with whatever new fields we may have added. dbt-metabase Inferring foreign keys is just one use of this tool, though - you can manually specific column types too. As example of where this can be useful, annotating columns as a 'name' allows Metabase to power typeaheads for that entities primary key by the name value, such as using a customer name in a customer ID filter. Here's an example of setting the type: columns:
  - name: name
    decription: "Name of the organisation"
    meta:
      metabase.semantic_type: type/Name
  - name: archived_at
    decription: "When the resource was archived"
    meta:
      metabase.semantic_type: type/DeletionTimestamp
  - name: opportunity_annual_value
    meta:
      metabase.semantic_type: type/Currency
    description: "If an opportunity event, annual value in in normal currency denomination" Onboarding and rollout That covers the entire stack. As a round-up: and to extract data into the warehouse Fivetran Stitch BigQuery to perform cleaning and transformations to aid analysis dbt for visualisation, with [dbt-metabase][tools/dbt-metabase] for syncing the schema Metabase It took about two weeks to figure all this out and get the bulk of our data sources exposed using dbt, and exposed in Metabase. That was a single engineer who hadn't ever used dbt before, and having to make a lot of these decisions on the fly. I expect someone following this guide could do it much faster. Once we had the stack up and running, we ran two workshops aiming at different audiences: , for engineers who will build dbt models. Intro to dbt We can't hire a full-time BI team just yet, and think we can get by with engineers surfacing more complex data features to Metabase by defining them in dbt. This session covered our dbt setup and we mobbed on adding a feature to a dbt model, manually running the schema sync into Metabase. , for the entire company. Intro to data Everyone is expected to use data in their work, so everyone needs to be familiar with Metabase. This session included a rundown of the stack, then working as a group to visualize a few key business metrics. Finally, we took a list of questions we've been dying to have answered and divided it among us, with people trying to answer them in pairs. Both workshops went great, but the entire company session was my favorite. Two weeks is an eternity in a company like ours, and most of my decisions had been made with clear goals but little experience. It was a big moment, then, when someone with no SQL experience came to show me a conversion funnel for our product tutorial, and it was exactly right: Parting thoughts Data is a currency at most companies, and it's extremely important that everyone has equal access to it. It's easy – especially when companies begin with highly technical teams – to end up with a few people who know all the answers, undermining the ownership and autonomy of those outside that group. I'm really happy with our current setup, primarily because it provides equality of access that I've not seen before. Finally, it's worth reiterating a point that can easily go unnoticed: this stack makes it easy to consolidate data from various sources into a single warehouse, which means you can combine it to unlock significantly more value than the data offers when separate. Connecting this data means your product activity can be joined with your Sales tools, and your customer success reports can take into account things like how often a customer reached out on Intercom, or if they're in arrears in Stripe. What this post describes are foundations for capitalizing on this industry-wide change. If you're a small company looking to punch above your weight, you'd be mad not to take that opportunity. Also published . here

We Built a Modern Data Stack for Startups

When building an incident response process, it's easy to get overwhelmed by all the moving parts. Less is more: focus first on building solid foundations that you can develop over time. Here are three things we think to form a key part of a strong process. I'd recommend taking these one at a time, introducing incident response throughout your organisation. . We believe that using our software will help build a good incident process, otherwise, we wouldn't be doing it. But beyond that, we've also got a lot of experience building and participating in incident processes and think this advice is generically useful, regardless of whether you choose our product. Just being transparent: we're a startup providing incident management software - incident.io 1. Define what an incident is This is, in the words of Julie Andrews, a very good place to start. Getting a common understanding of what an incident is (and isn't) is the first step in bringing people into your incident response process. An incident is where something unexpected happens that has (or might have) a negative consequence. Particularly when you're getting started, the best way to embed a process into your organisation is to use it. A lot. This also helps everyone learn the process, and get better at incident response overall, meaning that when something really bad happens it feels like a well-oiled machine. To this end, you want a broad and inclusive definition of an incident: Declaring an incident should be easy, and everyone should be able to do it without fear of repercussions. Avoid long questionnaires which add friction to the process, and de-incentivise declaring incidents at all. Don't assign blame (or other negative consequences) to declaring an incident. This includes not using 'number of incidents' as a metric for team health or performance. Your lowest severity should represent something that happens regularly and has a low impact, so people feel comfortable declaring incidents (and don't feel like a boy crying wolf) Don't limit the definition to engineering: getting your whole org to think about anything unexpected in the same way means you can collaborate better (see section 2) 2. Keep everyone updated Transparency by default is a really important value to bake into your incident process. First up: make sure it's clear who is responsible for communicating. Whether that's the incident lead, or another chosen individual, making some explicitly accountable is the best way to keep the updates coming. Make it really easy to tell stakeholders what's going on, and use the tool that makes the information easiest to consume (whether that's email, Slack or something else entirely). Use a predictable format for the updates, as this makes them easier to parse and scan if you're a busy stakeholder flying through their notifications. These updates also advertise your incident process and normalise the fact that incidents happen. Ideally, someone's first interaction with the incident process at your org should be as a consumer of an update --- not being parachuted into the middle of something. Get comfortable admitting that things go wrong, both internally and (to some extent) externally with customers. This builds trust and enables people to adapt their behaviour to mitigate the impact on their side (e.g. if a customer knows you are having an outage, they'll move to another task and come back tomorrow instead of furiously refreshing the page). 3. Remember: it's not over until it's over Your incident process shouldn't end once the problem is resolved. To get value from your incidents, you want to be using them to learn and improve your day-to-day operations. There are often follow-up actions that need to be put on someone's backlog or wider problems that should be considered and prioritised. Post-mortem documents and incident reviews are a great way to extract learnings from an incident. They help encourage reflection and often bring up related concerns about the way the team is working. However, there's a Goldilocks' zone. If the post-incident process feels painful, people will stop declaring incidents at all. Clearly communicate the value of your post-incident process, making sure it doesn't feel like a box-ticking exercise. Don't make it harder than it needs to be: try to avoid repetitive manual work (e.g. using a tool or template). Give people autonomy to decide what's appropriate for the incident they've experienced. Sometimes a simple update in the incident channel explaining what happened and why is completely sufficient. Sometimes you'll want to run a full-blown cross-team workshop to understand what happened and why the response didn't go as smoothly as you'd hoped. Summary: Building an Incident Response Process When you're first building an incident response process, focus on a few key things: Get a shared understanding of what an incident is, and don't make it too specific Make it easy to provide updates, both to internal and external stakeholders Reflect on incidents to gather learnings, but make sure the process is adding value Once you've got these nailed, you can start layering more stuff on top. But don't try to run before you can walk.

The 3 Tips to Creating a Solid Incident Response Process

For the vast majority of organizations, it's necessary to have some form of round-the-clock cover to support the business. Whilst it's most commonly a concern for engineering, it's increasingly common to have folks from various disciplines available out-of-hours. Irrespective of role, compensating people fairly is an important factor of running a healthy and effective on-call system. In this guide, we'll cover the various options for paying folks who are on-call, along with some advice, trade-offs, and pitfalls to avoid. And ps, if happen to be on-call and dealing with incidents, check out --- out-of-the-box IR automation tool for on-call folks, that handles the tedious manual processes on Slack during an incident, making responding quick and easy so that you can focus on fixing the problem. you incident.io A couple of terms worth knowing Someone who's the first point of contact when something goes wrong. Most commonly on-call is linked to out-of-hours work. On-call: We used to carry actual pagers back in the day, but these days the phrase 'holding the pager' really means being the first person whose phone will be called when something goes wrong. Pager: Common Approaches to on-call Compensation There are a number of different approaches to on-call comp... 1. No pay for on-call Many organizations don't pay anything for on-call. The folks taking part in an on-call rotation are doing so out of the goodness of their heart, or because they've been asked (or forced!) to do so. This approach is particularly prevalent in smaller companies where incidents aren't especially severe, or the impact of downtime isn't considered a significant concern. It's also something we see in much larger organizations, where historical norms prevail and folks feel less empowered to ask for change. If you're considering this approach to compensation, it's worth keeping an eye on how people are feeling over time. There'll come a point where the impact on home life is meaningful enough that they expect to be compensated for the inconvenience. Left unchecked, it's easy to end up with burnt-out or undervalued people who may look for opportunities elsewhere. 2. Pay as part of your base salary In this model on-call compensation is positioned as part of your base salary, and is commonly combined with expectations around how often you'll be required to work out-of-hours. For example, you might have a base salary of $100k and an expectation that you'll be on-call no more than one week in eight. There's a clear appeal to this approach as it's a simple mechanic for paying folks without any complex monthly compensation calculations. There are a few trade-offs to consider, however. Firstly, this model prevents folks from trading time without feeling indebted to each other. Since everyone is compensated equally, overrides are highly transactional; "I'll take your week, if you take mine". Additionally, people have a tendency to forget these kinds of agreements exist, and over time they may forget that they are being compensated at all. And since it doesn't feel like they're being paid 'extra' for that out-of-hours time, there's a risk they'll begrudge doing it. 3. Pay per incident With a pay-per-incident approach, on-callers are paid a fixed amount for every incident they attend. Typically it'll apply to out-of-hours only, but every time you open your laptop to investigate an issue you were paged for, you get rewarded. It's clear to see where the positives are here: pay is correlated with the amount of active pain you feel, so the more you're disrupted the more you're compensated. A bad week of incidents is made a little less bad when you pocket a nice bump on your paycheck. There are clear drawbacks to this method though, and in the worst case you can end up disincentivizing work to make systems more reliable. If you're going to be paid every time an alert fires, and you don't mind being woken up, why take the time to fix things? This is clearly an extreme, and few people are likely to deliberately leave things in a poor state, but while the incentive structures are in place the risk is there. Another failure mode is the fact that no incidents mean no pay. It might sound reasonable on the surface, but when being on-call means you need to carry your laptop with you for a week, that's an inconvenience you're giving away for free. 4. Pay by time This approach works by paying folks solely for the hours they spend on-call. Most commonly this is run as a weekly rotation, where on-callers are paid a fixed amount for the time they're 'holding the pager'. In an ideal world, the compensation is worked out down to the minute taking overrides into account. In this world, when someone takes over part of your shift for you, they're automatically compensated for their time which removes the pressure to trade shifts. Ultimately this approach pays folks for the on their home life, rather than their time directly. If you're paid $300 for a full week of on-call, you're essentially being compensated at a rate of $1.78 per hour. Not an amazing rate by any standards, but if it's just covering you to carry a laptop around it's a pretty fair exchange. inconvenience As with the other approaches, paying by time also comes with some trade-offs. Firstly, it doesn't work well when someone is on the receiving end of a bad week of on-call: that $300 is going to feel like a rough deal when you've been woken several nights in a row. For this reason, it's helpful to pair it with an environment of proactive overrides: if someone's been up in the night, offer to give them a 24-hour break. The other failure mode is fairness when there's an imbalance of pager-load between teams. If I'm paid $300 per week and my team receives double the number of alerts of another, I'm likely to feel aggrieved. There's no easy solution here, but one option is to consider different rates for 'high' and 'low' load teams. Our advice on Getting Started There's clearly no shortage of options for compensating folks who are on-call, but you might be wondering which is best. As with most things, the most accurate answer is "it depends", but for the majority of people, paying by time is a sensible default. Below are a couple of example models which incorporate this approach. A single rotation in smaller Organisations In smaller organisations, like , it's likely that a single rotation will have you covered. In this world, you're in easy mode, and you can pick a compensation amount that feels fair based on the criticality of your service and the typical frequency of incidents. incident.io For us, we pay ~$350 per week and expect to be paged no more than once or twice (and sometimes never!). We've written about our setup in more depth in . On-call by default A more sophisticated Default for larger Organisations When I worked at Monzo -- a bank here in the UK -- a large proportion of the company was on-call. Things were a little more complex and we wanted to structure our compensation to cover a few different scenarios: Critical teams who needed to be available at very short notice Teams with lower alert load, or who could afford to respond with less urgency Fairly compensating folks who weren't on-call, but were opportunistically pulled into incidents To account for these we had two levels for weekly compensation; one where a team would be paid a higher rate in return for a stricter SLA, and another where a team would be paid slightly less in return for a slightly relaxed response time. Since you're paying folks for the inconvenience on the home life this approach feels justified. If you're on a critical rotation and need your laptop with you at all times, it's fair that you're compensated higher than someone who can leave it at home whilst they head out for groceries. To account for folks who were opportunistically pulled into incidents, we introduced the concept of 'sideways escalations'. If you were on-call and needed someone else who wasn't, you could page them to help out and they'd be paid a fixed amount in return. Since their involvement was opportunistic, and the only inconvenience was for this event in particular this approach felt fair. Moreover, it reduced social barriers to escalation as you knew the person you were paging would be compensated for their involvement. If you're interested in reading more, Monzo have written about their on-call setup and . here here ℹ️ If you're planning on implementing time-based compensation and you're using PagerDuty to manage your on-call, this API call can grab you all the time slots that folks were on-call on a particular schedule, taking overrides into account. From here's it's simple to import into a spreadsheet and multiply the time on-call by the pro-rated weekly amount. curl --request GET \\
 --url '<https://api.pagerduty.com/schedules/><SCHEDULE>?since=2022-01-01T00:00:00+0000&until=2022-02-01T00:00:00+0000' \\
 --header 'Accept: application/vnd.pagerduty+json;version=2' \\
 --header 'Authorization: Token token=<TOKEN>' \\
 --header 'Content-Type: application/json' | jq -r '.schedule.final_schedule.rendered_schedule_entries[]|[.start, .end, .user.summary]|@tsv'

**From To Person**
2022-01-01T00:00:00+00:00	2022-01-02T08:00:00+00:00	Pete Hamilton
2022-01-02T08:00:00+00:00	2022-01-03T08:00:00+00:00	Chris Evans
2022-01-03T08:00:00+00:00	2022-01-04T08:00:00+00:00	Sophie Koonin Getting started There's no such thing as a one-size-fits-all approach to on-call compensation. What you pay is heavily context-dependent and will vary based on your business, your incident load, the people who'll be on-call, and a number of other factors. Overall, you'll struggle to go too wrong if you keep things simple, pay people fairly, and remain cognisant of the impact out-of-hours work can have. In 2019, @evnsio and @spikelindsey surveyed over 300 people to understand how they're compensated for on-call. The full dataset of responses can be found . here Also Published here

INCIDENT.IO

incident.io

-- employees

Since n.d.

COMPANY RANKING

INCIDENT.IO

EVERGREEN INDEX #5443

Representing this company? Click here to claim and customize this page!

incident.io WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

INCIDENT.IO

incident.io

-- employees

Since n.d.

COMPANY RANKING

INCIDENT.IO

EVERGREEN INDEX #5443

RELATED COMPANIESRANK

STORIES ABOUT INCIDENT.IO

Representing this company? Click here to claim and customize this page!

incident.io WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!