The damaging effects of unplanned work

For practically anyone, unplanned work kills several hours of planned productivity. For creative workers, such as those who write software, it kills days.

A while ago at work, a prospective customer told us that if we could analyze their data in a certain way then they were interested in doing business with us. We have a team that handles ad-hoc analytic requests, but that team didn’t have the capacity at the moment to take the job on. They asked around for help and I that’s how I got involved.

I’ve done a lot of customer-facing analytics over the course of my career and I don’t particularly like it. When the only definition of “done” is “the customer said they were satisfied with the analysis”, you know the scope of your project is going to forever creep until the customer decides to pay attention to something else. Despite knowing this, when I became involved I made the project managers agree to very specific acceptance criteria: I would do X, Y, and Z, and when X, Y, and Z were done, my involvement would be done.

I know my scope definition wouldn’t really matter, but I felt better for having made the effort. I did X, Y, and Z, and every said they were happy. I didn’t didn’t hear anything for a couple weeks, so I went on my with my normal work, only occasionally hearing the the voice of my past experience occasionally telling me that it was just a matter of time. Suddenly, I got an email with new asks from the customer. I’d agreed to X, Y, and Z, but now they wanted A, B, and C. As always happens with these types of projects, my initial scoping meant nothing at all — I wouldn’t be done until the customer said so.

I pretty much expected an ever-expanding scope so I couldn’t feel too frustrated about that. What drove me crazy was my absolute inability to plan for the work. My team work on a sprint schedule, and these unplannable asks preempted my scheduled work multiple times over the course of several sprints. All of my normal commitments got pushed further and further down the calendar because, when a single resource has to handle both product development and customer-facing deliverables, customer-facing deliverables always win.

The whole experience got me thinking about why I feel so strongly about unplanned work. Most managers and executives I’ve spoken to about this issue have held an attitude along the lines of: “we pay for your time, so we can decide what fills that time however and whenever we want.” I think that’s true, but I also think that preempting planned work with unplanned work is bad for business — sometimes bad enough that I’ve seen it grind entire departments to a halt.

The first reason I came up with for hating unplanned work is something I’ve seen a lot of other people mention:

When working on something creative like writing code, you experience different levels of productivity. The most productive levels are what some people refer to as “being in the zone”. You can’t just jump into the zone whenever you want — you have to work less productively for a while until you mind gets used to doing certain things automatically, at which point you can start working more productively. It’s similar to how sleep works:

You don’t just drop from being awake into deep sleep. You have to descend through different stages. If someone wakes you up repeatedly at unpredictable intervals, you’ll have to take time getting back to sleep and even then you won’t fully benefit from the sleep you do get. Likewise, if someone interrupts your regular work repeatedly to pull you onto emergency projects, your regular work is delayed by more than just the time it took to do the side project. It isn’t that the amount of time each day or week not spent on the emergency project isn’t enough time, when taken altogether, to fit in the activities that would complete your regular work. It’s that the interruptions keep you from getting into a productive-enough zone to really make consistent progress.

As soon as I got an email about a new ask, I stopped being productive on whatever I was working on, even if the new ask wasn’t due for another day or even a week. Knowing that I now has a new deadline that I hadn’t planned for forced me to juggle my priorities in order to make sure I could meet that deadline which pulled me out of the productivity that I had been experiencing before I got the ask. Once I rescheduled everything I sometimes had time to get back into my normal work before shifting to work on the new ask, but by then the new ask had already wasted several hours by forcing me to slowly work my way back into a productive zone.

All that being said, as I thought about it more, I realized that being forced out of my productive zone wasn’t the main reason I was frustrated about the unplanned work.

Data science, like many other coding professions, is largely design work. Design work requires a lot of stuff to be kept in your head, and then you periodically move pieces of it out into code, documentation, diagrams, policy, or other forms of collateral. In other words, it’s a creative exercise. Once you have pieces instantiated, you don’t need to keep them in memory any more, but you do have to remember where they came from — all of the parts of the problem that touched the topics that got put into documents but didn’t themselves get included in those documents.

The human brain can’t remember everything, so it regularly consolidates memories to keep only information about things that were rewarding enough or costly enough that the information is likely to be useful to us in the future. While we sleep, our brain goes through our recent memories and edits out information that hasn’t been tied to costs or rewards. (You can find a technical discussion here and a non-technical discussion here).

That’s the problem with interrupted design work. At any given time, the really important stuff — information tied to costs or rewards — is already offloaded into documents. Everything else is stuff we haven’t made up our minds about yet (in other words, our code only contains the methods we know we’ll need, not the methods we don’t yet know we’ll need). So most of the information not yet documented at the time unplanned work interrupts us is information our brains will edit out over the course a few nights of sleep.

I think of it this way: a new project is like a blank sheet of paper. Each time I write some code or documentation, I cut a shape out of that paper, label it, and set it aside. That allows me to quickly reference all the pieces I’ve cut out, so if I need to put the pieces in a different order or arrange them all in a certain way, all I need to do is look at the shapes and labels and I can figure out what goes where.

That’s not the case with the rest of that sheet of paper — the parts I haven’t cut out. Maybe I’ve jotted a few notes here and there or started a few incisions without fully cutting out a new shape. As long as I keep coming back to that paper every day and reminding myself of what I’ve already done and what I thought I might want to do next, I can for the most part keep ahead of my brain’s nightly memory consolidation. But if suddenly I have to stop thinking about the paper entirely because I have to unexpectedly jump over to another project, my brain essentially takes whatever portions of the paper are left uncut and crumples it into a little ball.

When my unplanned work is over and I go back to what I had been working on before the interruption, I don’t get to un-crumple that paper. I either have to cut shapes out of the wadded up little ball, or else I have to pull out a new sheet of paper. If I try to make do with the consolidated memories my brain has left me, I end up designing not for the problem as it actually exists, but for the problem as it exists after passing through my brain’s filters. If I give up on using my brain’s leftovers and try to design from whole cloth, I have to pull out all the pieces I have already designed to make sure I don’t design obviously redundant or contradictory solutions. Either way, I have to spend huge amounts of time (days or even weeks, compared to the hours I wasted from having to get out of my productivity zone) re-learning the scope and shape of the problem.

Now, I have to admit, that sometimes memory consolidation works to my benefit. I’ve written before about the benefits of stepping away from a problem in order to let my brain consolidate its memories and, if I’m lucky, leave me with a cleaner view of what it is I really need to do. But just because memory consolidation is good at some times doesn’t mean it’s good at any given time. If I’ve fought a problem for months and can’t seem to find a satisfactory solution, then it’s probably good to step away. But if I’m forced to step away at unexpected intervals and each time for an unforeseeable duration in order to accommodate the latest office emergency, there’s no reason to expect that my brain will be full enough of information about my project to have the consolidation of that information be at all useful.

For practically anyone, unplanned work kills several hours of planned activity every time it happens. For creative workers such as data scientists and engineers, it kills days. Even a “quick ask” carries huge costs in terms of productivity and lost opportunity. Teams that recognize this, and structure their priorities accordingly, get more done.