How I Learned to Stop Worrying and Love Technical Debt

It started with the Slack channel aptly named “wtf”. Jamal stared at his screen, perplexed by the convoluted SQL query he was trying to decipher. ”Hey @channel, what in the actual f*** is this mess?” he typed into the channel, pasting the 800-line query. The query twisted and turned, joining tables in a mind-bending maze. He could imagine a big red button, warning “Do Not Touch!” The team used the “wtf” channel to bond over the legacy code they had inherited. It contained years of contributions from countless developers, many of whom were lost to the sands of time. The codebase was the nucleus of the company, but sometimes it felt like an artifact from the Jurassic period.

Ayse chimed in instantly. “Ugh, that part of the code is the stuff of nightmares. Let’s just say Vladimir was ‘creative’ with his use of SQL and ORM libraries.”

Jamal smiled. Ayse always managed to say what everyone was thinking. The two of them had bonded over their mutual hatred of Vladimir’s legacy code in the past. He quickly typed a reply, “I’m so down. Want to team up on Operation: Vladimir’s Revenge?” Ayse responded with a gif of Rosie the Riveter flexing her muscles. “Let’s do this!”

“I know that area of the code has gotten out of hand. Vlad’s libraries were state-of-the-art back in the day, but modern ORMs would make this so much cleaner. It’s well past due for a refresh,” added Shirley, their tech lead.

Ayse and Jamal volunteered eagerly to lead the migration to a modern ORM. As the resident SQL experts, they dove into researching options.

“Sequelize is way too bloated! We need something lightweight like Knex,” argued Ayse.

Jamal shook his head. “But Sequelize has way better documentation and support. We’d be shooting ourselves in the foot not to use it!”

Back and forth they went, each arguing passionately for their preferred tool. But finally they reached an agreement - for this massive undertaking, Sequelize’s popularity tipped the scales. They dove into the work, heads down, squashing bug after bug.

The next step was mapping out just how deep the Vladimir mess went. Ayse and Jamal worked late into the night, tracing dependencies, untangling convoluted logic, and building a checklist. When they finally compared notes, their eyes widened in disbelief. “86 files? Seriously?” Jamal groaned. But their resolve only strengthened. They would be heroes who slew the Vladimir dragon and rescued the codebase from its own cruelties.

Over the next month, Jamal and Ayse carefully executed the plan. Their Slack banter kept the mood light even when they encountered gnarly tangles of code. The work was meticulous, but they kept chipping away. Ever since Vladimir had left the company years ago, his clunky home-brewed ORM had languished, accruing technical debt like layers of dust.

Finally the day arrived. Jamal took a deep breath and clicked “Open Pull Request”, merging hundreds of commits into one massive change. Ayse gave him a high-five. “This is it - the moment we’ve been waiting for!” The PR linked to their checklist, now fully green.

Shirley felt uneasy reviewing such a massive PR. As the tech lead, she knew it was her responsibility. But each time Ayse and Jamal presented it, she found an excuse to postpone. “Let me review it over the weekend,” she would say, or “I need another day to properly vet this much code change.” Deep down, Shirley feared something might slip through the cracks in their refactor. The PR loomed over her like a ominous storm cloud that she couldn’t bring herself to confront just yet.

They decided to have the team review the PR thoroughly before merging to production. Smaller changes were cherry-picked so progress wouldn’t stall entirely. The next few weeks were a flurry of comments, reviews and re-reviews. Merge conflicts flooded the “wtf” channel as other team members integrated their work. Jamal and Ayse patiently rebased, squashing bugs as they went.

Finally, it was time for the big merge. On Friday afternoon, they all gathered around Shirley’s computer for the climactic moment. With a deep breath and closed eyes, she clicked merge. They all stared, waiting for the green checkmark of success. Finally, it flashed onto the screen. “Huzzah!” cheered Ayse and Jamal, exchange a high-five. They were giddy with excitement to see their work go live.

But just twenty minutes later, the PagerDuty alerts started dripping in. Ayse and Jamal stared at their phones, smiles fading. “No no no, this can’t be happening,” Ayse said, frantically scrolling through the alerts. Errors were cascading through the system. Their faces turned ghostly white.

They rushed back to the office, hearts racing faster than their cars. Jamal sprinted to his computer, pulling up dashboards. Errors everywhere.”How did this happen? It was working perfectly!” Ayse said, fists clenched. They worked feverishly through the night, reverting code deploys, digging into log files.

Reverting such a massive change that touched so many parts of the codebase turned out to be an even greater challenge than they anticipated. Simply rolling back the merge didn’t undo all the downstream effects of the refactor.

Jamal and Ayse worked through the weekend trying to unravel the complex knot of code. But with each file they reverted, new bugs seemed to emerge. Key parts of the system no longer functioned as expected. They raced to push out fixes while also trying to rollback their work, but it was like trying to unbake a cake.

By Monday morning, they were exhausted and no closer to resolving the issues. The codebase was now in an even more fragile state. After an emergency meeting, the team made the difficult call to pause all new feature work and rally everyone to help “unmerge” the refactor. It took two entire days of around-the-clock debugging to roll it back fully and restore stability.

As they combed through logs and code, Jamal had an idea. “What if we search for all conditional usages of the ORM and see if any were missed?”

He searched for all conditions used across the codebase. Most looked syntactically correct. But then Ayse spotted one without a space after the if.

“Wait, show me that one again,” she said. Jamal pulled it up and her eyes widened. “Of course, our regex is looking for if followed by a space. But it’s valid JavaScript to omit that space.”

They dove into the specific file and saw the problem immediately. That single if statement without a space had been overlooked in their rewrite. And because it was calling an important validation function, its error was cascading downstream.

The offending regex was tamed and submitted back into the codebase. With the bug now fixed, Jamal excitedly opened a new pull request to merge their changes.

However, Shirley was still hesitant to approve it. “I know you both worked really hard on this, but I’m just not confident enough yet to merge it,” she said. “As annoying as it is, another bug could still be hiding somewhere. Perhaps we should put this pull request on hold for now and turn our attention to other priorities. We can revisit it again in a few sprints after some of the pressure has eased off this feature.”

Jamal pulled out his phone with a sigh as he and Ayse closed the unfinished pull request. “Welp, I suppose it’s time to memorialize this bug fix that will never be,” he said wryly. He opened the team’s “wtf” channel and typed out:

“#wtf guys, I think we just found THE bug to end all bugs in Vladimir’s code. Took us weeks to track down but we’ve got a 100% fix. Too bad it will now live in purgatory forever thanks to bureaucracy! This one’s going down in history as the most perfectly solved yet completely undispatchable issue ever. You’re welcome future developers!”

[Next part of the series: Killing One Bug with Two Stones]