Rethinking Logging: Log Less, Learn More

The Logging Dilemma

Logs can be our lifeline when things go awry. To solve a problem, we must reproduce the problem, and debug logging can be invaluable. But there's a catch - the costs. As applications scale, so do the costs of ingesting and storing our logging, and this forces us into a tradeoff between logging too little to be useful or paying for logs we’re not using.

Thom Shutt once made an argument for logging only errors, suggesting that warn, info, and debug levels be discarded. His rationale? The difficulty in changing log levels in production:

Throw away Levels - When was the last time you ran Production at anything other than Info? Is changing the level a quick process that everyone on the team knows how to do?

And indeed, most of us don't change log levels often because it requires a commit, a build, and a deployment. If changing the log level is a tedious process, these additional lines do indeed become mere noise.

The Power of Dynamic Configuration

Enter dynamic configuration. While many might be unfamiliar with it, those who've embraced it can't imagine life without it. Unlike static configuration, where apps read a property file or an environment variable, dynamic configuration is, as the name suggests, dynamic.

It offers the ability to set new values through a UI, ensuring each evaluation retrieves the latest version.

This technology is the backbone of dynamic logging. By storing the Logger -> LogLevel mapping in our config system and updating the logger on changes, we can shift from warn to debug in milliseconds, not hours.

Introducing Prefab: The Future of Logging

Prefab is revolutionizing the way we approach logging. With Prefab, you can:

• Real-time Log Level Changes: Adjust log levels on the fly. No need to wait for the server to restart. Change and you're done.

• Targeted Logging: Prefab's targeted logging lets you see debug logs for just a single user, transaction, or availability zone, enhancing precision and reducing noise. Prefab allows you to focus on the relevant code, job, or even a specific user.

• Integration with Existing Systems: Prefab isn't another logging tool. It integrates seamlessly with your existing logging provider, be it DataDog, Sumo Logic, or an ELK stack.

Saving Money On Logging

The costs of logging gets serious at scale, and are all tied to volume. The aggregators are offering new ways to try to reduce this spending, like Datadog's Log Without Limits, and Cribl is basically a whole company devoted to this problem.

But all of these solutions charge you based on usage. At the end of the day, you're still paying for the logs you send.

Because your logs don't pass through Prefab, it doesn't charge based on the volume of logs but simply on serving the dynamic configuration to your servers. Prefab simply controls the values of a filter, ensuring you only log what's necessary. This approach can lead to significant savings, especially for large-scale applications.

Redefining the Logging Culture

Adopting dynamic logging is not just about using a new tool; it's a cultural shift. Initially, it might seem daunting. But as teams grow accustomed to the power of targeted logging, it fosters a stronger operational mindset. Developers can now think, "If this goes sideways, what information would I want?" and prepare accordingly.

For instance, consider this code snippet:

def charge(user, amount)
  balance = get_balance(user)
  min_balance = get_min_balance(user)
  do_charge = balance > min_balance
  logger.debug("charge decision", charge: do_charge, amount: amount, balance: balance, user_id: user.id, min_balance: min_balance)
  if do_charge
    charge_user(user, amount)
  else
end

Previously, many would have seen the logger.debug line as mere development noise. But with dynamic logging, such lines become invaluable tools, ready to be activated when needed.

Conclusion

The landscape of logging is undergoing a seismic shift. With tools like Prefab, developers no longer have to choose between detailed logs and cost efficiency. Dynamic logging offers the best of both worlds, providing insights when needed without the associated hefty price tag.

Happy debugging! 🚀