The Logging Dilemma Logs can be our lifeline when things go awry. To solve a problem, we must reproduce the problem, and debug logging can be invaluable. But there's a catch - the costs. As applications scale, so do the costs of ingesting and storing our logging, and this forces us into a tradeoff between logging too little to be useful or paying for logs we’re not using. Thom Shutt once made an argument for , suggesting that warn, info, and debug levels be discarded. His rationale? The difficulty in changing log levels in production: logging only errors Throw away Levels - When was the last time you ran Production at anything other than Info? Is changing the level a quick process that everyone on the team knows how to do? And indeed, most of us don't change log levels often because it requires a commit, a build, and a deployment. If changing the log level is a tedious process, these additional lines do indeed become mere noise. The Power of Dynamic Configuration Enter dynamic configuration. While many might be unfamiliar with it, can't imagine life without it. Unlike static configuration, where apps read a property file or an environment variable, dynamic configuration is, as the name suggests, dynamic. those who've embraced it It offers the ability to set new values through a UI, ensuring each evaluation retrieves the latest version. This technology is the backbone of dynamic logging. By storing the Logger -> LogLevel mapping in our config system and updating the logger on changes, we can shift from warn to debug in milliseconds, not hours. Introducing Prefab: The Future of Logging is revolutionizing the way we approach logging. With Prefab, you can: Prefab : Adjust log levels on the fly. No need to wait for the server to restart. Change and you're done. Real-time Log Level Changes : Prefab's targeted logging lets you see debug logs for just a single user, transaction, or availability zone, enhancing precision and reducing noise. Prefab allows you to focus on the relevant code, job, or even a specific user. Targeted Logging : Prefab isn't another logging tool. It integrates seamlessly with your existing logging provider, be it DataDog, Sumo Logic, or an ELK stack. Integration with Existing Systems Saving Money On Logging The costs of logging gets serious at scale, and are all tied to volume. The aggregators are offering new ways to try to reduce this spending, like Datadog's , and is basically a whole company devoted to this problem. Log Without Limits Cribl But all of these solutions charge you based on usage. At the end of the day, you're still paying for the logs you send. Because your logs don't pass through Prefab, it doesn't charge based on the volume of logs but simply on serving the dynamic configuration to your servers. Prefab simply controls the values of a filter, ensuring you only log what's necessary. This approach can lead to significant savings, especially for large-scale applications. Redefining the Logging Culture Adopting dynamic logging is not just about using a new tool; it's a cultural shift. Initially, it might seem daunting. But as teams grow accustomed to the power of targeted logging, it fosters a stronger operational mindset. Developers can now think, "If this goes sideways, what information would I want?" and prepare accordingly. For instance, consider this code snippet: def charge(user, amount) balance = get_balance(user) min_balance = get_min_balance(user) do_charge = balance > min_balance logger.debug("charge decision", charge: do_charge, amount: amount, balance: balance, user_id: user.id, min_balance: min_balance) if do_charge charge_user(user, amount) else end Previously, many would have seen the line as mere development noise. But with dynamic logging, such lines become invaluable tools, ready to be activated when needed. logger.debug Conclusion The landscape of logging is undergoing a seismic shift. With tools like , developers no longer have to choose between detailed logs and cost efficiency. Dynamic logging offers the best of both worlds, providing insights when needed without the associated hefty price tag. Prefab Happy debugging! 🚀
