Managing logs is critical for every business in every industry. With proper management and practices in place, logs have the power to help software engineers optimize the performance of systems and operations, identify and resolve technical issues, better manage resources, and strengthen security. But where does log management stand in 2022? To answer that question, my team at New Relic examined petabytes of data gathered from millions of applications within the company’s observability platform. We anonymized and deliberately coarse-grained the appropriate data to give a general overview of how logs are used and managed. Today, we released the findings from our , which provides a detailed view of how this practice is shaping engineering and the technologies of the future. 2022 State of Logs Report Here are 5 key takeaways from the 2022 State of Logs Report: Software Engineers Don’t Want Logs to Exist in a Vacuum. We observed a 35% year-over-year increase in logging data. As the volume of log files grows, software engineers want to have log data available in one place to speed up the time to detect and respond to transactions, errors, and security incidents. The practice of centralized log management was created out of the frustration and time commitment felt by software engineers in examining thousands of log files across a number of sources to pinpoint and resolve incidents. Fluent Bit is the most used open-source tool for logs. Fluent Bit is the most used open-source processor and forwarder tool amongst New Relic users (38% use it). Fluent Bit started in 2017 to help solve logging challenges in containerized environments like Docker and Kubernetes. It has a massive following today because of its ability to manage observability data at scale across cloud-native, Internet of Things (IoT), and bare metal environments. Its popularity will continue due to its ease of use in Kubernetes environments. Following in usage is the New Relic infrastructure agent (16%), followed by sending log data directly to New Relic (14%) via an HTTP endpoint. New Relic infrastructure agents, APM agents, and the sending of log data directly via custom code illustrate the importance of making data more accessible to developers. NGINX is the most common type of log. When looking at the most popular log type, NGINX is the most used, capturing 38%, followed by Syslog (25%), ALB (20%), and Microsoft IIS W3C (9%). Web servers, load balancers, and content delivery networks (CDNs) are often used by software engineers to perform wellness checks on applications. They are not easily monitored by agents, which is why these three are the most common type of logs sent to New Relic. Firehose will soon be the de facto log forwarder for AWS serverless users. When looking at the most used Amazon Web Services (AWS) in cloud environments, Amazon Lambda is in the top spot, with 46% of all AWS users in our customer pool adopting the technology. While Lambda has been the main serverless tool for software engineers to use to transfer logs and metrics into CloudWatch automatically, the growth rate of users adopting Amazon Firehose is staggering and New Relic anticipates Firehose taking the lead spot later this year or in early 2023. Currently, only 32% of New Relic accounts that use AWS have adopted the Firehose technology, but its usage has increased 62% year-over-year compared to 23% year-over-year growth for Lambda. The data indicates that software engineers will continue to use Lambda, but new applications will be managed with Firehose. The rise in popularity of AWS Firehose can be attributed to a significant investment in integrations, with broad availability across its partner ecosystem. Java has a commanding lead with the most application log data. When examining the popularity of languages, the data shows that 50% of all logs ingested by language agents come from Java. Java has a commanding lead over .Net (26%), Ruby (22%), and Node.js (2%). The Java lead aligns with the overall popularity of Java with software developers and the adoption of Apache Log4j. To read further about the current state of log management, read the 2 022 State of Logs Report from New Relic .
