The latest released study on Global Knowledge as a Service Market report 2023-2029, covering all features of the market, providing up-to-date data on current trends and it is a valuable source of ...

Knowledge as a Service Market is Thriving Worldwide |LogDNA, Papertrail, Sumo Logic

LogDNA has announced Spike Protection to give companies more control over fluctuations in their data and spend. LogDNA Spike Protection gives DevOps teams tools to understand and manage ...

Boomi adds Data Catalog and Preparation service to AtomSphere Platform

LogDNA, the leading observability data platform, today announced it has rebranded to Mezmo, a new corporate identity that reflects the company’s expanding capabilities and vision for observability.

LogDNA unveils new brand identity reflecting vision for observability data

The MarketWatch News Department was not involved in the creation of this content. May 10, 2023 (Heraldkeepers) -- New Jersey, United State: A new intelligence report released by Infinity Business ...

Log Monitoring Tools Market SWOT Analysis by Leading Key Players like SolarWinds, Kiwi Syslog Server, ManageEngine, LogDNA

LogDNA announced a new browser logging capability designed to enable full-stack and front-end developers to ingest frontend log data into LogDNA more efficiently to debug web applications.

SD Times news digest: LogDNA’s Browser Logger, Rust 1.5.2.0 released, and Google Play to launch new safety section

Despite the effort being placed on gathering data, it seems that most organizations still aren’t utilizing data properly once they have it. According to Splunk’s State of Dark Data Report, a ...

Splunk: Many organizations are hosting “dark data”

XBRL, the extensible business reporting language, was conceived for financial reporting. But Altova’s CEO Alexander Falk said he’s seeing an evolution in the European Union, where banking ...

Altova announces RaptorXML validation server

Software tools provider Altova today announced the release of Version 2023 Release 2 of its desktop developer tools, server software, and regulatory solutions, offering users several new features.

Altova 2023 Release 2 provides several important new features

In this monthly feature, we’ll keep you up-to-date on the latest career developments for individuals in the big data community. Whether it’s a promotion, new company hire, or even an accolade, we’ve got the details Read more…

Vendor » LogDNA

More information is available here. LogDNA’s Usage Quotas are a new feature that allows users to set daily or monthly limits on the volume of logs stored and gives users more granular control ...

SD Times news digest: Okta to acquire Auth0, Altova 2021 release 2, and LogDNA Usage Quotas

LogDNA

logdna

This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. Thanks to Mike for his time and input! Mike Hu is the Head of Engineering at LogDNA , a cloud-based log management system that allows engineering and devops to aggregate all system and application logs into one efficient platform. For context, how big is your engineering team? Are you using microservices and can you give a general overview of how you’re using them? We are at 7 engineers. Our infrastructure is segmented by function, with multiple redundancies at every layer structured such that there is never a single point of failure. Did you start with a monolith and later adopt Microservices? If not, did you start directly with microservices? Why? We started with a microservices model from the beginning, taking lessons from established companies and proven strategies for deployment. The product we build requires a high level of availability and the only way to achieve this from what we know is to ensure from the ground-up that there was no single point of failure, which in our world would simply translate to downtime. How did you approach the topic of microservices as a team, and was there discussion on aligning around what a microservice is? We had the opportunity to begin everything with a very small team of two and there wasn’t really any organizational need for us [to formally align on microservices]. Most of the discussions were impromptu discussions with a single goal in mind. How much freedom is there on technology choices? Did you all agree on sticking with one stack or is there flexibility to try new? We’ve experimented with different technologies both in previous pivots as well as during the building of our current product. We’ve tried to stick to things that we were familiar with unless there was a special need. It’s always better to deal with the devil you know than the devil you don’t. How do you determine service boundaries and what lessons have you learned around sizing services? Most of our service boundaries are split based on function; some of which are business (like a business process), some of which are technical (like one service per code repository). As an example, we have workers fanned out to various servers, each that perform a specific function, but all of which listen to a message queue for work. The intentions were to minimize impact due to bad code. From an engineering organization’s perspective, as you make changes to the existing code-base, it is inevitable that you run into bugs which arguably is not really in your control, but you can certainly control how much of your infrastructure is impacted in case that does happen. How have microservices impacted your development process or your ops and deployment processes? I think the most impact is in automating the deployment process. There are many ways to achieve automation, we happen to have written our own Slackbot to handle deployments. The deployments can happen very quickly across all of our services, so it’s important to ensure services can stop and start in a timely manner, otherwise it could mean that all of our services (that perform the same function) become inaccessible for a short period of time. We tackled this through two different means: one is to ensure our code is written to gracefully shut down and capable of spin-up within a set amount of time; two is to ensure the deployment automation performs rolling restarts of our services rather than restarting everything all at once. How have microservices impacted the way you approach testing? The biggest gain is to actually be able to have two (or even more) different versions of code running within the same infrastructure. This allows for the ability to test new code changes in parallel to existing services. For example, we transitioned some pipelining services by first splitting the message queue into both a test cluster as well as our live cluster, which enabled us to perform data integrity checks before pushing out to the rest of the infrastructure. How have microservices impacted security and controlling access to data? This is probably the biggest headache in all honesty. We have to build a LOT of firewall rules to limit access between clusters/services. Thankfully there are deployment frameworks (like SaltStack) that helps to bake this into the management of the infrastructure. The biggest takeaway for us was to try to use tools that support some kind of auth mechanism, rather than control inbound and outbound firewall rules. Have you run into issues managing data consistency in a microservice architecture? Not really. We run a centralized Mongo DB which has it’s own replication mechanisms to deal with data consistency. Thanks again to Mike for his time and input! This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. This interview was originally published at buttercms.com and was done for the eBook Microservices for Startups . You can sign up to receive new chapters as soon as they are released. If you’ve enjoyed this article, please help it spread by clapping below! For more content like this, follow ButterCMS on Twitter and subscribe to our blog . And if you want to add a blog or CMS to your website without messing around with Wordpress, you should try ButterCMS .

Microservices for Startups: An Interview with Mike Hu of LogDNA

Meet the Winners of Startups in North and Central America

There are lots of videos and tutorials about how to get started with Kubernetes scattered about online, but what about resources that set you up with your own real Kubernetes environment? I've compiled a list of three resources where you can get started with Kubernetes and Red Hat OpenShift, Red Hat's enterprise Kubernetes platform. No payment is required. 1. OpenShift Open Labs Want to dive deep into Kubernetes and Red Hat OpenShift? Open Labs gives you six guided walkthroughs to explore scenarios in your own Red Hat OpenShift cluster. These include: Building hybrid cloud apps by connecting to a Cloudant service using OpenShift operators Logging with LogDNA Cloud monitoring with SysDig Securing microservices . ℹ️ Tip: In order to save resources, you'll have access to the cluster for , after which you can request another cluster one hour ℹ️ Tip: There are also Kubernetes-only versions of some labs available . here 🔗 Link: https://developer.ibm.com/openlabs/openshift 🧑‍🎓 Good For: Learning about OpenShift and Cloud services, logging and SysDig Installing and configuring OpenShift Operators 🤔 Be Aware: Clusters time out after one hour 2. Earn your Build Smart on Kubernetes Badge If you only have a couple of hours to spare and want to learn about deploying applications to Kubernetes and Red Hat OpenShift, this program will get you there. You'll learn how to: Deploy a node.js application using source-to-image, Deploy a Java microservice application on Open Liberty, Deploy a machine learning model and use it to classify images At the end of the workshop, you'll receive a badge certifying completion that you can share on social media (if you're into that kind of thing!) ℹ️ Tip: If you're going for the badge, make sure you open the quiz before starting the workshops! The quiz asks for the output of a command from each lab. So it's easier to complete the quiz as you run through labs. . ℹ️ Tip: The badges are issued once per week, so you may have to wait up to seven days after completion until your badge is issued 🔗 https://www.credly.com/org/ibm/badge/build-smart-on-kubernetes/ 🧑‍🎓 Good For: Application developers interested in deploying apps to k8s Sharing your achievement on social media 🤔 Be Aware: Complete the quiz as you do the labs You may wait a week for the badge 3. Try.OpenShift.com Red Hat’s Developer Sandbox contains a number of workshops that give you a deep knowledge of Kubernetes and Red Hat OpenShift, including: Learn Kubernetes using Red Hat Developer Sandbox ℹ️ Tip: Red Hat’s Developer Sandbox comes pre-configured with CodeReady Workspaces, Red Hat's productized version of . Eclipse Che 🔗 Link: https://try.openshift.com 🧑‍🎓 Good For: Learning through kubectl and git on your local machine Working with , React, and MariaDB quay.io Deploying applications working with Kafka and JavaScript 🤔 Be Aware: Some of these labs have detailed command line requirements -- but this can be an opportunity for education! Resources Free links from the Linux Foundation Kubernetes Training Keep up to date with the Kubernetes.io blog David's Twitter

3 Free Ways to Learn Kubernetes and Red Hat OpenShift

I started looking into Heroku as an option for creating personal applications in my free time. In fact, I converted an existing application from the AWS ecosystem to Heroku which was captured in a series on DZone.com: Moving Away From AWS and Onto Heroku Starting with a brand new idea with Heroku, I was able to quickly create a fitness-based SaaS solution as well, which was documented in another series on DZone.com: Using Heroku to Quickly Build a Multi-Tenant SaaS Product Well over a year into using Heroku for several of my applications, I thought I would take a step back and describe how Heroku works and offer thoughts on why the Salesforce-owned solution should be a consideration for your next project. What is Heroku? Founded in 2007, Heroku is a platform as a service (PaaS) ecosystem which currently supports Ruby, Java, Node.js, Scala, Clojure, Python, PHP, and Go programming languages (plus community support for many other languages). Because of its ability to support multiple languages to accomplish the same result, Heroku is considered a platform. polyglot At the highest level, Heroku intends to serve the needs of applications looking for a place to live. Developers get started by creating a Heroku "app" and introduce their original code via a standardized git repository. Heroku simply takes things from there—building and deploying the application, then serving it up for consumption as needed. This includes static resources (like an Angular or React.js application), which can be served from a Node.js implementation. Heroku provides over 175 add-on services to complement each app, featuring: database services uptime alerts messaging services automatic backups search metrics mail delivery Below is an illustration of one of my personal applications leveraging database, security, and messaging all within Heroku: Heroku strives to do everything possible in order to allow users to focus on building features and functionality in their applications. In taking this approach, you can avoid the time typically required to learn, implement, manage, and support the DevOps-driven path toward application deployment. Heroku has been a Salesforce company since 2010 as part of a $212 million acquisition. In fact, Salesforce CEO Marc Benioff stated: “ Marc Benioff - Salesforce chairman and CEO With Heroku, our platform can become the cornerstone of the next-generation of cloud computing.” Features of Heroku Heroku is far more than a place to deploy and house applications – it provides all the necessary aspects to complement application hosting and management. For this article, I will focus on four interesting aspects of Heroku. 1. Buildpacks and Slugs Buildpacks are the mechanism inside the Heroku ecosystem which compiles application code and creates a "slug." The slug is a pre-packaged copy of the application (including any necessary runtime layer) that is optimized for distribution into a dyno. To gain a better understanding of this concept, consider the illustration below: As part of creating a new application in Heroku, a git remote is created from the repository containing the source code to deploy. With this in place, a standardized git push command is issued, along with specifying the target host and branch. In the example above, git push heroku master is issued to push the current branch to the "master" branch in the "heroku" remote. This triggers Heroku to start processing the request. The first step is to scan the project to understand if a buildpack has been specified in the Heroku configuration. If using one of the supported languages noted above, this step is not necessary as Heroku will automatically select the correct buildpack. If you are not using one of the officially supported languages, you can create a custom buildpack, or you can select from over 6,000 third-party buildpacks in the . Elements Marketplace The buildpack then tells Heroku how to retrieve all the needed dependencies and runtime required to run the application. The result of this action is creating an application slug. Heroku then deploys that slug to one (or many) dynos and runs the buildpack or user-specified command to starts the application. 2. Dynos, Config Vars, and Releases In the prior section, a buildpack and slug were utilized to launch a new application in Heroku. The application itself runs in a Heroku dyno, which is the heart of the Heroku platform. Dynos are Linux-based containers that are isolated and designed to execute code based upon a user-specified command. (Heroku came up with the term dyno because, at the time, Docker hadn't yet come up with or popularized the term container.) Because of this design, Heroku allows dynos to be scaled up (or down) to meet capacity needs. In order to avoid including sensitive and configuration information in the source code of the application, Heroku provides a mechanism to house configuration variables. For command-line users, the heroku config command provides the following information: ╭─me@mbp ~ someService?reconnect= CLIENT_ID:         myAuthClientId
CLIENT_SECRET:     myAuthSecreteId /new-dyno ‹master›
╰─$ heroku config
=== new-dyno Config Vars
DATABASE_URL:      mysql:/ /someUser:somePassword@someHost.net/ true There is a complimentary web-based version in the Settings tab of the Heroku Dashboard: Whenever code is deployed, a configuration variable is changed, or an add-on resource is modified, Heroku creates a new release and restarts the dyno application. This is a feature of Heroku that often goes overlooked, as Heroku always provides the ability to roll back to a prior state with ease. The release history is available in the Heroku Dashboard and also available via the heroku releases command line: ╭─me@mbp ~ / : : v2 CLIENT_ID, CLIENT_SECRET config vars   me@example.com / / : : v1  Deploy me@example.com / / : : /new-dyno ‹master›
╰─$ heroku releases
=== new-dyno Releases - Current: v3
v3  Deploy 2b93a215                            me@example.com  2021/ 03 23 07 49 35 -0400 Set 2021 03 19 07 26 34 -0400 40e37807 2021 03 19 06 48 00 -0400 3. Heroku Add-Ons The adaptability of a product or a platform is largely tied to libraries or services which are available with the mere click of a mouse button or issuing a simple command-line statement. Currently, Heroku includes over 175 add-on products that cover over 24 different categories of add-ons. Below, is a current screenshot of the Heroku Marketplace where add-ons can be viewed and instantly added to an existing app. The majority of Heroku add-ons provide a free level for use or a trial period. Both options allow the developer to gain a better understanding of the product before making a long-term investment. Personally, I have taken advantage of the free trial tiers to compare solutions that compete in the same space. In fact, add-ons can be removed just as quickly as they were installed. 4. Logging and Routing Heroku simplifies logging by automatically collating and routing logs from every part of your application into a single channel. This provides a truly comprehensive and extensible app-centric logging. Heroku uses for distributed log routing and collation. Using the command-line interface, the following command can be issued to view the current logs: Logplex ╭─me@mbp ~ / , time= : PM T20: : app[web ]: : : INFO --- [scheduling ] c.g.j.f.services.SmsService : end date= / / , time= : AM /new-dyno ‹master›
╰─$ heroku logs --tail 

2021-03-28T20:45:30 app[web.1]: 2021-03-28 20:45:30.020  INFO 4 --- [scheduling-1] c.g.j.f.services.SmsService : sendSmsSummaryMessages() completed
2021-03-28T20:50:00 app[web.1]: 2021-03-28 20:50:00.019  INFO 4 --- [scheduling-1] c.g.j.f.services.SmsService : start date=03/ 28 2021 08 50 2021 -03 -28 50 00 .1 2021 -03 -28 20 50 00.019 4 -1 03 30 2021 02 50 Leveraging the power of add-ons can introduce tools like Coralogix Logging, Logentries, LogDNA, Papertrail, and Sumo Logic to the app in order to provide fully functional logic analysis and debugging. Who Should (and Should Not) Use Heroku? With several options competing for customer acquisition, the following table illustrates cases where Heroku is (and is not) an option: From my personal analysis of the IT industry over a 30-year time period, I feel comfortable asserting the fact that Heroku provides the necessary solution set to serve the needs of over 80% of the applications currently running today. The only limiting factor is to perform the necessary analysis and planning to take advantage of the services offered by Heroku. Once Heroku becomes the destination, every application will begin to benefit from the time which can be applied to future features and functionality. Conclusion If I were to draft a concise mission statement for any IT professional it would be quite simple: "Focus your time on delivering features/functionality which extends the value of your intellectual property. Leverage frameworks, products, and services for everything else." Basically, if you are in the business of building an application that helps do something with widgets, commit your time and efforts toward maximizing ways to give widgets a competitive advantage. Avoid spending time doing things that a framework, product, or service offering already provides— knowing they will likely always provide a better solution than you can—because that is why they exist. Prior to using Heroku for my applications, I found myself spending unnecessary time trying to figure out aspects that Heroku already does (and does very well) for me. This effort came with a serious consequence of not being able to allocate my limited time in the right manner. Now, a year later, I am amazed at the results which have been achieved by sticking to the simple mission statement above. From the latest information I could find, my applications are part of over 9 million applications currently running on Heroku—clearly playing a part in the next generation of cloud computing. Have a really great day! Also published at https://dzone.com/articles/how-heroku-works

What Heroku is Good For

TL;DR We ended the year with our highest quarterly revenue in company history while building some important product integrations (with Google Analytics , Algolia , Unsplash & GUN ), and dark mode . Our Alexa ranking is at 3.5k worldwide, up from a 5.2k ranking before removing Medium, proving the overall success of the move into our own infrastructure. We also had a company wide year end review where 100% of the team answered “yes” to the question “are you happy?” Traffic In 2019, Hacker Noon had 38M total site visitors, an increase of 35% from 2018 year over year. From Nov 2016 through early Dec 2019, Hacker Noon totaled ≈77M visitors. Redacted We closed the year as a top , which is up from worldwide Alexa ranking of 5.2k at the time of from our site. From mid July (launch of our software) through December 2019, we published 4,523 stories, which is an average of 27 stories per day, and our library created . 3.5k site in the world removing the Medium infrastructure 48+ years of time reading Internally, we have promoted from Technology Editor to Managing Editor, and expanded the editorial team with the part-time hiring of top community commenter as Software Editor. Natasha’s voice and data driven approach will be important to scaling our brand voice and rate of publishing. Natasha Nel Arthur Tkachenko Over the course of the year, Hacker Noon stories have been featured in , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and elsewhere around the internet. Forbes ReadWrite VICE Coin tele graph Coindesk The Next Web CSO Tech Startups MarTechSeries TheNews.Asia Real Vail Decrypt Entrepreneur.com Code Opinion NewsBTC Tokenist Network World Robotics Online Silicon Republic Salon ProductHunt Stories GiveBat Reddit Crypto Daily The Block Seeking Alpha JJ’s Javascript Blog Connecticut Post Inverse Sociable Metafilter Inside Blogging JB Supra Crypto Briefing Business.com PR Daily Daily Hodl, Yahoo Yahoo Finance Value Walk Security Boulevard Merca 2.0 Tech Republic CNN CCN Newsmax Windows Club Aithority Packt Wright State University KingsCrowd Wikipedia Everipedia Dice CBS University World News IAB Tech Lab News Click Common Dreams Detailed CryptoNewsZ Tech Story Bitcoinist Interesting Engineering Study International Tech Advocate Blockonomi Hackaday, Singapore Fintech News Torrent Freak SiteJabber Business2Community TechDirt KeenGamer, Popular Mechanics GitHub G2 Jaxenter MarTech Today Irish Times Digital Journal, StackOverflow Blog Thrive Global Thomas ZDnet @Jack @Naval Tech Domains a number of Books Revenue Our 2019 revenue (over ) increased by 73% a year from 2018 year over year. accounted for the majority of our revenue at 55%, even though we could only start serving sponsors mid July after launched. The rest of our yearly revenue was made up of podcast sponsorships at 14% (which ends up breaking-even after expenses), program at 12%, at 6%, at 5%, and events/ /interest/ accounting for 8%. redacted The Top Navigation Billboard sponsorship Hacker Noon 2.0 brand as author newsletter sponsorships The Noonies shirts other In the less than five months of running the brand as author program and newsletter sponsorships, we totaled over 100 paying customers. For 2019 Q4 (pictured below), the ordered revenue by program breakdown was top navigation (64%), brand as author (18%), newsletters (14%), followed by interest and schwag sales for the rest (4%). Redacted Our largest two customers for the year are and . We have a number of six digit sponsorship deals in our pipeline, and will be investing in scaling our sales programs in 2020. In 2019, some of our more notable customers also include , , , , , , , , , , , , , , , , and . Seen by Indeed Heroku by Salesforce Google Cloud Codacy Passbase Stream Kloudless Nodle Polyient Labs Vettery PubNub Mabl Radix LogDNA ByBit SurrveySparrow Snyk Digital Ocean Kin We’ve done a lot of optimization work with our top navigation ad and expect to increase the revenue per day from this. Additionally, we plan to explore sponsorship packages by (i.e. placement by content relevancy) in 2020. tags Product Q4 2019, we integrated with , , , and the . We also . The GUN integration is our first use of blockchain technology in the Hacker Noon publishing platform. Starting in Q1 2020, this open source decentralized database will power our annotations and inline commenting. For the first time, we will be putting some content on the blockchain. With this integration, users browsers can contribute storage and anyone can also run their own “miner” on their computer or a cloud, distributing some of our hosting costs to our readership. Unsplash Algolia Google Analytics GUN blockchain launched dark mode For the full view of our year in product development, please read our CPO Dane Lyons’ . In summary, we now have the initial experience live for our core users: writers, readers, editors, sponsors and admins; and integrated with many great technologies like: Google Cloud, SlateJS, Algolia, Filestack, Unsplash, Google Analytics, Discourse, GUN, Twitter, and more. Product Update 2019-2020 Building a digital product is always a trade-off between inventing the wheel, re-inventing the wheel and figuring out a way to plug in an existing solution — all while making the thing feel like just one quality thing. We're getting there. Very excited to build more this year. We will continue investing heavily in improving and iterating on the core product experience, reading and writing, while also launching more curation tools, such as annotations and collections, to drive account creation and micro contributions. Profit & Loss Redacted Business Development Redacted Outlook and how you can help Redacted

State Of The Noonion Q4 2019: On Dark Mode, Integration With Unsplash and GUN

Tech CEO Interview Please welcome Iron.io CEO Dylan Stamat to Hacker Noon! Iron.io is a cloud application services provider that eliminates the need to worry about infrastructure, and a marketplace partner of our past weekly sponsor, Manifold. Manifold is the marketplace for independent developer services. You can learn more about their partners by checking out this interview with JawsDB Founder John Willman and this interview with LogDNA Founder Chris Nguyen . For Hacker Noon readers, Manifold is offering $10 off any service with code HACKERNOON2018. David : Let’s talk numbers, what is the scale of your business? And what metrics do you use to gauge company progress? Our metric for company progress is the growth of our customers that are actively using our products. When you provide cloud services, you’re operating in a landscape that didn’t exist 10 years ago. And in cloud specifically, you’ve got one major player that’s dominant and unafraid to eat its ecosystem. So what we’ve been focusing on is delivering long lasting value to those folks where our particular offer is a great enduring fit. Dylan: How has the technical infrastructure to rapid scaling evolved over the last 5 years? Like many startups, our initial prototypes were in self-contained code we architected with centralized services and bottlenecks. As we’ve handled scale of 100MM requests/second bursts, we’ve had to remove those bottlenecks. You’ll see in our codebase similar architectural choices to what’s happening in most other distributed systems: leader election that moved from paxos to raft, a reduction of dependencies on underlying services that just don’t scale very well (sorry Mongo), etc. On a language level, we’ve moved from lightweight scripting in our earliest prototypes to Golang. Our latest products are being written and profiled in Rust and proving to be strong in their operational guarantees and performance. Your flagship products are IronWorker and IronMQ. Could you break down how these products fit together, and usage numbers for each? It’s funny, in the race to taking a cloud computing lens on every problem, it’s batch processing, a 50 year old challenge that we’re great at solving. This shows itself in many, many diverse business opportunities: ETL, at-scale messaging, data consumption and aggregation. For instance, we being used for ad-hoc data transition services by one of the country’s largest telecom companies; and simultaneously we batch process IoT data from an extremely successful bar automation service. What all of these folks have in common is a need to massively scale out a worker pool to a varying influx of tasks. We started IronWorker around providing the best ad-hoc worker experience (and we still do!) but found that the queues available to run it (SQS/Rabbit/etc didn’t cut it) didn’t meet our needs, so we built IronMQ from scratch. We want our products to be stable and scalable, but also extremely easy to use. If you want to use IronMQ for example, you don’t need to set up security groups, roles, network configuration endpoints, etc. You grab a client library (or use our API’s) in whatever language you’d like and start using it. There are some nice synergies between products as well. For example, IronMQ has a type of queue we call “Push Queues” which can fire off events when messages are received. One type of event is a direct hook into IronWorker, which means you can fire off workers based on message payloads. This cuts down a lot of the code you’d need to write to coordinate things. Another benefit is that we offer on-premise deployments… you’re not tied to a public cloud. You’re able to future-proof your choice in a way by minimizing lock-in. Just as an example, the operational expense of moving IronMQ to a different platform would be much, much lower than that of migrating your SQS implementation to Cloud Pub/Sub. No actual application code changes needed. In May 2017, Xenon Ventures acquired Iron.io . Over the last year, how has the company evolved post acquisition? And does Iron.io fit into Xenon Ventures bigger vision? Xenon was one of the first investors in Iron and our team founded an early cloud management platform called RightScale. So, we have an appetite for cloud services and understood the team and technology. What Xenon is able to do is treat cloud services as independent businesses that do not need to find a future acquirer to be cash flow positive and growing. It’s a bit weird to say this since you might think that every company wants to be able to survive without needing to raise more money or get acquired, but that’s what Xenon’s vision is for Iron. To be self-sustaining and under no pressures except to win customers and keep their business. Yes, we see lots of trends and cycles in cloud infrastructure, but we’re not so big that we need to be part of any bright and shiny framework in order to thrive. If you spoke to me two years ago you might wonder if we were going to support Meteor, now it’s probably whether we’re going to support a GraphQL interface :) The reality is we’re attacking a constant, plaguing business problem and it will appear in whatever greater movements are happening in the cloud. I see you’re based in Las Vegas. What is the Las Vegas tech scene like? The Downtown Project has been something I’ve been following from afar. How does the tech community fit into to everything else that is happening in Vegas? Vegas is awesome and downtown is amazing. You’ll often find me at The Laundry Room, Atomic Liquors, or Palomino. Vegas is the antithesis to the bubbly feel of San Francisco. It’s an hour away, incredibly easy on the budget, and has a chance to become a second hub for Silicon Valley companies. We hold meetups in San Francisco and, well, sometimes folks here might be a bit opinionated about the quality of the free pizza or even the free non-craft beer. It’s like Richie Rich took over the tech scene. Ironically, in the glitz and glamor of Vegas it’s the opposite. Hard working teams really standing for something by going against the grain. The focus is not on getting that $1000 Boosted Board or trying to rent in the city, but instead making real business impact and focusing on growth and customers. Yes, there’s downsides, but if you asked me where a startup with $1MM would get the most value, I’d say Vegas. Our building has a coworking space in it and you’d be amazed the companies that have a presence there. Why did you decide to partner with Manifold? We’ve actually had some past success in other marketplaces, so we started an initiative to seek out other marketplaces we may want to move into. When we ran across Manifold I was immediately interested. I really, really, like the fact that it’s platform agnostic. Like many engineers, I’m not a fan of coding myself *into* an individual platform. The integration was crazy easy and we were integrated, tested, and on the platform way faster than we originally planned for. Another benefit of Manifold is that we weren’t just dumped into the marketplace and left to wither. Their team is extremely customer centric, they have marketing strategies, and they know technology. It was definitely a perfect storm type of situation for us and they are an amazing team to work with. Check out Iron.io on Manifold

The Evolution of Cloud Native Products

Founder Interview Disclosure: the developer marketplace, has previously sponsored Hacker Noon. Manifold , Use code HACKERNOON2018 to get $10 off any service. Please welcome LogDNA Co-founder Chris Nguyen to Hacker Noon! LogDNA is log management system that will provide deep insights into your business’ production environment, and a marketplace partner Manifold. David: Let’s talk numbers, what is the scale of your business? And what metrics do you use to gauge company progress? : It’s been one hell of a journey. Since launching in February 2016, we now work with over 2,000 customers who love LogDNA. Our customers varies in sizes, from 1 GB per day of log ingestion to 10 TB a day. Keep in mind that log traffic is not always steady. We’ve had customers automatically spin up thousands of spot instances that all start authenticating and logging simultaneously. It’s not just about total numbers, but also about adapting to fluctuating load. Chris We measure progress based on a number of indicators such as new sign ups, product engagement, customer growth, and monthly storage. How has Log Management evolved over the last 5 years? Log management used to be an afterthought to many engineering teams. There was often a lack of adherence to best practices as customers would simply write log data to a random file on that server. To debug, they would then SSH into each of these machines and grep these files to view log lines. This process is typically time consuming and relies on specific infrastructure knowledge to act on. Not to mention the security implications of access to production servers. In terms of scale, the volume of generated log data is growing exponentially. When adoption of cloud hosting started picking up, we saw the introduction of early logging aggregation platforms. These players educated the market about the importance of log management and focused primarily on the aggregation and searching of log data. When we launched 2 years ago, we took a different approach to logging. Instead of solely emphasizing capability, we closely studied logging use cases and used this data to inform the design, experience, and interface of . We essentially re-imagined the typical flow of interactions, and created what we believe is the next evolution of log management. LogDNA For example, when debugging an issue and a log line of interest is identified, we consider two very important points: What information is required to determine if this log line is the source of an issue? If it is not the culprit, where do you go next? These considerations strongly influenced the development of a feature we refer to as the context menu. Each line can be expanded to reveal a set of parsed metadata that can be used to pivot to other lines sharing a similar property or set of properties. Simply clicking on a field inside this context menu can either populate your search query or even jump straight to another set of log lines containing the property of interest. Not having to manually dredge up data or how to form a specific query enables incredibly natural and rapid debugging flows. What are the technical advantages versus other log management systems? While our product is continuously built on the use cases and feedback from our customers, here are just some technical advantages that our product offers: Blazing fast search speed from a responsive distributed database High availability by eliminating single points of failure Dynamic load capacity and metered billing powered by quick scaling Hosting provider agnostic deployments lower costs for our customers Rapid development iteration and testing enable high feature velocity Low ingestion latency stems from our tail-heavy logging focus Developer ownership of individual repositories drive responsible coding practices Flexible pipeline management results in no data caps for our customers Why do customers trust LogDNA? We work with a lot of larger customers like Instacart, OpenAI, Segment where we have earn their trust to be their logging system of choice. Majority of our customers have come from a competing product and it’s our job to earn their business by showing great value right away. We are big believers in compliance and currently HIPAA, SOC2, with PCI and GDPR around the corner. Its earning the respect and business of large enterprises with a security compliance focus that allowed us to truly gain trust. In addition to securing their trust, we also earn and keep customers because of the seamless experience that we offer. We listen to our users and act on their feedback, ruthlessly optimize for speed, carefully design our UI/UX, and offer fair pricing. By way of analogy, we are the iPhone of logging in an industry full of antiquated BlackBerries. In November, you raised $7m from Initialized Capital ( Alexis Ohanian & Garry Tan ). Those two have amazing experience and post/write some great startup stuff. What do they bring to the table beyond the money? We’ve been fortunate to have worked with the Initialized Capital team since the inception of our company (pre-LogDNA, pre-Y Combinator days). They witnessed the journey since the beginning, and what they bring to the table is great sound advice. Having been Y Combinator partners in the past, both Garry Tan and Alexis Ohanian have seen all ends of the spectrum when it comes to hyper growth, as well as mistakes companies make along the way. I love the fact that they were founders themselves, they understand all the challenges and opportunities we encounter month to month. And how have you started allocating the money? After raising our Series A, we’ve been hiring the right leaders and team members to fill in new positions in sales, marketing, and engineering. 5 months ago we had fewer than 10 people working at LogDNA. We’re now at 30 people and growing. This funding has significantly increased our product and sales velocity and allow us to scale faster than ever before. LogDNA was in YC 15. I saw you pivoted from marketing platform to log management system. Was the pivot the primary benefit? How else did YC change your growth trajectory? During our time at Y Combinator, we had an email personalization tool for ecommerce companies. We wanted to help independent ecommerce stores to send Amazon-like personalized messages. After graduating YC, we realized our internal logging system was a pretty compelling product to what was current available on the market. We decided to re-focus our energy and launch in February 2016. YC allowed us to truly understand product market fit, unit economics, growth strategies and constantly listening to customers to build something they truly want. LogDNA You’ve had entrepreneurial ventures across many different industries — Recruitment (co-founder JobLoft — Acquired), Dating (GM Cupid, Acquired), Social Commerce (co-founder TeamSave — Strategic partner with eBay Classifieds), and DevOps (LogDNA). What is it that attracts you to business you must work on? Because we never started off in DevOps, we approach opportunities through a different lens based on our eclectic background in recruitment, dating, and commerce. We take what we’ve learned and apply it to the way we problem solve, always with the attempt to offer a solution that is innovative and effective. I’ve been fortunate to have the same co-founder/CTO for all our companies. He’s the Ying to my Yang when it comes to executing a new business. I love taking on the challenge of tackling new industries, but it helps tremendously to bring a fresh perspective to resolve obstacles along the way. I would analyze the market size of the opportunity, observe the current landscape and determine any feature gaps that exist. We enjoy pursuing ideas that we’ve experience personal pain points with… like LogDNA. What is your company’s biggest threat? Companies rarely fail due to competitors, but instead fall to their own worst enemy: themselves. Early company failures are often the result of poor execution, interpersonal conflicts, or a bad product-market fit. How we execute as a team is paramount to the evolution of our product, our sales strategy, and even our support plan. We have to love LogDNA ourselves if we expect our customers to love our product as well. At the end of the day, all our success metrics are all about gauging one very important thing: customer happiness, and that is what we strive for in our work. Why did you decide to partner with Manifold? Already knowing the founding team, I instantly saw the big vision they wanted to accomplish. We are honored to be partners of Manifold and looking forward to growing with their team. What does LogDNA look like as a company in 5 years? In 5 years, we want to be the logging system of choice for all software-related teams and organizations. If you weren’t building LogDNA what would you be doing? Having 3 nieces who are all under the age of 10, I’d love to be in an environment that allows me to inspire and mentor the next generation of founders. Having worked in many industries, I think I’d be a decent investor/mentor. What’s easiest way to uncover the value of LogDNA? Try us out for a free 14 day trial!

Log Management Used to be an Afterthought for Engineers

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. The last decade of my professional life has focused on making , , and apps faster and behave in a calmer fashion. Below are five language-agnostic lessons I’ve learned the hard way. Rails Elixir Python 1. Don’t motivate via case studies, motivate with the emotional toll of latency Every loses Amazon 1% in sales! of page load time drops Google’s traffic by 20%! An electronic trading platform that is than the competition triggers $4M in lost revenue per millisecond! 100 ms of latency An extra 500 ms 5 ms slower The propaganda is frequently sound bites from cases studies of scenarios. You might expect me — a co-founder at , an application performance management (APM) company, to produce infographics showing how your slow, un-indexed SQL queries are slowing killing your business. YOU will be responsible for the premature death of your employer if you don’t install Scout! Web performance companies love fearmongering. extreme Scout The reality is that most of us in the Hacker Noon community are not responsible for apps the size of Amazon or Google. We cannot produce a scientifically-sound study tying the latency of our apps to revenue. A toxic work environment and customer relationship forms when on-call developers are worn out and sales and support teams can’t trust the technical team. The real problem for every team — be it Google-scale or the 99% of apps that aren’t — is a stream of unpredictable stability problems spreading distrust. You need a reliable web app in the same way you don’t want to stress about how your car will holdup over a week-long road trip. A reliable app is a core emotional need of a healthy company. 2. Less focus on averages, more focus on outliers After ten years, it’s still hard not to think of the response time distribution of requests to a web app as a bell curve. What feels natural vs. what actually happens in real life: This means that . To better understand what’s going on for the slowest requests, . For example, requests may only be slow for a power user that has a lot of data in your system. if you are optimizing your app based on aggregate data, you are optimizing a scenario that doesn’t exist you need access to unfiltered events Tooling is still catching up to handling these high-dimensionality events. Scout is one tool that lets you explore data collected from raw transaction traces via Trace Explorer: Sending structured logs and events to systems like and is also a great way to explore more generic multi-dimensional data. LogDNA Honeycomb 3. Staging environments are pretty much useless Outside of practicing for major changes that can’t be simulated in development, I’ve found staging environments to be a wasteland. It’s typically cost prohibitive to replicate a production environment exactly in staging and the load on staging is far less than production. Staging ends up becoming an odd limbo phase halfway between development and product. 4. No shame in paying for database help There’s a lot involved in scaling a database. Some of the best money we’ve ever spent at Scout was when we brought in to assist with configuring our databases as we grew. Be okay knowing you don’t possess DBA superpowers. For web apps, the database is the most frequent bottleneck. Percona 5. Logging is cheap. Use it. When a customer reports an issue and all I can say is “works for me”, I feel pretty inept. I’m much more aggressive logging sensitive areas of code today than ten years ago. A robust test suite isn’t a replacement for all of the odd edge cases that can occur in a production. Summary When you build a calmer web app, you instill those same traits inside your company’s DNA.

Five Lessons From 3,650 Days of Web Performance Work

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. We at always strive to get the most out of everything we do. For this reason, we continuously evaluate what we’ve done to see if it still holds up to our standards. A while back, we decided to take a deeper look at our infrastructure setup. Manifold In this blog post, we’ll look at the reasons why we moved to and the questions we asked ourselves. We’ll then look at some of the compromises we had to make and why we had to make them. We’ll also have a look how we configured our cluster to achieve our goals. Kubernetes When we started Manifold, we did what we knew worked well. Use to deploy containers on and expose these through . We found ourselves in a position where we could spend some extra time on building a more mature platform. The initial implementation was very simple to begin with, but we started to see some pain points: Terraform AWS EC2 ELBs Deploying was slow (~15min) No meant only the Ops people knew how to deploy Continuous Delivery Running a single container per instance can become expensive. By increasing , we could decrease cost container density In the past year, . With the experience the team had, we strongly believed in the future of this new technology. For this reason, we created our first . We also started thinking about integrations which would make more accessible. This is where the idea of building was born. Kubernetes has become very popular Kubernetes Integration Kubernetes Heighliner This leads us to another principle we live by: . By using Manifold to build Manifold, we’d know exactly what our users need. dogfooding Choosing a cluster The first question we asked ourselves was “where are we going to run this cluster?”. AWS does not offer a Kubernetes solution yet but and do. Do we need to stay within AWS and manage our own cluster or do we want to move everything to another Cloud Provider? Azure Google Cloud Platform The key questions we wanted answers to were: Can we create a High Availability cluster on AWS and how easy is it to manage this? How do we connect to our instance and what will the latency be? RDS What do we do about our encryption? KMS High Availability (HA) with kops If we can easily create and manage a cluster within AWS, it would lower the necessity to move providers. The initial tests we did with looked promising and we decided to take it a step further. It’s time to set up a High Availability cluster. kops To understand what HA means for Kubernetes, we need to first understand what it means in general. The central foundation of a highly available solution is a redundant, reliable storage layer. The number one rule of high-availability is to protect the data. Whatever else happens, whatever catches on fire, if you have the data, you can rebuild. If you lose the data, you’re done. - Kubernetes docs Kubernetes components in a High Availability configuration Within a Kubernetes cluster, this storage layer is and runs on the master instances. etcd is a distributed key/value store which follows the to achieve quorum. Achieving quorum means having a set of servers agreeing on a set of values. To reach this consensus, it needs parties to agree. Therefore we always need an uneven amount of instances with at least 3. etcd Raft consensus algorithm lower(n/2)+1 Below, we’ll look at a few possible disruption cases. Tolerating instance failure The first scenario we’ll look at is to see what happens when a single instance terminates. Can we recover from this? Tolerating instance failure By specifying the amount of nodes we want, kops creates an per instance group. This ensures that when an instance terminates, a new one gets created. This allows us to keep consensus across our cluster when we lose an instance. Auto Scaling Group Tolerating zone failure Having set up instance failure allows us to tolerate failure of a single machine. But what happens when the whole datacenter is having issues due to a power cut for example? This is where come into play. and Regions Availability Zones Let’s look back at our consensus formula: . We can translate this to zones, which would result in . lower(n/2)+1 instances with at least 3 instances lower(n/2)+1 zones with at least 3 zones 3 master nodes spread across 2 zones 3 master nodes spread across 3 zones With kops, this too is simple. By specifying the zones we want to run both our masters and nodes in, we can configure HA at the zone level. This however is where we ran into our first roadblock. For arbitrary reasons, when we started Manifold, we decided to use the region. As it turns out, this region only has 2 zones available. This meant that we’d have to find another solution to tolerate zone failure. us-west-1 Tolerating region failure (and beyond) The main goal was to replicate the existing infrastructure. Our legacy setup did not run across multiple regions, so the new setup didn’t have to either. We do believe that with the help of , this will be easier to set up. Kubernetes Federation Internal Networking with Peering Because of our regional restrictions, we had to find other ways to tolerate zone failure. One option is to create our cluster in a separate region. Each region runs its own separated network. This means that we can’t just use resources from one region in the other. For this, we looked into . This would allow us to connect to our us-west-1 region and access and . inter-region VPC peering RDS KMS Inter Region peering between us-west-1 and us-west-2 This too set us back. As it turns out, the region isn’t the best region you could use. At the time we investigated this, didn’t support inter-region VPC peering. This means that we couldn’t use this solution either. us-west-1 us-west-1 Decisions and compromises With all this new knowledge, it was time to make a decision. Would we stay with AWS or move over to another provider? It’s worth noting that moving to another provider comes with a lot of extra overhead as well. We’d have to expose our database, migrate our KMS and re-encrypt all our data. In the end, we decided to and run with the tolerating node failure solution. With and inter-region peering coming soon, we felt like this was a good enough first step. stick with AWS the announcement of Amazon EKS Managing your own cluster can be time consuming. To date, we’ve seen minimal impact, but we definitely . The most time consuming would be . accounted for cluster maintenance cluster updates From a financial standpoint, we also compromised. Yes, it’d be cheaper than the legacy setup, but it’d be . Azure and GCP both provide the master nodes for free, which cuts down in cost quite a bit. more expensive than the competitors kops tips For us, kops has worked great. It does come with a set of defaults that you should be aware of and overwrite. One of the key things to do is to . This is done by providing the flag. enable etcd encryption — encrypt-etcd-storage By default, kops also doesn’t . is a great mechanism to limit the scope of your applications within your cluster. We highly recommend enabling this and setting up appropriate roles. enable RBAC RBAC For security reasons, we’ve into our instances. This ensures that no one can access these boxes, even when we run with a . disabled SSH private network topology Configuring the cluster With a cluster up and running, it’s time to get to work. The next step would be to configure it so that we can start deploying applications into it. Having managed our services with before meant that we had quite a bit of control on how to set things up. We managed our ELBs, DNS, logging etc. through our Terraform configuration. We needed to make sure we can do this with our Kubernetes setup as well. Terraform Load Balancing Kubernetes has the notion of and . With a Service, it’s possible to group pods — usually managed by a — and expose them under the same endpoint. This endpoint could either be internal or external. When configuring a Service as a , Kubernetes will generate an ELB. This ELB is then linked to the configured service. Services Ingresses Deployment LoadBalancer Service LoadBalancer This is great, but there are you can have. By using an Ingress, we can create a single ELB and route the traffic within our cluster. There are several Ingresses available, but we went with the default . limits on the amount of ELBs Nginx Ingress Ingress LoadBalancer Now that we can route traffic to a service, it’s time to expose these through a domain. To do this, we used the project. This is a great way to keep the configured domain names close to your application. External DNS The last step we had to look at for exposing our services was making sure we served traffic through SSL. This turned out to be easy as well as there are already available solutions to this. We settled on , which integrates with our Nginx Ingress. cert-manager Service Configuration Service Configuration was an easy win for us. We already started building Manifold on top of Manifold with our . Because of this, all the credentials we needed were already configured. Terraform Integration We designed our with our Terraform Integration in mind. We kept the underlying semantics the same which meant that migrating credentials was a breeze. Kubernetes Integration We also added the . This allowed us to configure a Auth secret. When , you need this secret. option for custom secret types Docker pulling Docker images from a private registry Telemetry One of the most important things of running a distributed system is knowing what’s going on inside of it. For this, you need to set up centralized logging and metrics. We had this in our legacy platform so we definitely needed this in our new platform. For logging, we expanded our dogfooding and set up to gather logs. themselves provide a . This allows you to ship the logs from your cluster to their platform. our LogDNA Integration LogDNA DaemonSet configuration For metrics, we were relying on which worked well for us so far. As with LogDNA, Datadog also . They even have a on how to set this up! Datadog provides a DaemonSet configuration great blog post Migration With the cluster configured and our applications deployed, it was time to migrate. To , we had to do this in several stages. ensure zero down time The first stage was running the cluster on a . By connecting the two systems, we could test it without interrupting anyone. This helped us find and fix some early stage issues. separate domain In the next stage, we’d . To do this, we set up . This is a great way to see how your cluster behaves with actual traffic. After about a week, we had enough confidence to move on to the next phase. route some traffic to the Kubernetes cluster round robin DNS round-robin between the Legacy infrastructure and our Kubernetes cluster The third stage involved . After removing the appropriate Terraform configuration, all our traffic would flow through Kubernetes! removing the legacy DNS records Because of DNS cache, we decided to keep the legacy up and running for a few more days. This way, people with a cached DNS entry would not encounter an error. This also gave us the possibility to rollback in case we saw something go wrong. Conclusion Now that we’ve migrated, we can reflect back on things. Our team — and the company — has called this migration a success. We and managed to by doing so. decreased our deployment time from ~15min to ~1.5min cut operational costs We still pipeline, but we’re working on it. We’ve started work on which will in the first place help ourselves, but hopefully help others as well. haven’t finished up our Continuous Delivery Heighliner We encountered one major setback: not having 3 availability zones available to us. This . It’s a compromise we decided to make to get us up and running and we’ll be looking at fixing that soon. prevented us from running Highly Available across zones Oh, and one more thing. . This is where will help us out. So. Much. YAML Heighliner Massive shout-out to for her work on the images for this blog post. Meg Smith

Migrating to Kubernetes with zero downtime: the why and how

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. It’s no surprise how quickly Kubernetes has become a star in the land of . Kubernetes takes care of the complexity of managing numerous containers along with the changes and configuration required with groups of containers called pods, and groups of nodes called clusters. In doing so, it lets you focus on what matters most to you — the code and data at the heart of your critical applications. Because of these advantages, Kubernetes has become the headlining container orchestration tool today. DevOps Kubernetes makes it easy to manage containers at scale, but it can come with a steep curve as well. This is the reason for the numerous offerings for managing Kubernetes services such as Platform9, Kismatic, OpenShift, and CoreOS Tectonic (now part of Redhat) to name a few. learning No matter which route you take to managing your Kubernetes cluster, one fundamental requirement is effective when things go upside down. Traditional application infrastructure utilized log data to troubleshoot performance issues, system failures, bugs, and attacks on a server by server basis. Modern infrastructure like Kubernetes exponentially increases the number of virtual systems(containers) where logs need to be aggregated and tagged for effective analysis. log analysis Disclaimer This article summarizes the insights and thoughts while building our Kubernetes support for . It was important to create a great experience for our customers but also needed it selfishly as we moved the majority of our own services onto Kubernetes. We specialize in building a beautiful logging platform that magically handles log collection, parsing, indexing, tagging, searching, analysis, alerting, and storage across all of your systems. LogDNA The Importance of Log Data in Kubernetes Log data is essential to Kubernetes management. Kubernetes is a very dynamic platform with multitudes of changes happening all the time. As containers are started and stopped, IP addresses and loads change — Kubernetes makes changes to ensure services are scaled correctly and performance is not impacted. Inevitably when things break, or performance slows down, you need the detail that only log data can provide. In the Kubernetes world, containers are constantly being deployed/terminated and it’s entirely possible that the containers generating logs for what you’re looking for may already be terminated by the time you investigate. The only way to trace back is to ensure the logs are available so you can trace back your steps to paint a picture of what happened. Outside of performance and troubleshooting, industries that require compliance will also need log data to ensure you’re meeting the needs of HIPAA or PCI DSS like requirements. In the egregious scenarios of a data breach, you’ll need to go back in time to identify the origin of the attack and its progression across your system. For all these use cases, log data is indispensable. There are many ways you can access and analyze Kubernetes log data ranging from simple to advanced. Let’s start with the simplest option and move up the chain. Logging within a Pod Containers are the lowest level element in Kubernetes, but Pod-level logging is the most basic form of viewing Kubernetes logs. Kubectl commands are used to fetch log data for each pod individually. These logs are stored in the pod and when the pod dies, the logs die with them. They are useful and effective when you have just a few pods. You can instantly check the health of pods without needing a robust logging setup for a big cluster. Logging within a Node Logs collected for each node are stored in a JSON file. This file can get really large, and to deal with that situation, you can use the logrotate function to split the log data in multiple files once a day, or when the data reaches a particular size quota. Node-level logs are more persistent than pod-level ones. Even if a pod is restarted, it’s previous logs are retained in a node. But if a pod is evicted from a node, its log data is deleted. While pod-level and node-level logging are important concepts in Kubernetes, they aren’t meant to be real logging solutions. Rather, they act as a building block for the real solution, cluster-level logging. Logging the entire Cluster Kubernetes doesn’t provide a default logging mechanism for the entire cluster, but leaves this up to the user and third-party tools to figure out. One approach is to build on the node-level logging. This way, you can assign an agent to log every node and combine their output. The default option is which uses a agent and writes log output to a local file. However, you can also set it to send the same data to Google Cloud. From here you can use to query the log data. This, however, is not the most powerful way to analyze your log data and can be a pain if you’re not already using GCP. Which leads us into discussing potential solutions that are out there today. Stackdriver Fluentd Google Cloud’s CLI DIY Kubernetes Logging with Elasticsearch One of the more popular ways to implement cluster-level logging is to use a Fluentd agent to collect logs from the nodes, and pass them onto an external Elasticsearch cluster. The log data is stored and processed using Elasticsearch, and can be visualized using a tool like Kibana. The ELK stack (Elasticsearch, Logstash, Kibana) or here we’re referring to an EFK (Elasticsearch, Fluentd, Kibana) is the most popular open source solution for logging today, and its components often form the base for many other modern search solutions. The ELK stack offers powerful logging, and more extensibility than the Stackdriver / Google Cloud option. While building your ELK stack isn’t difficult initially, configuring Logstash or Fluentd for every input source can be a handful and furthermore scaling your own ELK stack for the amount of log data you’ll soon be getting may end up requiring more time and effort than you think. Collecting logs using a Sidecar Container Another popular method of collecting logs is by deploying collector as a sidecar container within each pod to pull logs on a pod level. Every sidecar container contains an agent for collecting and transporting logs to a destination. Most sidecar container implementations are lightweight but requires additional resource for every pod within your node/cluster. For large scale applications, that means you’re required to configure every podspec which can be cumbersome and not a great practice at scale. Collecting logs using a DaemonSet The most effective log collection method we found was to simply deploy a collector as a . This deploys a resource on a node level versus the pod level, and maintains the the same capabilities of a sidecar implementation without requiring the need for an additional process/container deployed for each pod. Deploying as a DaemonSet also inherently allows you to automate the deployment across your entire cluster with a very minimal set of kubectl commands. Set it and forget it! DaemonSet Rich contextual Log metadata Screenshot of Kubernetes logs metadata captured in dropdown (Nginx running Kubernetes). While collecting the logs to a searchable and centralized data store may the most important functional concept, logs are useless if you can’t find what you’re looking for. The most beautiful aspect of logging in Kubernetes is the framework and organization that a single orchestration system brings. Kubernetes stores logs in a way that provides rich contextual information for a logging solution to automatically tag each log with rich metadata. Information such as which node, pod, containers and even labels can be attached to the logs to make log analysis and aggregation much easier when you need. A proper logging solution should take advantage and allow interesting use cases directly labeling commit hashes to specific containers and filtering and searching logs against that label. TLDR; & Summary Kubernetes is the leading container orchestration platform available today. Yet, running a production cluster of Kubernetes takes a lot of familiarity with the system and robust tooling. Kubernetes presents a new scale of complexity when it comes to application logs and when it comes to log analysis, Kubernetes offers basic log collection for pods, nodes, and clusters, but for a production cluster you want unified logging at the cluster level. Building and designing a solution which can take advantage of the strengths of Kubernetes can ultimately be more elegant and scalable than keeping track of applications and servers in more traditional deployments. Building your own ELK or EFK stack is a common way to access and manage Kubernetes logs, but it can be quite complex with the number of tools to setup and maintain. Ideally, you want your logging tool to get out of the way and let you focus on your log data and your Kubernetes cluster. A deeply customized logging solution for Kubernetes should automatically recognize all metadata for your Kubernetes cluster including pods, nodes, containers, and namespaces. It lets you analyze your Kubernetes cluster in real-time, and provides powerful natural language search, filters, parsing, shortcuts, and alerts. Learn more about LogDNA for Kubernetes here.

Stories, News & Mentions about LOGDNA

Logdna’s Stories on HackerNoon

Logdna's News & Mentions