Get the most recent info and news about LogDNA on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

LogDNA

logdna

The latest released study on Global Knowledge as a Service Market report 2023-2029, covering all features of the market, providing up-to-date data on current trends and it is a valuable source of ...

Knowledge as a Service Market is Thriving Worldwide |LogDNA, Papertrail, Sumo Logic

This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. Thanks to Mike for his time and input! Mike Hu is the Head of Engineering at LogDNA , a cloud-based log management system that allows engineering and devops to aggregate all system and application logs into one efficient platform. For context, how big is your engineering team? Are you using microservices and can you give a general overview of how you’re using them? We are at 7 engineers. Our infrastructure is segmented by function, with multiple redundancies at every layer structured such that there is never a single point of failure. Did you start with a monolith and later adopt Microservices? If not, did you start directly with microservices? Why? We started with a microservices model from the beginning, taking lessons from established companies and proven strategies for deployment. The product we build requires a high level of availability and the only way to achieve this from what we know is to ensure from the ground-up that there was no single point of failure, which in our world would simply translate to downtime. How did you approach the topic of microservices as a team, and was there discussion on aligning around what a microservice is? We had the opportunity to begin everything with a very small team of two and there wasn’t really any organizational need for us [to formally align on microservices]. Most of the discussions were impromptu discussions with a single goal in mind. How much freedom is there on technology choices? Did you all agree on sticking with one stack or is there flexibility to try new? We’ve experimented with different technologies both in previous pivots as well as during the building of our current product. We’ve tried to stick to things that we were familiar with unless there was a special need. It’s always better to deal with the devil you know than the devil you don’t. How do you determine service boundaries and what lessons have you learned around sizing services? Most of our service boundaries are split based on function; some of which are business (like a business process), some of which are technical (like one service per code repository). As an example, we have workers fanned out to various servers, each that perform a specific function, but all of which listen to a message queue for work. The intentions were to minimize impact due to bad code. From an engineering organization’s perspective, as you make changes to the existing code-base, it is inevitable that you run into bugs which arguably is not really in your control, but you can certainly control how much of your infrastructure is impacted in case that does happen. How have microservices impacted your development process or your ops and deployment processes? I think the most impact is in automating the deployment process. There are many ways to achieve automation, we happen to have written our own Slackbot to handle deployments. The deployments can happen very quickly across all of our services, so it’s important to ensure services can stop and start in a timely manner, otherwise it could mean that all of our services (that perform the same function) become inaccessible for a short period of time. We tackled this through two different means: one is to ensure our code is written to gracefully shut down and capable of spin-up within a set amount of time; two is to ensure the deployment automation performs rolling restarts of our services rather than restarting everything all at once. How have microservices impacted the way you approach testing? The biggest gain is to actually be able to have two (or even more) different versions of code running within the same infrastructure. This allows for the ability to test new code changes in parallel to existing services. For example, we transitioned some pipelining services by first splitting the message queue into both a test cluster as well as our live cluster, which enabled us to perform data integrity checks before pushing out to the rest of the infrastructure. How have microservices impacted security and controlling access to data? This is probably the biggest headache in all honesty. We have to build a LOT of firewall rules to limit access between clusters/services. Thankfully there are deployment frameworks (like SaltStack) that helps to bake this into the management of the infrastructure. The biggest takeaway for us was to try to use tools that support some kind of auth mechanism, rather than control inbound and outbound firewall rules. Have you run into issues managing data consistency in a microservice architecture? Not really. We run a centralized Mongo DB which has it’s own replication mechanisms to deal with data consistency. Thanks again to Mike for his time and input! This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. This interview was originally published at buttercms.com and was done for the eBook Microservices for Startups . You can sign up to receive new chapters as soon as they are released. If you’ve enjoyed this article, please help it spread by clapping below! For more content like this, follow ButterCMS on Twitter and subscribe to our blog . And if you want to add a blog or CMS to your website without messing around with Wordpress, you should try ButterCMS .

Microservices for Startups: An Interview with Mike Hu of LogDNA

Meet the Winners of Startups in North and Central America

There are lots of videos and tutorials about how to get started with Kubernetes scattered about online, but what about resources that set you up with your own real Kubernetes environment? I've compiled a list of three resources where you can get started with Kubernetes and Red Hat OpenShift, Red Hat's enterprise Kubernetes platform. No payment is required. 1. OpenShift Open Labs Want to dive deep into Kubernetes and Red Hat OpenShift? Open Labs gives you six guided walkthroughs to explore scenarios in your own Red Hat OpenShift cluster. These include: Building hybrid cloud apps by connecting to a Cloudant service using OpenShift operators Logging with LogDNA Cloud monitoring with SysDig Securing microservices . ℹ️ Tip: In order to save resources, you'll have access to the cluster for , after which you can request another cluster one hour ℹ️ Tip: There are also Kubernetes-only versions of some labs available . here 🔗 Link: https://developer.ibm.com/openlabs/openshift 🧑‍🎓 Good For: Learning about OpenShift and Cloud services, logging and SysDig Installing and configuring OpenShift Operators 🤔 Be Aware: Clusters time out after one hour 2. Earn your Build Smart on Kubernetes Badge If you only have a couple of hours to spare and want to learn about deploying applications to Kubernetes and Red Hat OpenShift, this program will get you there. You'll learn how to: Deploy a node.js application using source-to-image, Deploy a Java microservice application on Open Liberty, Deploy a machine learning model and use it to classify images At the end of the workshop, you'll receive a badge certifying completion that you can share on social media (if you're into that kind of thing!) ℹ️ Tip: If you're going for the badge, make sure you open the quiz before starting the workshops! The quiz asks for the output of a command from each lab. So it's easier to complete the quiz as you run through labs. . ℹ️ Tip: The badges are issued once per week, so you may have to wait up to seven days after completion until your badge is issued 🔗 https://www.credly.com/org/ibm/badge/build-smart-on-kubernetes/ 🧑‍🎓 Good For: Application developers interested in deploying apps to k8s Sharing your achievement on social media 🤔 Be Aware: Complete the quiz as you do the labs You may wait a week for the badge 3. Try.OpenShift.com Red Hat’s Developer Sandbox contains a number of workshops that give you a deep knowledge of Kubernetes and Red Hat OpenShift, including: Learn Kubernetes using Red Hat Developer Sandbox ℹ️ Tip: Red Hat’s Developer Sandbox comes pre-configured with CodeReady Workspaces, Red Hat's productized version of . Eclipse Che 🔗 Link: https://try.openshift.com 🧑‍🎓 Good For: Learning through kubectl and git on your local machine Working with , React, and MariaDB quay.io Deploying applications working with Kafka and JavaScript 🤔 Be Aware: Some of these labs have detailed command line requirements -- but this can be an opportunity for education! Resources Free links from the Linux Foundation Kubernetes Training Keep up to date with the Kubernetes.io blog David's Twitter

3 Free Ways to Learn Kubernetes and Red Hat OpenShift

I started looking into Heroku as an option for creating personal applications in my free time. In fact, I converted an existing application from the AWS ecosystem to Heroku which was captured in a series on DZone.com: Moving Away From AWS and Onto Heroku Starting with a brand new idea with Heroku, I was able to quickly create a fitness-based SaaS solution as well, which was documented in another series on DZone.com: Using Heroku to Quickly Build a Multi-Tenant SaaS Product Well over a year into using Heroku for several of my applications, I thought I would take a step back and describe how Heroku works and offer thoughts on why the Salesforce-owned solution should be a consideration for your next project. What is Heroku? Founded in 2007, Heroku is a platform as a service (PaaS) ecosystem which currently supports Ruby, Java, Node.js, Scala, Clojure, Python, PHP, and Go programming languages (plus community support for many other languages). Because of its ability to support multiple languages to accomplish the same result, Heroku is considered a platform. polyglot At the highest level, Heroku intends to serve the needs of applications looking for a place to live. Developers get started by creating a Heroku "app" and introduce their original code via a standardized git repository. Heroku simply takes things from there—building and deploying the application, then serving it up for consumption as needed. This includes static resources (like an Angular or React.js application), which can be served from a Node.js implementation. Heroku provides over 175 add-on services to complement each app, featuring: database services uptime alerts messaging services automatic backups search metrics mail delivery Below is an illustration of one of my personal applications leveraging database, security, and messaging all within Heroku: Heroku strives to do everything possible in order to allow users to focus on building features and functionality in their applications. In taking this approach, you can avoid the time typically required to learn, implement, manage, and support the DevOps-driven path toward application deployment. Heroku has been a Salesforce company since 2010 as part of a $212 million acquisition. In fact, Salesforce CEO Marc Benioff stated: “ Marc Benioff - Salesforce chairman and CEO With Heroku, our platform can become the cornerstone of the next-generation of cloud computing.” Features of Heroku Heroku is far more than a place to deploy and house applications – it provides all the necessary aspects to complement application hosting and management. For this article, I will focus on four interesting aspects of Heroku. 1. Buildpacks and Slugs Buildpacks are the mechanism inside the Heroku ecosystem which compiles application code and creates a "slug." The slug is a pre-packaged copy of the application (including any necessary runtime layer) that is optimized for distribution into a dyno. To gain a better understanding of this concept, consider the illustration below: As part of creating a new application in Heroku, a git remote is created from the repository containing the source code to deploy. With this in place, a standardized git push command is issued, along with specifying the target host and branch. In the example above, git push heroku master is issued to push the current branch to the "master" branch in the "heroku" remote. This triggers Heroku to start processing the request. The first step is to scan the project to understand if a buildpack has been specified in the Heroku configuration. If using one of the supported languages noted above, this step is not necessary as Heroku will automatically select the correct buildpack. If you are not using one of the officially supported languages, you can create a custom buildpack, or you can select from over 6,000 third-party buildpacks in the . Elements Marketplace The buildpack then tells Heroku how to retrieve all the needed dependencies and runtime required to run the application. The result of this action is creating an application slug. Heroku then deploys that slug to one (or many) dynos and runs the buildpack or user-specified command to starts the application. 2. Dynos, Config Vars, and Releases In the prior section, a buildpack and slug were utilized to launch a new application in Heroku. The application itself runs in a Heroku dyno, which is the heart of the Heroku platform. Dynos are Linux-based containers that are isolated and designed to execute code based upon a user-specified command. (Heroku came up with the term dyno because, at the time, Docker hadn't yet come up with or popularized the term container.) Because of this design, Heroku allows dynos to be scaled up (or down) to meet capacity needs. In order to avoid including sensitive and configuration information in the source code of the application, Heroku provides a mechanism to house configuration variables. For command-line users, the heroku config command provides the following information: ╭─me@mbp ~ someService?reconnect= CLIENT_ID:         myAuthClientId
CLIENT_SECRET:     myAuthSecreteId /new-dyno ‹master›
╰─$ heroku config
=== new-dyno Config Vars
DATABASE_URL:      mysql:/ /someUser:somePassword@someHost.net/ true There is a complimentary web-based version in the Settings tab of the Heroku Dashboard: Whenever code is deployed, a configuration variable is changed, or an add-on resource is modified, Heroku creates a new release and restarts the dyno application. This is a feature of Heroku that often goes overlooked, as Heroku always provides the ability to roll back to a prior state with ease. The release history is available in the Heroku Dashboard and also available via the heroku releases command line: ╭─me@mbp ~ / : : v2 CLIENT_ID, CLIENT_SECRET config vars   me@example.com / / : : v1  Deploy me@example.com / / : : /new-dyno ‹master›
╰─$ heroku releases
=== new-dyno Releases - Current: v3
v3  Deploy 2b93a215                            me@example.com  2021/ 03 23 07 49 35 -0400 Set 2021 03 19 07 26 34 -0400 40e37807 2021 03 19 06 48 00 -0400 3. Heroku Add-Ons The adaptability of a product or a platform is largely tied to libraries or services which are available with the mere click of a mouse button or issuing a simple command-line statement. Currently, Heroku includes over 175 add-on products that cover over 24 different categories of add-ons. Below, is a current screenshot of the Heroku Marketplace where add-ons can be viewed and instantly added to an existing app. The majority of Heroku add-ons provide a free level for use or a trial period. Both options allow the developer to gain a better understanding of the product before making a long-term investment. Personally, I have taken advantage of the free trial tiers to compare solutions that compete in the same space. In fact, add-ons can be removed just as quickly as they were installed. 4. Logging and Routing Heroku simplifies logging by automatically collating and routing logs from every part of your application into a single channel. This provides a truly comprehensive and extensible app-centric logging. Heroku uses for distributed log routing and collation. Using the command-line interface, the following command can be issued to view the current logs: Logplex ╭─me@mbp ~ / , time= : PM T20: : app[web ]: : : INFO --- [scheduling ] c.g.j.f.services.SmsService : end date= / / , time= : AM /new-dyno ‹master›
╰─$ heroku logs --tail 

2021-03-28T20:45:30 app[web.1]: 2021-03-28 20:45:30.020  INFO 4 --- [scheduling-1] c.g.j.f.services.SmsService : sendSmsSummaryMessages() completed
2021-03-28T20:50:00 app[web.1]: 2021-03-28 20:50:00.019  INFO 4 --- [scheduling-1] c.g.j.f.services.SmsService : start date=03/ 28 2021 08 50 2021 -03 -28 50 00 .1 2021 -03 -28 20 50 00.019 4 -1 03 30 2021 02 50 Leveraging the power of add-ons can introduce tools like Coralogix Logging, Logentries, LogDNA, Papertrail, and Sumo Logic to the app in order to provide fully functional logic analysis and debugging. Who Should (and Should Not) Use Heroku? With several options competing for customer acquisition, the following table illustrates cases where Heroku is (and is not) an option: From my personal analysis of the IT industry over a 30-year time period, I feel comfortable asserting the fact that Heroku provides the necessary solution set to serve the needs of over 80% of the applications currently running today. The only limiting factor is to perform the necessary analysis and planning to take advantage of the services offered by Heroku. Once Heroku becomes the destination, every application will begin to benefit from the time which can be applied to future features and functionality. Conclusion If I were to draft a concise mission statement for any IT professional it would be quite simple: "Focus your time on delivering features/functionality which extends the value of your intellectual property. Leverage frameworks, products, and services for everything else." Basically, if you are in the business of building an application that helps do something with widgets, commit your time and efforts toward maximizing ways to give widgets a competitive advantage. Avoid spending time doing things that a framework, product, or service offering already provides— knowing they will likely always provide a better solution than you can—because that is why they exist. Prior to using Heroku for my applications, I found myself spending unnecessary time trying to figure out aspects that Heroku already does (and does very well) for me. This effort came with a serious consequence of not being able to allocate my limited time in the right manner. Now, a year later, I am amazed at the results which have been achieved by sticking to the simple mission statement above. From the latest information I could find, my applications are part of over 9 million applications currently running on Heroku—clearly playing a part in the next generation of cloud computing. Have a really great day! Also published at https://dzone.com/articles/how-heroku-works

What Heroku is Good For

TL;DR We ended the year with our highest quarterly revenue in company history while building some important product integrations (with Google Analytics , Algolia , Unsplash & GUN ), and dark mode . Our Alexa ranking is at 3.5k worldwide, up from a 5.2k ranking before removing Medium, proving the overall success of the move into our own infrastructure. We also had a company wide year end review where 100% of the team answered “yes” to the question “are you happy?” Traffic In 2019, Hacker Noon had 38M total site visitors, an increase of 35% from 2018 year over year. From Nov 2016 through early Dec 2019, Hacker Noon totaled ≈77M visitors. Redacted We closed the year as a top , which is up from worldwide Alexa ranking of 5.2k at the time of from our site. From mid July (launch of our software) through December 2019, we published 4,523 stories, which is an average of 27 stories per day, and our library created . 3.5k site in the world removing the Medium infrastructure 48+ years of time reading Internally, we have promoted from Technology Editor to Managing Editor, and expanded the editorial team with the part-time hiring of top community commenter as Software Editor. Natasha’s voice and data driven approach will be important to scaling our brand voice and rate of publishing. Natasha Nel Arthur Tkachenko Over the course of the year, Hacker Noon stories have been featured in , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , and elsewhere around the internet. Forbes ReadWrite VICE Coin tele graph Coindesk The Next Web CSO Tech Startups MarTechSeries TheNews.Asia Real Vail Decrypt Entrepreneur.com Code Opinion NewsBTC Tokenist Network World Robotics Online Silicon Republic Salon ProductHunt Stories GiveBat Reddit Crypto Daily The Block Seeking Alpha JJ’s Javascript Blog Connecticut Post Inverse Sociable Metafilter Inside Blogging JB Supra Crypto Briefing Business.com PR Daily Daily Hodl, Yahoo Yahoo Finance Value Walk Security Boulevard Merca 2.0 Tech Republic CNN CCN Newsmax Windows Club Aithority Packt Wright State University KingsCrowd Wikipedia Everipedia Dice CBS University World News IAB Tech Lab News Click Common Dreams Detailed CryptoNewsZ Tech Story Bitcoinist Interesting Engineering Study International Tech Advocate Blockonomi Hackaday, Singapore Fintech News Torrent Freak SiteJabber Business2Community TechDirt KeenGamer, Popular Mechanics GitHub G2 Jaxenter MarTech Today Irish Times Digital Journal, StackOverflow Blog Thrive Global Thomas ZDnet @Jack @Naval Tech Domains a number of Books Revenue Our 2019 revenue (over ) increased by 73% a year from 2018 year over year. accounted for the majority of our revenue at 55%, even though we could only start serving sponsors mid July after launched. The rest of our yearly revenue was made up of podcast sponsorships at 14% (which ends up breaking-even after expenses), program at 12%, at 6%, at 5%, and events/ /interest/ accounting for 8%. redacted The Top Navigation Billboard sponsorship Hacker Noon 2.0 brand as author newsletter sponsorships The Noonies shirts other In the less than five months of running the brand as author program and newsletter sponsorships, we totaled over 100 paying customers. For 2019 Q4 (pictured below), the ordered revenue by program breakdown was top navigation (64%), brand as author (18%), newsletters (14%), followed by interest and schwag sales for the rest (4%). Redacted Our largest two customers for the year are and . We have a number of six digit sponsorship deals in our pipeline, and will be investing in scaling our sales programs in 2020. In 2019, some of our more notable customers also include , , , , , , , , , , , , , , , , and . Seen by Indeed Heroku by Salesforce Google Cloud Codacy Passbase Stream Kloudless Nodle Polyient Labs Vettery PubNub Mabl Radix LogDNA ByBit SurrveySparrow Snyk Digital Ocean Kin We’ve done a lot of optimization work with our top navigation ad and expect to increase the revenue per day from this. Additionally, we plan to explore sponsorship packages by (i.e. placement by content relevancy) in 2020. tags Product Q4 2019, we integrated with , , , and the . We also . The GUN integration is our first use of blockchain technology in the Hacker Noon publishing platform. Starting in Q1 2020, this open source decentralized database will power our annotations and inline commenting. For the first time, we will be putting some content on the blockchain. With this integration, users browsers can contribute storage and anyone can also run their own “miner” on their computer or a cloud, distributing some of our hosting costs to our readership. Unsplash Algolia Google Analytics GUN blockchain launched dark mode For the full view of our year in product development, please read our CPO Dane Lyons’ . In summary, we now have the initial experience live for our core users: writers, readers, editors, sponsors and admins; and integrated with many great technologies like: Google Cloud, SlateJS, Algolia, Filestack, Unsplash, Google Analytics, Discourse, GUN, Twitter, and more. Product Update 2019-2020 Building a digital product is always a trade-off between inventing the wheel, re-inventing the wheel and figuring out a way to plug in an existing solution — all while making the thing feel like just one quality thing. We're getting there. Very excited to build more this year. We will continue investing heavily in improving and iterating on the core product experience, reading and writing, while also launching more curation tools, such as annotations and collections, to drive account creation and micro contributions. Profit & Loss Redacted Business Development Redacted Outlook and how you can help Redacted

State Of The Noonion Q4 2019: On Dark Mode, Integration With Unsplash and GUN

Tech CEO Interview Please welcome Iron.io CEO Dylan Stamat to Hacker Noon! Iron.io is a cloud application services provider that eliminates the need to worry about infrastructure, and a marketplace partner of our past weekly sponsor, Manifold. Manifold is the marketplace for independent developer services. You can learn more about their partners by checking out this interview with JawsDB Founder John Willman and this interview with LogDNA Founder Chris Nguyen . For Hacker Noon readers, Manifold is offering $10 off any service with code HACKERNOON2018. David : Let’s talk numbers, what is the scale of your business? And what metrics do you use to gauge company progress? Our metric for company progress is the growth of our customers that are actively using our products. When you provide cloud services, you’re operating in a landscape that didn’t exist 10 years ago. And in cloud specifically, you’ve got one major player that’s dominant and unafraid to eat its ecosystem. So what we’ve been focusing on is delivering long lasting value to those folks where our particular offer is a great enduring fit. Dylan: How has the technical infrastructure to rapid scaling evolved over the last 5 years? Like many startups, our initial prototypes were in self-contained code we architected with centralized services and bottlenecks. As we’ve handled scale of 100MM requests/second bursts, we’ve had to remove those bottlenecks. You’ll see in our codebase similar architectural choices to what’s happening in most other distributed systems: leader election that moved from paxos to raft, a reduction of dependencies on underlying services that just don’t scale very well (sorry Mongo), etc. On a language level, we’ve moved from lightweight scripting in our earliest prototypes to Golang. Our latest products are being written and profiled in Rust and proving to be strong in their operational guarantees and performance. Your flagship products are IronWorker and IronMQ. Could you break down how these products fit together, and usage numbers for each? It’s funny, in the race to taking a cloud computing lens on every problem, it’s batch processing, a 50 year old challenge that we’re great at solving. This shows itself in many, many diverse business opportunities: ETL, at-scale messaging, data consumption and aggregation. For instance, we being used for ad-hoc data transition services by one of the country’s largest telecom companies; and simultaneously we batch process IoT data from an extremely successful bar automation service. What all of these folks have in common is a need to massively scale out a worker pool to a varying influx of tasks. We started IronWorker around providing the best ad-hoc worker experience (and we still do!) but found that the queues available to run it (SQS/Rabbit/etc didn’t cut it) didn’t meet our needs, so we built IronMQ from scratch. We want our products to be stable and scalable, but also extremely easy to use. If you want to use IronMQ for example, you don’t need to set up security groups, roles, network configuration endpoints, etc. You grab a client library (or use our API’s) in whatever language you’d like and start using it. There are some nice synergies between products as well. For example, IronMQ has a type of queue we call “Push Queues” which can fire off events when messages are received. One type of event is a direct hook into IronWorker, which means you can fire off workers based on message payloads. This cuts down a lot of the code you’d need to write to coordinate things. Another benefit is that we offer on-premise deployments… you’re not tied to a public cloud. You’re able to future-proof your choice in a way by minimizing lock-in. Just as an example, the operational expense of moving IronMQ to a different platform would be much, much lower than that of migrating your SQS implementation to Cloud Pub/Sub. No actual application code changes needed. In May 2017, Xenon Ventures acquired Iron.io . Over the last year, how has the company evolved post acquisition? And does Iron.io fit into Xenon Ventures bigger vision? Xenon was one of the first investors in Iron and our team founded an early cloud management platform called RightScale. So, we have an appetite for cloud services and understood the team and technology. What Xenon is able to do is treat cloud services as independent businesses that do not need to find a future acquirer to be cash flow positive and growing. It’s a bit weird to say this since you might think that every company wants to be able to survive without needing to raise more money or get acquired, but that’s what Xenon’s vision is for Iron. To be self-sustaining and under no pressures except to win customers and keep their business. Yes, we see lots of trends and cycles in cloud infrastructure, but we’re not so big that we need to be part of any bright and shiny framework in order to thrive. If you spoke to me two years ago you might wonder if we were going to support Meteor, now it’s probably whether we’re going to support a GraphQL interface :) The reality is we’re attacking a constant, plaguing business problem and it will appear in whatever greater movements are happening in the cloud. I see you’re based in Las Vegas. What is the Las Vegas tech scene like? The Downtown Project has been something I’ve been following from afar. How does the tech community fit into to everything else that is happening in Vegas? Vegas is awesome and downtown is amazing. You’ll often find me at The Laundry Room, Atomic Liquors, or Palomino. Vegas is the antithesis to the bubbly feel of San Francisco. It’s an hour away, incredibly easy on the budget, and has a chance to become a second hub for Silicon Valley companies. We hold meetups in San Francisco and, well, sometimes folks here might be a bit opinionated about the quality of the free pizza or even the free non-craft beer. It’s like Richie Rich took over the tech scene. Ironically, in the glitz and glamor of Vegas it’s the opposite. Hard working teams really standing for something by going against the grain. The focus is not on getting that $1000 Boosted Board or trying to rent in the city, but instead making real business impact and focusing on growth and customers. Yes, there’s downsides, but if you asked me where a startup with $1MM would get the most value, I’d say Vegas. Our building has a coworking space in it and you’d be amazed the companies that have a presence there. Why did you decide to partner with Manifold? We’ve actually had some past success in other marketplaces, so we started an initiative to seek out other marketplaces we may want to move into. When we ran across Manifold I was immediately interested. I really, really, like the fact that it’s platform agnostic. Like many engineers, I’m not a fan of coding myself *into* an individual platform. The integration was crazy easy and we were integrated, tested, and on the platform way faster than we originally planned for. Another benefit of Manifold is that we weren’t just dumped into the marketplace and left to wither. Their team is extremely customer centric, they have marketing strategies, and they know technology. It was definitely a perfect storm type of situation for us and they are an amazing team to work with. Check out Iron.io on Manifold

The Evolution of Cloud Native Products

Founder Interview Disclosure: the developer marketplace, has previously sponsored Hacker Noon. Manifold , Use code HACKERNOON2018 to get $10 off any service. Please welcome LogDNA Co-founder Chris Nguyen to Hacker Noon! LogDNA is log management system that will provide deep insights into your business’ production environment, and a marketplace partner Manifold. David: Let’s talk numbers, what is the scale of your business? And what metrics do you use to gauge company progress? : It’s been one hell of a journey. Since launching in February 2016, we now work with over 2,000 customers who love LogDNA. Our customers varies in sizes, from 1 GB per day of log ingestion to 10 TB a day. Keep in mind that log traffic is not always steady. We’ve had customers automatically spin up thousands of spot instances that all start authenticating and logging simultaneously. It’s not just about total numbers, but also about adapting to fluctuating load. Chris We measure progress based on a number of indicators such as new sign ups, product engagement, customer growth, and monthly storage. How has Log Management evolved over the last 5 years? Log management used to be an afterthought to many engineering teams. There was often a lack of adherence to best practices as customers would simply write log data to a random file on that server. To debug, they would then SSH into each of these machines and grep these files to view log lines. This process is typically time consuming and relies on specific infrastructure knowledge to act on. Not to mention the security implications of access to production servers. In terms of scale, the volume of generated log data is growing exponentially. When adoption of cloud hosting started picking up, we saw the introduction of early logging aggregation platforms. These players educated the market about the importance of log management and focused primarily on the aggregation and searching of log data. When we launched 2 years ago, we took a different approach to logging. Instead of solely emphasizing capability, we closely studied logging use cases and used this data to inform the design, experience, and interface of . We essentially re-imagined the typical flow of interactions, and created what we believe is the next evolution of log management. LogDNA For example, when debugging an issue and a log line of interest is identified, we consider two very important points: What information is required to determine if this log line is the source of an issue? If it is not the culprit, where do you go next? These considerations strongly influenced the development of a feature we refer to as the context menu. Each line can be expanded to reveal a set of parsed metadata that can be used to pivot to other lines sharing a similar property or set of properties. Simply clicking on a field inside this context menu can either populate your search query or even jump straight to another set of log lines containing the property of interest. Not having to manually dredge up data or how to form a specific query enables incredibly natural and rapid debugging flows. What are the technical advantages versus other log management systems? While our product is continuously built on the use cases and feedback from our customers, here are just some technical advantages that our product offers: Blazing fast search speed from a responsive distributed database High availability by eliminating single points of failure Dynamic load capacity and metered billing powered by quick scaling Hosting provider agnostic deployments lower costs for our customers Rapid development iteration and testing enable high feature velocity Low ingestion latency stems from our tail-heavy logging focus Developer ownership of individual repositories drive responsible coding practices Flexible pipeline management results in no data caps for our customers Why do customers trust LogDNA? We work with a lot of larger customers like Instacart, OpenAI, Segment where we have earn their trust to be their logging system of choice. Majority of our customers have come from a competing product and it’s our job to earn their business by showing great value right away. We are big believers in compliance and currently HIPAA, SOC2, with PCI and GDPR around the corner. Its earning the respect and business of large enterprises with a security compliance focus that allowed us to truly gain trust. In addition to securing their trust, we also earn and keep customers because of the seamless experience that we offer. We listen to our users and act on their feedback, ruthlessly optimize for speed, carefully design our UI/UX, and offer fair pricing. By way of analogy, we are the iPhone of logging in an industry full of antiquated BlackBerries. In November, you raised $7m from Initialized Capital ( Alexis Ohanian & Garry Tan ). Those two have amazing experience and post/write some great startup stuff. What do they bring to the table beyond the money? We’ve been fortunate to have worked with the Initialized Capital team since the inception of our company (pre-LogDNA, pre-Y Combinator days). They witnessed the journey since the beginning, and what they bring to the table is great sound advice. Having been Y Combinator partners in the past, both Garry Tan and Alexis Ohanian have seen all ends of the spectrum when it comes to hyper growth, as well as mistakes companies make along the way. I love the fact that they were founders themselves, they understand all the challenges and opportunities we encounter month to month. And how have you started allocating the money? After raising our Series A, we’ve been hiring the right leaders and team members to fill in new positions in sales, marketing, and engineering. 5 months ago we had fewer than 10 people working at LogDNA. We’re now at 30 people and growing. This funding has significantly increased our product and sales velocity and allow us to scale faster than ever before. LogDNA was in YC 15. I saw you pivoted from marketing platform to log management system. Was the pivot the primary benefit? How else did YC change your growth trajectory? During our time at Y Combinator, we had an email personalization tool for ecommerce companies. We wanted to help independent ecommerce stores to send Amazon-like personalized messages. After graduating YC, we realized our internal logging system was a pretty compelling product to what was current available on the market. We decided to re-focus our energy and launch in February 2016. YC allowed us to truly understand product market fit, unit economics, growth strategies and constantly listening to customers to build something they truly want. LogDNA You’ve had entrepreneurial ventures across many different industries — Recruitment (co-founder JobLoft — Acquired), Dating (GM Cupid, Acquired), Social Commerce (co-founder TeamSave — Strategic partner with eBay Classifieds), and DevOps (LogDNA). What is it that attracts you to business you must work on? Because we never started off in DevOps, we approach opportunities through a different lens based on our eclectic background in recruitment, dating, and commerce. We take what we’ve learned and apply it to the way we problem solve, always with the attempt to offer a solution that is innovative and effective. I’ve been fortunate to have the same co-founder/CTO for all our companies. He’s the Ying to my Yang when it comes to executing a new business. I love taking on the challenge of tackling new industries, but it helps tremendously to bring a fresh perspective to resolve obstacles along the way. I would analyze the market size of the opportunity, observe the current landscape and determine any feature gaps that exist. We enjoy pursuing ideas that we’ve experience personal pain points with… like LogDNA. What is your company’s biggest threat? Companies rarely fail due to competitors, but instead fall to their own worst enemy: themselves. Early company failures are often the result of poor execution, interpersonal conflicts, or a bad product-market fit. How we execute as a team is paramount to the evolution of our product, our sales strategy, and even our support plan. We have to love LogDNA ourselves if we expect our customers to love our product as well. At the end of the day, all our success metrics are all about gauging one very important thing: customer happiness, and that is what we strive for in our work. Why did you decide to partner with Manifold? Already knowing the founding team, I instantly saw the big vision they wanted to accomplish. We are honored to be partners of Manifold and looking forward to growing with their team. What does LogDNA look like as a company in 5 years? In 5 years, we want to be the logging system of choice for all software-related teams and organizations. If you weren’t building LogDNA what would you be doing? Having 3 nieces who are all under the age of 10, I’d love to be in an environment that allows me to inspire and mentor the next generation of founders. Having worked in many industries, I think I’d be a decent investor/mentor. What’s easiest way to uncover the value of LogDNA? Try us out for a free 14 day trial!

Log Management Used to be an Afterthought for Engineers

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. The last decade of my professional life has focused on making , , and apps faster and behave in a calmer fashion. Below are five language-agnostic lessons I’ve learned the hard way. Rails Elixir Python 1. Don’t motivate via case studies, motivate with the emotional toll of latency Every loses Amazon 1% in sales! of page load time drops Google’s traffic by 20%! An electronic trading platform that is than the competition triggers $4M in lost revenue per millisecond! 100 ms of latency An extra 500 ms 5 ms slower The propaganda is frequently sound bites from cases studies of scenarios. You might expect me — a co-founder at , an application performance management (APM) company, to produce infographics showing how your slow, un-indexed SQL queries are slowing killing your business. YOU will be responsible for the premature death of your employer if you don’t install Scout! Web performance companies love fearmongering. extreme Scout The reality is that most of us in the Hacker Noon community are not responsible for apps the size of Amazon or Google. We cannot produce a scientifically-sound study tying the latency of our apps to revenue. A toxic work environment and customer relationship forms when on-call developers are worn out and sales and support teams can’t trust the technical team. The real problem for every team — be it Google-scale or the 99% of apps that aren’t — is a stream of unpredictable stability problems spreading distrust. You need a reliable web app in the same way you don’t want to stress about how your car will holdup over a week-long road trip. A reliable app is a core emotional need of a healthy company. 2. Less focus on averages, more focus on outliers After ten years, it’s still hard not to think of the response time distribution of requests to a web app as a bell curve. What feels natural vs. what actually happens in real life: This means that . To better understand what’s going on for the slowest requests, . For example, requests may only be slow for a power user that has a lot of data in your system. if you are optimizing your app based on aggregate data, you are optimizing a scenario that doesn’t exist you need access to unfiltered events Tooling is still catching up to handling these high-dimensionality events. Scout is one tool that lets you explore data collected from raw transaction traces via Trace Explorer: Sending structured logs and events to systems like and is also a great way to explore more generic multi-dimensional data. LogDNA Honeycomb 3. Staging environments are pretty much useless Outside of practicing for major changes that can’t be simulated in development, I’ve found staging environments to be a wasteland. It’s typically cost prohibitive to replicate a production environment exactly in staging and the load on staging is far less than production. Staging ends up becoming an odd limbo phase halfway between development and product. 4. No shame in paying for database help There’s a lot involved in scaling a database. Some of the best money we’ve ever spent at Scout was when we brought in to assist with configuring our databases as we grew. Be okay knowing you don’t possess DBA superpowers. For web apps, the database is the most frequent bottleneck. Percona 5. Logging is cheap. Use it. When a customer reports an issue and all I can say is “works for me”, I feel pretty inept. I’m much more aggressive logging sensitive areas of code today than ten years ago. A robust test suite isn’t a replacement for all of the odd edge cases that can occur in a production. Summary When you build a calmer web app, you instill those same traits inside your company’s DNA.

Five Lessons From 3,650 Days of Web Performance Work

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. We at always strive to get the most out of everything we do. For this reason, we continuously evaluate what we’ve done to see if it still holds up to our standards. A while back, we decided to take a deeper look at our infrastructure setup. Manifold In this blog post, we’ll look at the reasons why we moved to and the questions we asked ourselves. We’ll then look at some of the compromises we had to make and why we had to make them. We’ll also have a look how we configured our cluster to achieve our goals. Kubernetes When we started Manifold, we did what we knew worked well. Use to deploy containers on and expose these through . We found ourselves in a position where we could spend some extra time on building a more mature platform. The initial implementation was very simple to begin with, but we started to see some pain points: Terraform AWS EC2 ELBs Deploying was slow (~15min) No meant only the Ops people knew how to deploy Continuous Delivery Running a single container per instance can become expensive. By increasing , we could decrease cost container density In the past year, . With the experience the team had, we strongly believed in the future of this new technology. For this reason, we created our first . We also started thinking about integrations which would make more accessible. This is where the idea of building was born. Kubernetes has become very popular Kubernetes Integration Kubernetes Heighliner This leads us to another principle we live by: . By using Manifold to build Manifold, we’d know exactly what our users need. dogfooding Choosing a cluster The first question we asked ourselves was “where are we going to run this cluster?”. AWS does not offer a Kubernetes solution yet but and do. Do we need to stay within AWS and manage our own cluster or do we want to move everything to another Cloud Provider? Azure Google Cloud Platform The key questions we wanted answers to were: Can we create a High Availability cluster on AWS and how easy is it to manage this? How do we connect to our instance and what will the latency be? RDS What do we do about our encryption? KMS High Availability (HA) with kops If we can easily create and manage a cluster within AWS, it would lower the necessity to move providers. The initial tests we did with looked promising and we decided to take it a step further. It’s time to set up a High Availability cluster. kops To understand what HA means for Kubernetes, we need to first understand what it means in general. The central foundation of a highly available solution is a redundant, reliable storage layer. The number one rule of high-availability is to protect the data. Whatever else happens, whatever catches on fire, if you have the data, you can rebuild. If you lose the data, you’re done. - Kubernetes docs Kubernetes components in a High Availability configuration Within a Kubernetes cluster, this storage layer is and runs on the master instances. etcd is a distributed key/value store which follows the to achieve quorum. Achieving quorum means having a set of servers agreeing on a set of values. To reach this consensus, it needs parties to agree. Therefore we always need an uneven amount of instances with at least 3. etcd Raft consensus algorithm lower(n/2)+1 Below, we’ll look at a few possible disruption cases. Tolerating instance failure The first scenario we’ll look at is to see what happens when a single instance terminates. Can we recover from this? Tolerating instance failure By specifying the amount of nodes we want, kops creates an per instance group. This ensures that when an instance terminates, a new one gets created. This allows us to keep consensus across our cluster when we lose an instance. Auto Scaling Group Tolerating zone failure Having set up instance failure allows us to tolerate failure of a single machine. But what happens when the whole datacenter is having issues due to a power cut for example? This is where come into play. and Regions Availability Zones Let’s look back at our consensus formula: . We can translate this to zones, which would result in . lower(n/2)+1 instances with at least 3 instances lower(n/2)+1 zones with at least 3 zones 3 master nodes spread across 2 zones 3 master nodes spread across 3 zones With kops, this too is simple. By specifying the zones we want to run both our masters and nodes in, we can configure HA at the zone level. This however is where we ran into our first roadblock. For arbitrary reasons, when we started Manifold, we decided to use the region. As it turns out, this region only has 2 zones available. This meant that we’d have to find another solution to tolerate zone failure. us-west-1 Tolerating region failure (and beyond) The main goal was to replicate the existing infrastructure. Our legacy setup did not run across multiple regions, so the new setup didn’t have to either. We do believe that with the help of , this will be easier to set up. Kubernetes Federation Internal Networking with Peering Because of our regional restrictions, we had to find other ways to tolerate zone failure. One option is to create our cluster in a separate region. Each region runs its own separated network. This means that we can’t just use resources from one region in the other. For this, we looked into . This would allow us to connect to our us-west-1 region and access and . inter-region VPC peering RDS KMS Inter Region peering between us-west-1 and us-west-2 This too set us back. As it turns out, the region isn’t the best region you could use. At the time we investigated this, didn’t support inter-region VPC peering. This means that we couldn’t use this solution either. us-west-1 us-west-1 Decisions and compromises With all this new knowledge, it was time to make a decision. Would we stay with AWS or move over to another provider? It’s worth noting that moving to another provider comes with a lot of extra overhead as well. We’d have to expose our database, migrate our KMS and re-encrypt all our data. In the end, we decided to and run with the tolerating node failure solution. With and inter-region peering coming soon, we felt like this was a good enough first step. stick with AWS the announcement of Amazon EKS Managing your own cluster can be time consuming. To date, we’ve seen minimal impact, but we definitely . The most time consuming would be . accounted for cluster maintenance cluster updates From a financial standpoint, we also compromised. Yes, it’d be cheaper than the legacy setup, but it’d be . Azure and GCP both provide the master nodes for free, which cuts down in cost quite a bit. more expensive than the competitors kops tips For us, kops has worked great. It does come with a set of defaults that you should be aware of and overwrite. One of the key things to do is to . This is done by providing the flag. enable etcd encryption — encrypt-etcd-storage By default, kops also doesn’t . is a great mechanism to limit the scope of your applications within your cluster. We highly recommend enabling this and setting up appropriate roles. enable RBAC RBAC For security reasons, we’ve into our instances. This ensures that no one can access these boxes, even when we run with a . disabled SSH private network topology Configuring the cluster With a cluster up and running, it’s time to get to work. The next step would be to configure it so that we can start deploying applications into it. Having managed our services with before meant that we had quite a bit of control on how to set things up. We managed our ELBs, DNS, logging etc. through our Terraform configuration. We needed to make sure we can do this with our Kubernetes setup as well. Terraform Load Balancing Kubernetes has the notion of and . With a Service, it’s possible to group pods — usually managed by a — and expose them under the same endpoint. This endpoint could either be internal or external. When configuring a Service as a , Kubernetes will generate an ELB. This ELB is then linked to the configured service. Services Ingresses Deployment LoadBalancer Service LoadBalancer This is great, but there are you can have. By using an Ingress, we can create a single ELB and route the traffic within our cluster. There are several Ingresses available, but we went with the default . limits on the amount of ELBs Nginx Ingress Ingress LoadBalancer Now that we can route traffic to a service, it’s time to expose these through a domain. To do this, we used the project. This is a great way to keep the configured domain names close to your application. External DNS The last step we had to look at for exposing our services was making sure we served traffic through SSL. This turned out to be easy as well as there are already available solutions to this. We settled on , which integrates with our Nginx Ingress. cert-manager Service Configuration Service Configuration was an easy win for us. We already started building Manifold on top of Manifold with our . Because of this, all the credentials we needed were already configured. Terraform Integration We designed our with our Terraform Integration in mind. We kept the underlying semantics the same which meant that migrating credentials was a breeze. Kubernetes Integration We also added the . This allowed us to configure a Auth secret. When , you need this secret. option for custom secret types Docker pulling Docker images from a private registry Telemetry One of the most important things of running a distributed system is knowing what’s going on inside of it. For this, you need to set up centralized logging and metrics. We had this in our legacy platform so we definitely needed this in our new platform. For logging, we expanded our dogfooding and set up to gather logs. themselves provide a . This allows you to ship the logs from your cluster to their platform. our LogDNA Integration LogDNA DaemonSet configuration For metrics, we were relying on which worked well for us so far. As with LogDNA, Datadog also . They even have a on how to set this up! Datadog provides a DaemonSet configuration great blog post Migration With the cluster configured and our applications deployed, it was time to migrate. To , we had to do this in several stages. ensure zero down time The first stage was running the cluster on a . By connecting the two systems, we could test it without interrupting anyone. This helped us find and fix some early stage issues. separate domain In the next stage, we’d . To do this, we set up . This is a great way to see how your cluster behaves with actual traffic. After about a week, we had enough confidence to move on to the next phase. route some traffic to the Kubernetes cluster round robin DNS round-robin between the Legacy infrastructure and our Kubernetes cluster The third stage involved . After removing the appropriate Terraform configuration, all our traffic would flow through Kubernetes! removing the legacy DNS records Because of DNS cache, we decided to keep the legacy up and running for a few more days. This way, people with a cached DNS entry would not encounter an error. This also gave us the possibility to rollback in case we saw something go wrong. Conclusion Now that we’ve migrated, we can reflect back on things. Our team — and the company — has called this migration a success. We and managed to by doing so. decreased our deployment time from ~15min to ~1.5min cut operational costs We still pipeline, but we’re working on it. We’ve started work on which will in the first place help ourselves, but hopefully help others as well. haven’t finished up our Continuous Delivery Heighliner We encountered one major setback: not having 3 availability zones available to us. This . It’s a compromise we decided to make to get us up and running and we’ll be looking at fixing that soon. prevented us from running Highly Available across zones Oh, and one more thing. . This is where will help us out. So. Much. YAML Heighliner Massive shout-out to for her work on the images for this blog post. Meg Smith

Migrating to Kubernetes with zero downtime: the why and how

, the marketplace for independent developer services_, has previously sponsored Hacker Noon._ Disclosure: Manifold Use code HACKERNOON2018 to get $10 off any service. It’s no surprise how quickly Kubernetes has become a star in the land of . Kubernetes takes care of the complexity of managing numerous containers along with the changes and configuration required with groups of containers called pods, and groups of nodes called clusters. In doing so, it lets you focus on what matters most to you — the code and data at the heart of your critical applications. Because of these advantages, Kubernetes has become the headlining container orchestration tool today. DevOps Kubernetes makes it easy to manage containers at scale, but it can come with a steep curve as well. This is the reason for the numerous offerings for managing Kubernetes services such as Platform9, Kismatic, OpenShift, and CoreOS Tectonic (now part of Redhat) to name a few. learning No matter which route you take to managing your Kubernetes cluster, one fundamental requirement is effective when things go upside down. Traditional application infrastructure utilized log data to troubleshoot performance issues, system failures, bugs, and attacks on a server by server basis. Modern infrastructure like Kubernetes exponentially increases the number of virtual systems(containers) where logs need to be aggregated and tagged for effective analysis. log analysis Disclaimer This article summarizes the insights and thoughts while building our Kubernetes support for . It was important to create a great experience for our customers but also needed it selfishly as we moved the majority of our own services onto Kubernetes. We specialize in building a beautiful logging platform that magically handles log collection, parsing, indexing, tagging, searching, analysis, alerting, and storage across all of your systems. LogDNA The Importance of Log Data in Kubernetes Log data is essential to Kubernetes management. Kubernetes is a very dynamic platform with multitudes of changes happening all the time. As containers are started and stopped, IP addresses and loads change — Kubernetes makes changes to ensure services are scaled correctly and performance is not impacted. Inevitably when things break, or performance slows down, you need the detail that only log data can provide. In the Kubernetes world, containers are constantly being deployed/terminated and it’s entirely possible that the containers generating logs for what you’re looking for may already be terminated by the time you investigate. The only way to trace back is to ensure the logs are available so you can trace back your steps to paint a picture of what happened. Outside of performance and troubleshooting, industries that require compliance will also need log data to ensure you’re meeting the needs of HIPAA or PCI DSS like requirements. In the egregious scenarios of a data breach, you’ll need to go back in time to identify the origin of the attack and its progression across your system. For all these use cases, log data is indispensable. There are many ways you can access and analyze Kubernetes log data ranging from simple to advanced. Let’s start with the simplest option and move up the chain. Logging within a Pod Containers are the lowest level element in Kubernetes, but Pod-level logging is the most basic form of viewing Kubernetes logs. Kubectl commands are used to fetch log data for each pod individually. These logs are stored in the pod and when the pod dies, the logs die with them. They are useful and effective when you have just a few pods. You can instantly check the health of pods without needing a robust logging setup for a big cluster. Logging within a Node Logs collected for each node are stored in a JSON file. This file can get really large, and to deal with that situation, you can use the logrotate function to split the log data in multiple files once a day, or when the data reaches a particular size quota. Node-level logs are more persistent than pod-level ones. Even if a pod is restarted, it’s previous logs are retained in a node. But if a pod is evicted from a node, its log data is deleted. While pod-level and node-level logging are important concepts in Kubernetes, they aren’t meant to be real logging solutions. Rather, they act as a building block for the real solution, cluster-level logging. Logging the entire Cluster Kubernetes doesn’t provide a default logging mechanism for the entire cluster, but leaves this up to the user and third-party tools to figure out. One approach is to build on the node-level logging. This way, you can assign an agent to log every node and combine their output. The default option is which uses a agent and writes log output to a local file. However, you can also set it to send the same data to Google Cloud. From here you can use to query the log data. This, however, is not the most powerful way to analyze your log data and can be a pain if you’re not already using GCP. Which leads us into discussing potential solutions that are out there today. Stackdriver Fluentd Google Cloud’s CLI DIY Kubernetes Logging with Elasticsearch One of the more popular ways to implement cluster-level logging is to use a Fluentd agent to collect logs from the nodes, and pass them onto an external Elasticsearch cluster. The log data is stored and processed using Elasticsearch, and can be visualized using a tool like Kibana. The ELK stack (Elasticsearch, Logstash, Kibana) or here we’re referring to an EFK (Elasticsearch, Fluentd, Kibana) is the most popular open source solution for logging today, and its components often form the base for many other modern search solutions. The ELK stack offers powerful logging, and more extensibility than the Stackdriver / Google Cloud option. While building your ELK stack isn’t difficult initially, configuring Logstash or Fluentd for every input source can be a handful and furthermore scaling your own ELK stack for the amount of log data you’ll soon be getting may end up requiring more time and effort than you think. Collecting logs using a Sidecar Container Another popular method of collecting logs is by deploying collector as a sidecar container within each pod to pull logs on a pod level. Every sidecar container contains an agent for collecting and transporting logs to a destination. Most sidecar container implementations are lightweight but requires additional resource for every pod within your node/cluster. For large scale applications, that means you’re required to configure every podspec which can be cumbersome and not a great practice at scale. Collecting logs using a DaemonSet The most effective log collection method we found was to simply deploy a collector as a . This deploys a resource on a node level versus the pod level, and maintains the the same capabilities of a sidecar implementation without requiring the need for an additional process/container deployed for each pod. Deploying as a DaemonSet also inherently allows you to automate the deployment across your entire cluster with a very minimal set of kubectl commands. Set it and forget it! DaemonSet Rich contextual Log metadata Screenshot of Kubernetes logs metadata captured in dropdown (Nginx running Kubernetes). While collecting the logs to a searchable and centralized data store may the most important functional concept, logs are useless if you can’t find what you’re looking for. The most beautiful aspect of logging in Kubernetes is the framework and organization that a single orchestration system brings. Kubernetes stores logs in a way that provides rich contextual information for a logging solution to automatically tag each log with rich metadata. Information such as which node, pod, containers and even labels can be attached to the logs to make log analysis and aggregation much easier when you need. A proper logging solution should take advantage and allow interesting use cases directly labeling commit hashes to specific containers and filtering and searching logs against that label. TLDR; & Summary Kubernetes is the leading container orchestration platform available today. Yet, running a production cluster of Kubernetes takes a lot of familiarity with the system and robust tooling. Kubernetes presents a new scale of complexity when it comes to application logs and when it comes to log analysis, Kubernetes offers basic log collection for pods, nodes, and clusters, but for a production cluster you want unified logging at the cluster level. Building and designing a solution which can take advantage of the strengths of Kubernetes can ultimately be more elegant and scalable than keeping track of applications and servers in more traditional deployments. Building your own ELK or EFK stack is a common way to access and manage Kubernetes logs, but it can be quite complex with the number of tools to setup and maintain. Ideally, you want your logging tool to get out of the way and let you focus on your log data and your Kubernetes cluster. A deeply customized logging solution for Kubernetes should automatically recognize all metadata for your Kubernetes cluster including pods, nodes, containers, and namespaces. It lets you analyze your Kubernetes cluster in real-time, and provides powerful natural language search, filters, parsing, shortcuts, and alerts. Learn more about LogDNA for Kubernetes here.

Nodes and pods and containers oh my…(The fundamentals of K8’s logging)

Data Consistency in Microservices Using Sagas Via ANDREW MORGAN The core principle of using Sagas is that instead of having long transactions that hold onto locks (e.g. a two-phase commit), you break them up into a series of short transactions that commit in sequence. Read about this approach in this post. Survey: What is difficult to get right with Microservices? Via RISINGSTACK Tell us in this super short survey, so we can cover the topic that interests you the most. Take the survey here. Microservices for Startups: An Interview with Mike Hu of LogDNA Via JAKE LUMETTA How did LogDNA implement Microservices into their architecture, and what was its impact? The RED Method: A New Approach to Monitoring Microservices Via JOAB JACKSON A former Google and Weave engineer has developed an approach to monitoring, called the RED Method, that seems to be gaining favor with administrators. Simplifying Microservices with Istio in Google Kubernetes Engine — Part III Via NITHIN MALLYA This is the 3rd part of the ‘Simplifying Microservices’ series. In this part, you can learn how to do Canary Deployments and ramp traffic with Istio. Rebel with a CORS Via ZANDER MARTINEAU How to make your own simple CORS enabled API from one that has CORS disabled. Super-Powering Your Enterprise Jenkins CI Pipeline with Docker-Compose Via DARIEN FORD Learn about a specific fix to make a Jenkins file more readable and make the usage of Docker images simpler. In brief Migrating from Monoliths to Micro-services Via DUNCAN FENNING Microservices and Data Virtualization Via THOMAS JARDINET My Microservices FAQ Via JIMMY BOGARD Role of APIS in Microservices Via SHRAVANTHI REDDY

The Microservice Weekly

Product Development Interview Disclosure: Manifold , the developer marketplace, has previously sponsored Hacker Noon. Use code HACKERNOON2018 to get $10 off any service. If you missed it, definitely check out the interview I did with them last quarter, . Today, we’re going to catch up with Manifold’s VP Product to find out what makes great products. We also discuss what makes boring people boring, product integrations, and what it’s like to live for over 100 years. Building the First Independent Marketplace for Developer Services Peter Cho David: On your LinkedIn profile, it says you worked at Heroku from 1900–2016. Do you think you held that job for too long? Peter: Those of us who work in tech love finding new challenges and learning new things. A byproduct of this culture is that many of us swap through roles, companies, or careers every 1–2 years rather than really, deeply learning something (I’m absolutely guilty of this). Many of us are in this loop where we go to and from each of the major tech companies or cycle between the top players in our particular niche, at times even “boomeranging” back for round two. (I’ve left jobs where on my last day they winked and said “If you decide to boomerang back we’ll be here..”). The great thing about this is that you get some awesome cross-pollination in the industry, spreading new ideas and processes in really short cycles. From a personal perspective, if done correctly you can also greatly accelerate your career growth. That said, it takes time to truly master something, and though I don’t believe we’ll revert back to being “lifers” at companies (e.g. IBM, GE, etc), I think we need to find a happy medium where people find career fulfillment for 5–10 years, even if it is at a fast-moving technology company. I think this will especially hold true as we start investing in deep tech, e.g. robotics, biotech, AI/ML. In all seriousness though, I actually put 116 years on LinkedIn to throw off recruiters. Ever since I revealed myself to be a vampire they’ve stopped sliding into my DMs. What makes a great product? A great product is one that fulfills the promise it makes to users. Every product promises , whether it’s an iMac, a stick of MAC lipstick, a Mack truck, or a Freddie Mac-backed mortgage. It promises to make your life easier, promises to make you feel a certain way, maybe it promises to provide nutrition. Bad products don’t fulfill their promise, they might be poorly designed laptops that run hot, have loud fans, and shut down a lot, making it hard to log onto Reddit. Bad products get your hopes up and shatter your dreams. something People like to say “oh, great products will change your life” and “they have to be world class” but I don’t think every product has to do that to be “world class”. You know, I don’t expect my living room rug to change my life. But when I bought it, I said as long as you provide me with some acoustic dampening, look nice and don’t shed, you are great in my book. I have that promise to set my mouth on fire, and they do the job admirably. Those are also great products. bottles of hot sauce With Manifold , you can add all kinds of services to your stack (Bonsai Elasticsearch, Mailgun, LogDNA, etc. ). How do you approach making the use of third party products within Manifold a seamless experience? The one thing that blows my mind about developers today is how patient they are. Every time they need a new service, they’ll check Google, Quora, Stack Overflow, StackShare, inevitably spending more time researching than they probably should. I am absolutely guilty of this. When I was deploying my first application to Heroku, I spent more time comparing Mandrill to Mailgun than I did on picking my general practitioner, the person responsible for my actual health. Once we do find a service that’s right for us, we create an account, set up a billing profile, find and read the relevant documentation, install the correct client libraries, and then go back to Stack Overflow to troubleshoot it. To be honest, that’s not a lot to ask for. You want a MySQL database? No worries y’all, I got this, be back in an hour. Oh wait, you want Email? ..and Monitoring? Redis? Elasticsearch? RabbitMQ? Memcache? Logging? Which credit card do each of these go on again? Frankie is leaving on Friday, which of these accounts were in his name again? Our credit card expired? Sorry, but I quit. The flip side of this is that you end up locked into one of the big box clouds, settling for a bunch of white-label services. Listen, we want to eat Cocoa Puffs, not CVS Chocolate-flavored Corn-Spheres. We want the convenience of a big box, with the freedom to choose best-in-class developer services. With Manifold, you use one account, a single credit card, and provision any of the aforementioned services within seconds. You want a Redis database that you can share with your team? Two clicks. Do you want to use that with Laravel, Terraform, or Kubernetes? We’ve got you. How and why did you end up becoming VP Product at Manifold? They assure me that it’s not because I’m the worst coder on the team, but I think they’re just being nice. Prior to , I was running Heroku’s Elements Marketplace. While I was there, the thing that struck me most was how everything still felt. As big as the industry feels right now, there are so many aspects of development that are unnecessarily painful, and many people think AWS has won the day. What I saw at Heroku was that there is still a huge opportunity to empower and nurture the next generation of Twilios, MongoDBs, and New Relics, hopefully preventing a future Parse or RethinkDB from happening again. The cloud today is the tip of a very large iceberg, and there are hundreds of entrepreneurs out there that are willing and able to create the next generation of developer tools and services, they just don’t know it yet. Manifold early What are your KPIs for product performance? How did you choose them? And what KPIs did you once think, but they weren’t? they may be important, Right now we’re really focused on onboarding, so funnel metrics are priority number one. What percentage of people are registering? Of those people, what portion verify, start to create a resource, finish creating a resource, and SSO into their dashboard? We’re dead-set on increasing these conversion metrics. Admittedly it’s not a glamorous answer, but we’re trying to be disciplined about the fundamentals before we get fancy. Before this, we had a really long list of vanity metrics like # of users, pageviews, etc, which felt good but didn’t help us understand our business. We’re a small, over-leveraged team, so prioritization is key. We decided that we need to do one thing, do it really well, and create a focused petri dish of strong engagement. (Focused Petri Dish of Strong Engagement was actually the name of my high school band). You could have a really successful Product Hunt launch and generate a ton of users, but if they’re just signing up and leaving, they’re not providing much value. I’m probably not going to invite these people to my wedding. Can you share some numbers about Manifold’s current usage? David, I am a huge fan of full transparency and would love to show you all our usage numbers, but unfortunately we’ve reached the max # of users on our Mode Analytics subscription. That said, my favorite number is 32, which is the percentage of users that come back for a second resource after provisioning their first. It’s a number we’re continuing to improve, and I love it because it shows the progress we are making on building a compelling, sticky product. What does your day to day work app usage look like? How do you maximize your time in these apps? 90% of my time is in Google Hangouts, the other 10% is a mix of Trello, GitHub, and Slack (coincidentally, these %s also accurately reflect my laptops CPU and RAM usage). Since we’re a distributed team with folks in San Francisco, Toronto, Hamilton, New York, Denver, Halifax, Fredericton, and Belgium, we rely a lot on tools that enable remote work. Most of our day to day happens in Slack, and we use Hangouts to coordinate, kickoff, and plan, the byproduct of which ends up in GitHub and Trello across our Engineering Waffle board and Product Roadmap boards. Although I’m happy with the current state of remote work apps, one of the things that we sorely lack is the ability to build camaraderie and create spontaneous watercooler moments. I’ve been pushing for us to incorporate apps like Overwatch and League of Legends to combat this, but apparently these aren’t considered “work apps”. fairly What changes to functionality of one of those products is an example of quality product development? My favorite new feature of recent memory is the Shared Channels feature on Slack. It’s not something that would be immediately obvious, but it’s been a big pain point for me, across my last few roles. When I was at Heroku, we partnered with Facebook on an integration with Parse. I had to create a new standalone Slack instance and invite everyone involved to it. And then at Manifold, with our first dozen integrations we had been creating one-off guest channels and inviting users, but it just adds a lot of overhead. (Can you invite this person? Can you create a new account for this one instance?) Being able to create a channel and share it using the same account details is amazing, and they made it really seamless. In building Manifold, what feature surprised you with high customer value? What feature surprised you with low customer value? I mentioned earlier we’ve been really focused on onboarding. One of the things that surprised me the most is that the things you think are really obvious, are typically not that obvious to your users. The things that provide the most value are the little things that push your users into better understanding your product, which we often take for granted. We’d spend days writing and rewriting all this amazing copy, then realize that users were getting hung up on what a “resource” was. A “resource” is a thing we sort of invented, it’s an instance of a service that you’ve provisioned. However without context, it was confusing users in our tests, which is when we realized we need to either stop inventing words or we needed to spend more time providing education and context. Ideally a little mix of both. In a similar vein, we’ll spend weeks or months building some “game-changing” feature, but take for granted that users might not understand or agree how revolutionary the feature actually is. Investing in the small things, in making sure your own biases and perspective don’t blindside you with users, is a “duh it makes sense in retrospect” lesson that happens more regularly than I’d like, but keeps us humble. According to your Twitter bio , you describe yourself as “a boring person.” Could you share something you did today that is so boring that it is interesting? My team’s reply to my first draft answer to this question was “you sound like a dad”. Which probably says enough in itself to explain my Twitter bio. I do want to mention that beyond KPIs, day to day tools, or the gift of immortality, the single most important factor that determines whether you will build a great product, is whether you have a great team. I would be nothing without our incredibly talented product management duo and , or the design powerhouse known as and . I would be a sad shell of a human being without them. Also, this interview couldn’t have happened without our marketing programs manager, . If we’re being honest I’m just Manifold’s product puppet figurehead. Thus, boring person. Ming Sitara Meg Nick Margaret MacAfee What’s the coolest new project built with Manifold? ! One of the most interesting projects we’ve got underway right now is figuring out how to build Manifold with Manifold. We’re going to be writing a blog post about how we made it happen, and how we use Kubernetes internally, so make sure to keep an eye out on ! Manifold our blog

Building Great Products with Manifold

Founder Interview Disclosure: Manifold, the developer marketplace, has previously sponsored Hacker Noon. Use code HACKERNOON to get $25 off any service. Today we’re going to catch up with Manifold co-founder Matt Creager to find out how he sees the future of software development, where Manifold is headed, and what makes him do what he does Could you share some details on Manifold’s progress, what have you learned since your recent launch? We couldn’t have predicted how excited developers would be by the prospect of an independent marketplace for developer services, an expanded selection of services on any cloud and the potential to avoid some of the pitfalls of cloud lock-in. We’ve been blown away by the reception and feedback we received during our beta. In terms of progress, we quickly discovered that developers want to be able to manage the services they build, alongside the ones they discover on Manifold. To that end, we’ve introduced custom services, which haven’t been announced yet and two new integrations for Kubernetes and Terraform — which launched today. You have a pretty wild & interactive company story, centered around a community being walled in with bland rations. Do you think developer services are too silo-ed? Why is that? What mix of your services are made by the same company you buy compute from? If you’re on AWS, I bet you use SES, RDS, etc. The question is, do we use these services because they’re the best, or because it saves us the hassle of having to set up another account and manage another billing profile? All things being equal, you’d select a service based on features, quality of docs, developer experience, support, etc. Except that in the real world — our options are limited based on where we buy compute — that’s messed up. You are building out a hub of developer services. Which pairing of developer services are most common amongst Manifold users? And how are you prioritizing which new developer services will be added next? It won’t surprise you to hear that almost every project includes JawsDB Postgres and LogDNA. Then, depending on the project we usually some form of queue either Redis Green or CloudAMQP by 84 Codes. So, the usual suspects. We have two new services on the horizon: Graphcool, a GraphQL BaaS/framework and IOPipe, who offer serverless monitoring. I expect you’ll see more services in the GQL and serverless on the platform soon. Each of the services currently available on Manifold has been handpicked. We want to work with partners that genuinely care about and are committed to developer experience and the open-source community. If that describes a service you’re building or using, drop me a line! What is Manifold’s business model? Manifold is free for developers. We don’t charge additional fees for access to teams, access controls, integrations and don’t plan to. You also won’t pay more for a service you buy on Manifold. That probably leaves you wondering how we make money? We’re wondering the same thing 😂 You list your job as Engineering and Developer Relations, and previously you were Director of Developer Relations at Heroku. How do you balance writing code with managing large communities of developers? And taking a step back, what is a week like in your founder life? I’ve always considered programming as much a hobby as a profession, and so until I became a founder, I never struggled to find time to build things. So, what does a week in the life look like? We’ve just shifted gears and started to scale up. Nick Tassone and I built the first version of Manifold’s dashboard with React, Redux and a bunch of other cool libraries and truth be told, I miss those days. We have a fantastic front-end team, so I’d just get in the way 😭 Now, I’m building developer relations and product marketing teams. Director of Developer Relations at Heroku is a great job. What advice would you give to the next engineering leader who is considering leaving their good job to lead their own startup? When you start a company, you convince yourself that the stories of burnout must be exaggerated for dramatic effect; they’re not. You’re going to make sacrifices, the sooner you recognize that the sooner you can start consciously prioritizing the things you’re not willing to sacrifice. My daughter was born shortly after we founded Manifold and my co-founder and CEO just had his third. We probably don’t get as much time in front of the Xbox as we might like, but we won’t miss an opportunity to spend time with our kids. You recently raised a $15M Series A. What did you learn from this funding raising process? And could you give us the high level view of how you are going to spend the money? It probably comes as no surprise with the growth in developer services, but there’s a healthy appetite in the market for products built either for developers or with developers in mind. If we’re going to build an independent ecosystem of developer services, we need a cloud agnostic solution for managing credentials. The Terraform and Kubernetes integrations we launched today are a start, but we believe developers should be able to use their favorite orchestration tools, languages and frameworks, without worrying about managing credentials — so, we’re going to continue to invest in integrations. Software development changes fast. In five years, how do you think the day to day work life of the software developer will be different? I think we tend to overestimate how much things will change in say the next two years and underestimate how much things will change in the say five years. Five years ago, we were talking about web 2.0, we were using.. what v0.8.0 of Node. Today, the enterprise has broadly adopted Node and.. Are we still on web 2.0? Ok, some things never change. In five years? This is going to be fun: Some combination of web technologies will be used to build interfaces for AR, VR, and hardware
We’ll spend a considerable amount of time working in AR, VR — especially those working on distributed teams
Automated testing be it unit, integration, visual regression, etc. will be powered by machine learning
We’ll have a thriving ecosystem of developer services at our disposal, it will be cloud agnostic and it will be called Manifold 😏
There will be a new front-end framework I see on your community page, you support Babel. Could you speak a bit to Babel’s value/meaning to you? And in general, how do you support the open source community? Our front-end projects, our homepage, dashboard, etc they all depend on Babel. Babel has made it possible to use modern javascript while supporting every browser. A luxury we want to make possible on the server, imagine if you could use any service with every platform? Everyone on the team at Manifold is given the opportunity to support the projects they use on a daily basis. We also try to open-source everything we believe might be useful — from our release tool Promulgate to a tool that powers the interactive prompts in Manifold CLI called promptui. Many of the providers on Manifold are active in the open-source community too, we’re working on ways we can surface this information on Manifold. Stay up to date with Manifold: Follow— @manifoldco, Manifold Read — Manifold Blog Watch — Manifoldco Like— Manifolddotco Founder Interview Disclosure: Manifold, the developer marketplace, has previously sponsored Hacker Noon. Use code HACKERNOON to get $25 off any service. Disclosure: Manifold, the developer marketplace, has previously sponsored Hacker Noon. Use code HACKERNOON to get $25 off any service. Disclosure: Manifold , the developer marketplace, has previously sponsored Hacker Noon. Manifold Use code HACKERNOON to get $25 off any service. Use code HACKERNOON to get $25 off any service. Today we’re going to catch up with Manifold co-founder Matt Creager to find out how he sees the future of software development, where Manifold is headed, and what makes him do what he does Matt Creager Could you share some details on Manifold’s progress, what have you learned since your recent launch? Could you share some details on Manifold’s progress, what have you learned since your recent launch? We couldn’t have predicted how excited developers would be by the prospect of an independent marketplace for developer services, an expanded selection of services on any cloud and the potential to avoid some of the pitfalls of cloud lock-in. We’ve been blown away by the reception and feedback we received during our beta. In terms of progress, we quickly discovered that developers want to be able to manage the services they build, alongside the ones they discover on Manifold. To that end, we’ve introduced custom services, which haven’t been announced yet and two new integrations for Kubernetes and Terraform — which launched today . Kubernetes Terraform launched today You have a pretty wild & interactive company story, centered around a community being walled in with bland rations. Do you think developer services are too silo-ed? Why is that? You have a pretty wild & interactive company story , centered around a community being walled in with bland rations. Do you think developer services are too silo-ed? Why is that? wild & interactive company story What mix of your services are made by the same company you buy compute from? If you’re on AWS, I bet you use SES, RDS, etc. The question is, do we use these services because they’re the best, or because it saves us the hassle of having to set up another account and manage another billing profile? All things being equal, you’d select a service based on features, quality of docs, developer experience, support, etc. Except that in the real world — our options are limited based on where we buy compute — that’s messed up. You are building out a hub of developer services. Which pairing of developer services are most common amongst Manifold users? And how are you prioritizing which new developer services will be added next? You are building out a hub of developer services. Which pairing of developer services are most common amongst Manifold users? And how are you prioritizing which new developer services will be added next? It won’t surprise you to hear that almost every project includes JawsDB Postgres and LogDNA . Then, depending on the project we usually some form of queue either Redis Green or CloudAMQP by 84 Codes. So, the usual suspects. JawsDB Postgres LogDNA We have two new services on the horizon: Graphcool , a GraphQL BaaS/framework and IOPipe , who offer serverless monitoring. I expect you’ll see more services in the GQL and serverless on the platform soon. Graphcool IOPipe Each of the services currently available on Manifold has been handpicked. We want to work with partners that genuinely care about and are committed to developer experience and the open-source community. If that describes a service you’re building or using, drop me a line ! drop me a line What is Manifold’s business model? What is Manifold’s business model? Manifold is free for developers. We don’t charge additional fees for access to teams, access controls, integrations and don’t plan to. You also won’t pay more for a service you buy on Manifold. That probably leaves you wondering how we make money? We’re wondering the same thing 😂 You list your job as Engineering and Developer Relations, and previously you were Director of Developer Relations at Heroku. How do you balance writing code with managing large communities of developers? And taking a step back, what is a week like in your founder life? You list your job You list your job as Engineering and Developer Relations, and previously you were Director of Developer Relations at Heroku. How do you balance writing code with managing large communities of developers? And taking a step back, what is a week like in your founder life? I’ve always considered programming as much a hobby as a profession, and so until I became a founder, I never struggled to find time to build things. So, what does a week in the life look like? We’ve just shifted gears and started to scale up. Nick Tassone and I built the first version of Manifold’s dashboard with React, Redux and a bunch of other cool libraries and truth be told, I miss those days. We have a fantastic front-end team, so I’d just get in the way 😭 Nick Tassone Now, I’m building developer relations and product marketing teams. developer relations product marketing Director of Developer Relations at Heroku is a great job. What advice would you give to the next engineering leader who is considering leaving their good job to lead their own startup? Director of Developer Relations at Heroku is a great job. What advice would you give to the next engineering leader who is considering leaving their good job to lead their own startup? When you start a company, you convince yourself that the stories of burnout must be exaggerated for dramatic effect; they’re not. You’re going to make sacrifices, the sooner you recognize that the sooner you can start consciously prioritizing the things you’re not willing to sacrifice. My daughter was born shortly after we founded Manifold and my co-founder and CEO just had his third. We probably don’t get as much time in front of the Xbox as we might like, but we won’t miss an opportunity to spend time with our kids. You recently raised a $15M Series A. What did you learn from this funding raising process? And could you give us the high level view of how you are going to spend the money? You recently raised a $15M Series A . What did you learn from this funding raising process? And could you give us the high level view of how you are going to spend the money? $15M Series A It probably comes as no surprise with the growth in developer services, but there’s a healthy appetite in the market for products built either for developers or with developers in mind. If we’re going to build an independent ecosystem of developer services, we need a cloud agnostic solution for managing credentials. The Terraform and Kubernetes integrations we launched today are a start, but we believe developers should be able to use their favorite orchestration tools, languages and frameworks, without worrying about managing credentials — so, we’re going to continue to invest in integrations. independent Software development changes fast. In five years, how do you think the day to day work life of the software developer will be different? Software development changes fast. In five years, how do you think the day to day work life of the software developer will be different? I think we tend to overestimate how much things will change in say the next two years and underestimate how much things will change in the say five years. Five years ago, we were talking about web 2.0, we were using.. what v0.8.0 of Node. Today, the enterprise has broadly adopted Node and.. Are we still on web 2.0? Ok, some things never change. In five years? This is going to be fun: Some combination of web technologies will be used to build interfaces for AR, VR, and hardware We’ll spend a considerable amount of time working in AR, VR — especially those working on distributed teams Automated testing be it unit, integration, visual regression, etc. will be powered by machine learning We’ll have a thriving ecosystem of developer services at our disposal, it will be cloud agnostic and it will be called Manifold 😏 There will be a new front-end framework Some combination of web technologies will be used to build interfaces for AR, VR, and hardware We’ll spend a considerable amount of time working in AR, VR — especially those working on distributed teams Automated testing be it unit, integration, visual regression, etc. will be powered by machine learning We’ll have a thriving ecosystem of developer services at our disposal, it will be cloud agnostic and it will be called Manifold 😏 There will be a new front-end framework I see on your community page, you support Babel. Could you speak a bit to Babel’s value/meaning to you? And in general, how do you support the open source community? I see on your community page , you support your community page Babel . Could you speak a bit to Babel’s value/meaning to you? And in general, how do you support the open source community? Babel Our front-end projects, our homepage, dashboard, etc they all depend on Babel. Babel has made it possible to use modern javascript while supporting every browser. A luxury we want to make possible on the server, imagine if you could use any service with every platform? Everyone on the team at Manifold is given the opportunity to support the projects they use on a daily basis. We also try to open-source everything we believe might be useful — from our release tool Promulgate to a tool that powers the interactive prompts in Manifold CLI called promptui . open-source everything promptui Many of the providers on Manifold are active in the open-source community too, we’re working on ways we can surface this information on Manifold. Stay up to date with Manifold: Stay up to date with Manifold : Manifold Follow— @manifoldco, Manifold Follow— @manifoldco , Manifold @manifoldco Manifold Read — Manifold Blog Read — Manifold Blog Manifold Blog Watch — Manifoldco Watch — Manifoldco Manifoldco Like— Manifolddotco Like— Manifolddotco Manifolddotco

Building the First Independent Marketplace for Developer Services

has long been used in development to ensure the reliability of an application. However, popular types of testing do not fully test the full operation of the system when interacting with the user. In this context, the use of browser-based tests helps to verify the real-time performance of a service by detecting errors in the display or operation of the product. In the ecosystem for such tests, there is a tool adapted to the framework that helps to simulate user actions. Today we will look at how it simulates user actions and what features it provides. Testing Laravel Let’s get started! Content Overview What is Dusk? Comparing Laravel Dust With Selenium WebDriver and Other Browser Tests What Can You Do with Dusk? Summary What is Dusk? According to its documentation, Laravel Dusk provides a simple API for browser automation and testing. And for those people who aren’t familiar, you’ll be pleased to know that it's a wrapper over a pretty well-known tool - ChromeDriver. Today, we will try to understand how to work with this driver and how to connect it to PHP. How often is it updated? The second important thing we want to discuss before we get started is how up-to-date Dusk is. If you look at the latest updates of the library and especially changes in major versions, you will notice that there are almost no updates. On the one hand, this is a disadvantage, as nothing new is introduced, but on the other hand, it can provide stability for your project without the constant need to update everything. Comparing Laravel Dusk With Selenium WebDriver and Other Browser Tests All browser tests are built on the WebDriver tool. Of course, the most popular one is Selenium WebDriver. In comparison, Dusk's differences will be significant, because by default, on the current version, it runs on ChromeWebDriver. However, in the current version of the package from Laravel, there is an option to switch the handling from Chrome to Selenium. This allows you to not get hung up on the implementation and switch the tool as needed. What Can You Do With Dusk? Laravel Dusk allows you to write Laravel-style browser tests. When writing browser tests, in addition to the standard features from ChromeDriver, you can use some Laravel features, such as working with databases or authorization. Let's see what you can do when writing browser tests. The entity of the browser In order to interact with the browser driver, we need to work with the method, which takes a closure with the class. This class actually provides an API for working with the browser via PHP. browse() Laravel\Dusk\Browser Navigation in the browser As far as browser navigation is concerned, it should be noted that Dusk repeats ChromeDriver, only in a beautiful and convenient shell. The methods , , and are worth mentioning. Based on their names, it becomes clear what navigation elements they work with. After installing Laravel Dusk, you can find the first browser test in the project. visit() back() forward(), refresh() Navigation is already used in it: <?php

namespace Tests\Browser;

use Laravel\Dusk\Browser;
use Tests\DuskTestCase;

class ExampleTest extends DuskTestCase
{
   /**
    * A basic browser test example.
    */
   public function testBasicExample(): void
   {
       $this->browse(function (Browser $browser) {
           $browser->visit('/')
                   ->assertSee('Laracasts');
       });
   }
} Test Migrations/Real Migrations When testing, there is always the question of what data to base tests on. In the context of browser tests, we may need it to check if the elements we show are displayed correctly. Laravel has a whole set of traits just for such needs. And if cannot be used in browser tests because of the peculiarities of transactions and HTTP requests, the other two can be used quite well and . The first one recreates the database and performs all migrations after that. The second one will only do TRUNCATE, which will make it run faster. RefreshDatabase DatabaseMigrations DatabaseTruncation has the ability to specify which tables to modify. This trait also requires the doctrine/dbal package. DatabaseTruncation DatabaseMigrations: <?php

namespace Tests\Browser;

use Illuminate\Foundation\Testing\DatabaseMigrations;
use Laravel\Dusk\Browser;
use Tests\DuskTestCase;

class ExampleTest extends DuskTestCase
{
   use DatabaseMigrations;


   /**
    * A basic browser test example.
    */
   public function testBasicExample(): void
   {
       $this->browse(function (Browser $browser) {
           $browser->visit('/')
                   ->assertSee('Laracasts');
       });
   }
} DatabaseTruncation: <?php

namespace Tests\Browser;

use App\Models\User;
use Illuminate\Foundation\Testing\DatabaseTruncation;
use Laravel\Dusk\Browser;
use Tests\DuskTestCase;

class ExampleTest extends DuskTestCase
{
   use DatabaseTruncation;

   protected $tablesToTruncate = ['users'];

   /**
    * A basic browser test example.
    */
   public function testBasicExample(): void
   {
       $this->browse(function (Browser $browser) {
           $browser->visit('/')
                   ->assertSee('Laracasts');
       });
   }
} Authentication Often, most of the project is closed to guests and open only to users or administrators. To check the view of pages protected from unauthorized eyes we don't need to go to the login form and enter login and password every time. For such cases, especially, Dusk has a method called . It allows us to log in for a specific user and takes in either a model that implements the Authenticatable interface or its primary key. The user's session will be saved inside all tests of a single file. loginAs() /**
* @throws \Throwable
*/
public function testUserSeeProfile(): void
{
   $this->browse(function (Browser $browser) {
       $browser->visit('/profile')
           ->loginAs(User::find(1))
           ->assertSee('Profile');
   });
} Storage When testing an application with browser-based tests, in addition to the boolean result of passing or failing the test, we want to get information about what exactly went correctly and what did not. For this purpose, Laravel Dusk provides methods for saving screenshots, as well as for storing console logs or page source code.  In real life, this can be useful for implementing autotests and error output or for forcing page content. Interacting with page elements To interact with elements on a page, Dusk has a large set of features inherited from ChromeDriver, such as working with selectors, getting and setting values to different form elements, and clicking buttons and links. An example would be checking if a menu button is clicked to display a drop-down list. public function testUserClickMenuDropdown(): void
{
   $this->browse(function (Browser $browser) {
       $user = $this->getUser();
       $browser->loginAs($user)
           ->visit(self::PROFILE_ROUTE)
           ->waitFor('a#navbarDropdown')
           ->click('a#navbarDropdown');


       $browser->assertVisible('.dropdown-menu.show');
   });
} It's worth noting here that Dusk also allows you to work with custom javascript code and use custom selectors to work with a particular object. To specify a custom selector, you need to specify the attribute to an object and then that object will be accessible via the selector. This can be useful when checking elements that have dynamic class names, or when checking a specific element from a set. dusk="name" '@name' Keyboard and mouse If you need to test hotkeys, scrolls, mouse clicks, and drag and drop items, Laravel Dusk has support for such methods. Among the most popular methods here, I would like to mention the , and methods. They, like many other methods, take a selector as the first parameter. Some web applications have redesigned the right mouse click in the browser. To test such a case, there is the method. click() keys(), dragOffset() rightClick() public function testUserLogin(): void
{
   $this->browse(function (Browser $browser) {
       $user = $this->getUser();
       $browser->visit(self::LOGIN_ROUTE)
           ->keys('form input[type="email"]', $user->email)
           ->keys('form input[type="password"]', 'password')
           ->press('Login')
           ->assertPathIs(self::HOME_ROUTE);
   });
} With When interacting with forms or code blocks, it is convenient to use the scope of selectors using the method. It allows you to limit the scope within a single selector, which allows you to make simpler and more readable test code. In this case, the previous example can be made like this: with() public function testUserLoginWith(): void
{
   $this->browse(function (Browser $browser) {
       $browser->visit(self::LOGIN_ROUTE)
           ->with('form', function (Browser $browser) {
               $user = $this->getUser();
               $browser->keys('input[type="email"]', $user->email)
                   ->keys('input[type="password"]', 'password')
                   ->press('Login')
                   ->assertPathIs(self::HOME_ROUTE);
           });
   });
} When interacting with a large number of elements, this method will be preferred. Waiting In some of the examples above, you may have noticed methods of waiting for some page elements by their selector. This is an essential tool when testing pages with dynamic content. When loading content with Javascript, we can wait for it using a bunch of methods such as , , . We can also use a group of these methods to interact with Javascript on a page, for example, wait for a value to be substituted into a component with or wait for an event with . waitFor() waitForText() waitUntilEnabled() waitUntilVue() waitForEvent() Assertions Laravel Dusk has a large range of assertion options, you can see the full list . Some of the most commonly used ones are , , , . Many other methods are part of those mentioned to improve code readability and syntactic sugar. here assertSee() assertValue() assertPathIs() assertVisible() Pages and Components When running complex browser tests, it can be necessary to go through multiple pages to validate a user action. To avoid an overly complex structure, Dusk has pages. Each page has , , and methods. And while the first one, based on the name, gives a path string to that page in the response, the second one checks if we are actually on that page, for example with a simple check: url() assert() elements() public function assert(Browser $browser): void
{
   $browser->assertPathIs($this->url());
} Then, the third has the most obvious behavior. The method stores macros for the selectors, typing the selectors, preventing simple mistakes from being made, and allowing work by reusing the described macros. For an example of how pages work, we'll rewrite the elements() testUserLoginWith() public function elements(): array
{
   return [
       '@email' => 'input[type="email"]',
       '@password' => 'input[type="password"]',
       '@submit' => 'button[type="submit"]',
   ];
} public function testUserLoginWith(): void
{
   $this->browse(function (Browser $browser) {
       $user = $this->getUser();
       $browser->visit(new Login())
           ->keys('@email', $user->email)
           ->keys('@password', 'password')
           ->press('@submit')
           ->assertPathIs(self::HOME_ROUTE);
   });
} If you compare this listing to our previous one, you'll notice how much more readable the test has become. Components are very similar to Pages, but they are used for interface components, such as the site header or modal window. The only difference is that instead of the method, components have a method. url() selector() Summary As you and I have seen, Dusk has a large number of features, including making it easy to write tests, not just working with ChromeDriver. Dusk developers' focus on code readability has resulted in the fact that many not-so-pleasant parts of writing tests, such as working with databases, authentication, or repeating a large number of selectors, are solved and we only have to write the test itself. Browser response logging features help when setting up tests and also when setting up test execution during the deployment. In conclusion, I would like to say that Laravel Dusk is a really handy tool that allows you to quickly make the necessary set of tests. Lead image by on Unsplash Mohammed Rahmani

How to Write Browser Tests Using Laravel Dusk 

User Authentication in Android using Firebase (Java)

Autenticación de usuarios en Android usando Firebase (Java)

Authentification de l'utilisateur dans Android à l'aide de Firebase (Java)

फायरबेस (जावा) का उपयोग करके एंड्रॉइड में उपयोगकर्ता प्रमाणीकरण

Firebase (Java) を使用した Android でのユーザー認証

Autenticação do usuário no Android usando Firebase (Java)

Xác thực người dùng trong Android bằng Firebase (Java)

使用 Firebase (Java) 在 Android 中进行用户身份验证

I have written a previous that talks about how to speed up the execution of the same type of field-filling operations using . Despite all its advantages, autofill can be accelerated even more with the help of a lower-level bash language and the use of hotkeys. article kotlin script (kts) Choosing an example We will use the same example as in the . last article You can download the full code here: https://gist.github.com/d8b96ef6d97927e811cfa292071d824a https://gist.github.com/6616b63942b3ec90ccf1fb4ad0329953 https://gist.github.com/2c1a7e43b0b2d38c21af75d139e6ac4d Let me remind you that to switch to , the fields must be filled in, and their values must meet certain requirements: MainActivity email: login@gmail.com phone: +1 650 123 4567 password: 123456 Here is a video of how it looks if the fields are filled in manually: https://youtu.be/oX3NIZMYpyo Also in the last article, we found all the necessary to fill in all the text fields and click on the button. Here is the full list: adb commands adb shell input tap 72 1092
adb shell input text 'login@gmail.com'
adb shell input tap 72 1227
adb shell input text '+1 650 123 4567'
adb shell input tap 72 1362
adb shell input text 123456
adb shell input tap 72 1497 Configuring External Tools for one command Let's try to fill in one line first. To do this, we will use a bunch of tools . External Tools + Adb + bash Let's launch and tap on the field using the terminal: LoginActivity adb shell input tap 72 1092 Everything should work. Next, we go: . Android Studio > Preferences > Tools > External Tools This is a tool that allows you to run commands and external scripts or programs inside . You can also pass arguments there. Android Studio However, there are a number of limitations compared to the terminal. For example, you cannot pass multiple commands. You can read more . here Let’s get into it. Click on the ) icon. Let’s enter the following values: Add (“+” Name: my_adb Program: /Users/{your name}/Library/Android/sdk/platform-tools/adb (insert the entire line) Arguments: adb shell input tap 72 1092 (usually it is added by itself) Working directory: $ProjectFileDir$ Let’s save: Now go to: Tools > External Tools > my_adb \ If everything is done correctly, the terminal window should start and the command will be executed. It is striking that the was executed much faster than . adb command kotlin script Configuring External Tools for multiple consecutive commands We managed to execute one adb command using External Tools. However, the reality is that more often than not it is necessary to enter several consecutive commands. Let's try to achieve this. Let's go back to the command editing window. You can see that you cannot enter more than one command there: This is also indicated by the of Jet Brains official support But, let me remind you that in order to achieve the result, we need to enter a list of commands: adb shell input tap 72 1092
adb shell input text 'login@gmail.com'
adb shell input tap 72 1227
adb shell input text '+1 650 123 4567'
adb shell input tap 72 1362
adb shell input text 123456
adb shell input tap 72 1497 To achieve this there is a workaround: create a bash script, and insert into it these commands. And through External Tools to run this script. That's what we'll do: Open the text editor Insert the lines there Save as my_bash.bash Run the command chmod +x /Users/{your name}/Downloads/my_bash.bash Insert the path into the terminal and try to execute through it: If everything is done correctly, the command should be executed, and the fields on the device should be filled in. Next, go to . Android Studio > Preferences > Tools > External Tools Creating new command: Name: my_bash (file path) Program: /Users/{your name}/Downloads/my_bash.bash Save Run . Tools > External Tools > my_bash If everything is done correctly, the command should be executed, and the fields on the device should be filled in. In the video it looks something like this: https://youtube.com/shorts/R484DB_RScc?feature=share It can be seen that the fields are filled in much times faster than with manual input. As in the case of the kotlin script, if you need time from the moment you enter one command to another, you can set the sleep delay (sec). adb shell input tap 72 1092
sleep 1
adb shell input text 'login@gmail.com' There is another optimization that can be done — to set hotkeys for a specific command. To do this, go to . Android Studio > Preferences > Keymap > External Tools > External Tools Next, find the command and set hotkeys, for example, in my case . Control+Option+Command+B Save it. Try to execute this and make sure that everything works. This is the second advantage compared to — you can set hotkeys to execute a command. kotlin script Conclusion Small optimizations of frequently repeated operations in the development process lead to long-term savings in time and resources. In the example discussed in , we saved time by automatically filling in authorization fields using , winning several tens of seconds. In the current example, we saved a few more seconds by using a lower-level language and using hotkeys. In a year, the seconds saved will turn into hours. this article kotlin script

Autofill Fields with Adb and External Tools

Anyone who has encountered non-trivial bugs on android probably knows that, sometimes, it takes a lot of time and effort to fix them. Other times, the situation is even worse - it takes a lot of time to reproduce the bug. There are cases when there is nothing to do but restart the application under different conditions: with a network, without a network, without cache, with a bad network, with erroneous responses from the server, etc. The situation can be aggravated by the fact that constant restarts require a lot of long single-type actions, e.g. filling fields with data. In this article, I want to describe one possible solution to this problem. Let’s get to it Observations and Assumptions It is assumed that you are familiar with adb This solution is suitable for filling fields of XML, compose, webView, and other visual types as well as other applications. Running a script from a scratch file sometimes causes an error. A quick way around the problem is to create another . kotlin There is currently no clean solution. scratch file Selecting an example Let's say we have an application that consists of two activities: and . To go to , you need to fill in the fields in and press : LoginActivity MainActivity MainActivity LoginActivity Enter Code: package com.leonidivankin.draftandroid.articles.autofill

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        title = "MainActivity"
    }
} <?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:padding="24dp"
    tools:context=".articles.autofill.LoginActivity">

    <EditText
        android:id="@+id/editTextEmail"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:hint="login@gmail.com"
        app:layout_constraintBottom_toTopOf="@+id/editTextPhone"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintVertical_chainStyle="packed" />

    <EditText
        android:id="@+id/editTextPhone"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:hint="+1 650 123 4567"
        app:layout_constraintBottom_toTopOf="@+id/editTextPassword"
        app:layout_constraintTop_toBottomOf="@+id/editTextEmail" />

    <EditText
        android:id="@+id/editTextPassword"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:hint="123456"
        app:layout_constraintBottom_toTopOf="@+id/buttonEnter"
        app:layout_constraintTop_toBottomOf="@+id/editTextPhone" />

    <Button
        android:id="@+id/buttonEnter"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:text="enter"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/editTextPassword" />

</androidx.constraintlayout.widget.ConstraintLayout> package com.leonidivankin.draftandroid.articles.autofill

import android.content.Intent
import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import com.leonidivankin.draftandroid.databinding.ActivityLoginBinding

class LoginActivity : AppCompatActivity() {

    private lateinit var binding: ActivityLoginBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityLoginBinding.inflate(layoutInflater)
        setContentView(binding.root)
        title = "LoginActivity"

        binding.buttonEnter.setOnClickListener {
            if (binding.editTextEmail.text.toString() == "login@gmail.com"
                && binding.editTextPhone.text.toString() == "+1 650 123 4567"
                && binding.editTextPassword.text.toString() == "123456"
            )
                startActivity(Intent(this, MainActivity::class.java))
        }
    }
} To move to , the fields must be filled in, and their values must meet certain requirements.
For example, take the following values: MainActivity email: phone: +1 650 123 4567 login@gmail.com https://youtu.be/oX3NIZMYpyo As you can see, it takes about 1 minute from startup to . This is still on the assumption that there were almost no errors in filling the data. Sometimes, finding complex bugs requires multiple rebuilds of the application and restarts on the device. This means constant filling of fields to get further into the app. MainActivity It turns out that with each restart we would lose about 1 minute extra. The ideal would be to jump directly into , bypassing , but in large projects, this is most often impossible due to the need to get a token from the server. MainActivity LoginActivity Another possible solution to this problem would be to assign default values to these fields. However, this is not always possible, because the fields may be in an external or in a . In this case, you won't be able to set default values. SDK webView As we continue I’ll show you how you can optimize the filling of the fields with kotlin-script. Autofill fields Finding the necessary adb commands I will use the commands to fill the text fields. To fill one text field, we need: adb Click on the text area. Fill it with text. To click on a text area, you must use the command: adb adb shell input tap x y where x and y are coordinates relative to the upper left corner of the device in pixels (px). They can be found in different ways: There is an that knows how to determine the click coordinates: adb-command adb shell getevent -l But there are difficulties with the translation of coordinates. You can read more . here On the real device, you can enable the developer's settings to display click coordinates or grid. You can add a listener, which will give absolute coordinates at each click. You can also find out through . Layout Inspector In this example, I am more comfortable using paragraph 4. Launch the Layout Inspector. Connect to the current process. Select the necessary . view Check its coordinates. It will depend on the size and type of the device. Most likely, your coordinates will be different. Note that these coordinates are in . And the command needs coordinates in . In my case it is x: 243=72, y: 3643=1092 dp adb px The final command turns out to be: adb shell input tap 72 1092 Go to the terminal and enter this command: The text input field is now active. To find out the pixel density, you need to go to : Device Manager In my case it is . Here the ratio between and is 3. The ratios are described in more detail . xxhdi dp px here Next, you must enter the desired text in the field. To do this, use the command: adb shell input text 'login@gmail.com' Enter it into the terminal and check that the necessary is entered.
Similar actions are carried out with other text fields: . Next, we need to click on the button, for this, there is a command: email phone, password adb shell input tap 72 1497 I've already put the coordinates I need there. We found all the necessary to fill in all the text fields and click on the button. Here is the complete list: adb commands adb shell input tap 72 1092
adb shell input text 'login@gmail.com'
adb shell input tap 72 1227
adb shell input text '+1 650 123 4567'
adb shell input tap 72 1362
adb shell input text 123456
adb shell input tap 72 1497 Now we need to build a script from these commands that will execute them instead of us. Creating a kotlin script You can build a script in different ways and in different languages. Here are some of the options: Make a . bash script Make a plugin for . Android Studio Make , etc. Gradle task But each of these methods has disadvantages. The bash script is difficult to maintain, the plugin for needs to be maintained for new versions of the environment, gradle tasks take a long time to run due to dependencies with and . Android Studio gradle gradle wrapper So I use another way - . The kotlin language is more familiar to the android developer, the script is independent of the project, android studio, and gradle. kotlin-script (kts) Create a scratch file using (Windows). In the window that appears, select : Ctrl+Alt+Shift+Insert Kotlin This will create a file. Notice the green arrow. This means that the file is executable and can be run. Uncheck so that the script won't restart every time you change the file. scratch.kts Interactive mode There is a special method that allows you to execute an from a kotlin script: adb command Runtime.getRuntime().exec() Insert all of our commands. My complete scratch file is as follows: fun run(){
   run("adb shell am force-stop com.leonidivankin.draftandroid")
   run("adb shell am start -n \"com.leonidivankin.draftandroid/com.leonidivankin.draftandroid.articles.autofill.LoginActivity\" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER")
   Thread.sleep(2000)
   run("adb shell input tap 72 1092")
   run("adb shell input text 'login@gmail.com'")
   Thread.sleep(500)
   run("adb shell input tap 72 1227")
   run("adb shell input text '+1 650 123 4567'")
   Thread.sleep(500)
   run("adb shell input tap 72 1362")
   run("adb shell input text 123456")
   Thread.sleep(500)
   run("adb shell input tap 72 1497")
}

fun run(command: String){
   Runtime.getRuntime().exec(command)
} This file consists of the we defined above with some additions. adb commands Note that I inserted the lines first: fun run(){
   run("adb shell am force-stop com.leonidivankin.draftandroid")
   run("adb shell am start -n \"com.leonidivankin.draftandroid/com.leonidivankin.draftandroid.articles.autofill.LoginActivity\" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER") These lines are needed when the application is running and you need to restart it. I also put the command in a separate method to reduce the amount of code: exec() fun run(command: String){
   Runtime.getRuntime().exec(command)
} Note that some commands have between them. This is to wait for the application to start or the previous command to be entered. The point is that adb commands are executed asynchronously. And there is a probability that the focus will be shifted to the next text field without waiting for text input in the current one. Thread.sleep() Total transition time to from launching is not more than 5 seconds, which reduced the time by 20 times. MainActivity https://youtube.com/shorts/R484DB_RScc?feature=share Conclusion As Steve McConnell wrote in Perfect Code, programming is a complicated thing. It gets even more complicated because you have to look for complex and nontrivial bugs. So, from my point of view, the role of any optimizations is rather high. Despite the fact that within the framework of the whole project, they seem microscopic. Try to optimize the launch of your application by autocomplete fields.
If you find bugs or know how else to improve the script, post in the comments.

Autofill Fields with a Kotlin Script

CTO (a handsome guy you see above😊) put his heart and soul into this article to teach you how to make logging great again. Keep reading and you will learn standard practices that everyone should take advantage of when dealing with logging. The article will give you a rough idea of what path you have to follow if you should deal with setting up logging principles on your project. Enjoy! Leobit Log purpose Problem statement 1 Hey %USERNAME%, yesterday %ANY_FEATURE_NAME% didn't work. There was absolutely no data displayed. The loader icon did not disappear for a few minutes. I had to cancel the demo, we can not proceed with this solution, we are losing customers. Fix this ASAP. URGENT!!! Sometimes something goes wrong, and we need to fix it, but in many cases, it will be unreproducible in your local environment. Moreover, you can have no access to production DB or event dumps (to reproduce it locally with dumped data). We definitely need logs here to figure out what happened. Let’s set up a logging system for a non-complex web application, but first, we need to understand what logs are usually used for. Logging in without objection is a necessary thing, which is usually used for: understand what the system is doing right now; trace chronology of actions performed by the system; information about various events in the system (errors, abnormal situations, etc.) other. Mandatory features of a well-designed logger should be: support of logging levels and filtering on these levels; rotation of logs (by date and size, for example); the ability to write logs, not only to files (for example, in the database, email, etc.) dynamic reconfiguration of logging (tracking the config file and changes in runtime according to changes). When to log? All cases when we need some logs can be actually covered by two scenarios: something is happening, and we need to know what something happened, and we need to know what It is simpler with the first one because we can most likely add some logs ad-hoc (using different techniques) to understand what is happening. A bit harder with the second one because you cannot add more logs if something already happened. So, in conclusion, try to log as much as possible. It is always easier to find something in a big heap if it is there than in a small one if it is not. At the same time, try to keep a balance. Overhead of logs usually decreases your app performance, be aware of it. And definitely use log levels. Log levels Depending on the logger, you can select the following levels of logging: Debug (the lowest level required for debugging); Info (informing about various actions of the system, steps of a process, for example); Warn (non-critical bug in the system, or remarks to the developers in connection with the behavior of the program); Error (an error that needs attention and solution, some failure); Fatal (a very critical error that causes the system to crash and malfunction, usually to stop the process). logger.Log(LogLevel.Debug, "Create a new instance of user");
var user = _userService.CreateNewUser();
logger.Log(LogLevel.Debug, "User created")

try 
{
  logger.Log(LogLevel.Information, "Setting up scope related permissions");
  _permissionService.ConfigurePermissions(identity.Scopes, user);
}
catch (Exception ex)
{
  logger.Log(LogLevel.Fatal, "Permissions could not be set", ex);
  ...
} In case your logger supports dynamic reconfiguration, you will be able to switch the default log level from to event on the running environment to have as much information as possible and change it back to after you figure out the issue. Info Debug Info What to log? Any valuable log record should contain at least three things. [ ] [ ] [ ] SystemInfo Context Message - timestamp, process id, thread id, or any other system tag SystemInfo - calling context can be shared for multiple log records, usually the name of caller module/service/method Context - log record message Message Most commonly used loggers will provide and parts out of the box. So you can care only about the . SystemInfo Context Message Information to log: Errors Requests and responses Valuable system information and events Usually, it is necessary to bound request and response log records. The HTTP header is often used to link these pairs. Such a header can be generated by the client, or it can be generated on the server-side. By adding it to the context of each line of the log, it will be possible with a slight movement of the hand to find all messages that have arisen within the framework of a specific request. And in the case of distributed systems, the header can be passed down the chain of network calls. X-Request-ID Centralized Log Storage Problem statement 2 Hey %USERNAME%, the payment did not succeed for yesterday order id is . I know you have access to logs. Can you check what happened? John Doe XYZ123 Our services: Public API gateway Authorization Product service Customer service Order service Pricing service Delivery Service Memes service Good luck to grep the necessary log record in one of these log storages (whatever these are a file, DB, or log storage). The solution is pretty obvious and straightforward. Here we need to set up centralized log storage. Ultimately, to have a single entry point with convenient analysis, so even Support can use the logging tool to some simple cases without developers’ involvement. CloudWatch In case you are using AWS as your hosting provider, you can try CloudWatch for your logs. Simply need to create IAM user with CloudWatch write access, create an access key/secret pair. And add AWS seq to your logger instance. In case of and it looks like C# Serilog profileService.RegisterProfile("PROFILE_NAME", new 
{
    AccessKey = "AccessKey",
    SecretKey = "SecretKey",
});

var configuration = new Configuration
{
    "LogGroup": "YOUR_LOG_GROUP",
    "Region": "eu-central-1",
}

loggerBuilder
      .WriteTo.AWSSeriLog(configuration)
      .CreateLogger(); Now all will be pre-aggregated and send to CloudWatch as a batch request on a scheduled basis. logger.Log("Something happened") LogAnalytics or AppInsights In case you are using Azure as a hosting provider, create and access token key for it. Azure Log Workspace loggerBuilder
    .WriteTo.AzureAnalytics(workspaceId: "WORKSPACE_ID_HERE", authenticationId: "SECRET_KEY_HERE")
    .CreateLogger(); Or you can use by merely connecting the necessary package (for it is ). Azure Application Insights .Net Microsoft.Extensions.Logging.ApplicationInsights Result: ELK "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that simultaneously ingests data from multiple sources, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. We will use to simplify the setup, but the same things can be done with pure ELK as well. Logz.io var configuration = new LogzioOptions
{
    DataCenterSubDomain = "listener-eu"
});

loggerBuilder
    .WriteTo.LogzIo("YOUR_SECRET", configuration)
    .CreateLogger() By launching the application, we can observe our messages not only in the console but also in Kibana. Structure logging Default text messages in logs are not widely used because they can be better. We shouldn't be working with flat text messages but with objects. Thanks to such logs, we can quickly build representations of our messages in different projections and conduct analytics. Structural logging complements this practice by providing developers with an easy way to annotate strings with additional data. The term "structural" has a double meaning: it can mean a natively structured data format like , but it often means that log lines are appended with key-value pairs (sometimes called logfmt). The additional structure allows developers to easily enrich strings with the information they need, rather than embedding it in the message texts themselves. JSON Briefly, the main idea is to use [Saturday, 20-Feb-21 09:37:21 UTC] [77238] OrderService: User { id: 12345, name: "John", age: 22 } ordered a product { id: 322, name: "Whiskey", price: 228 } or [Saturday, 20-Feb-21 09:37:21 UTC] [77238] OrderService: User userId=12345 userName=John userAge=22 ordered a product productId=322 productName=Whiskey productPrice=228 instead of [Saturday, 20-Feb-21 09:37:21 UTC] [77238] OrderService: User 12345 ordered a product 322 Having a data structure also makes the generated logs machine-readable (the key-value convention is used as a compromise in which both machines and humans can read data). This makes them suitable for various logging tools, many of which can request records from the production environment in near real-time. All progressive data storages provide the ability to quickly search necessary log records by applying filters in SQL style or event by making a few clicks on the properties you are interested in. So you will be able to filter all logs where user ages were more than 20 or so. This kind of logs usually contain redundant data, which allows you to extract almost anything from them - even information that no one thought about. For example, you can find out which HTTP path is the most requested. And if the API returns errors with a code of 500 (internal server error), you can find out the duration of the failed requests to accurately assess whether database timeouts cause this. Maximum efficiency Problem statement 3 Hey %USER_NAME%, I have just checked logs for recently failed order. There are about 500 records. It looks like the user used system quite intensively. Can you help to figure out steps to reproduce this issue? To make our life easier one more tip can be used. We have to keep an end-to-end correlation between the logs of independent services so that you can easily view all the activities that were caused by a specific request from the client. Progressive Web Frameworks have an excellent ability to pass correlation context in headers, so if your applications use direct calls for interservice communication, you can take advantage of this boxed functionality. However, suppose your backend architecture involves interaction through a message broker (RabbitMQ, Kafka, etc.). In that case, you still need to attend to passing the correlation context through these messages yourself. HTTP HTTP Tracing and Monitoring By tracing requests, we understand everything that happens inside, how the request lives further inside our application. We just need to build call chains, see data inside requests, errors in call chains, response times, the number of RPS. There are great tools for tracing are or . You to use , which they both support. Jaeger Zipkin OpenTracing Usually, tracing information can be collected from three sources: If you use load balancers, parse the logs from them and send them to Jaeger. Service mesh, which knows how to collect and send traces by itself (i.e., Istio). The services themselves, if they receive addresses through Service Discovery system or communicate directly. In this case, the trace from the services can be sent directly to Jaeger. To simplify our tracing experience, we can generate some kind of token that will bound all request lifecycle stages into one chain. So we need to generate this token, let’s call it a correlation identifier, as close as possible to the start of the activity, i.e., on the gateway or directly on the client (mobile or web). Since we are dealing with a backend application today, we will simply indicate the requirement for a mandatory "X-Correlation-ID" header in all requests to the web API. CONFIRMED TRACING HEADER BY W3C Note: X-Correlation-ID IS NOT The next step is to include received into your request handling pipeline until returning a response. Since the application that receives the events from the message broker queue has access to all the properties, we can use the obtained correlation identifier inside the consumer application and pass it further along the entire call chain. CorrelationId As a result, our logs began to contain not only within one service but also when interacting with other applications services. CorrelationId As a next developments stage you can categorize your logs and put them to more powerful systems Errors/warnings --> Sentry Tracing --> Jaeger Execution time and performance --> Prometheus On the output, you will be able to set up notifications, alerts, and any automated rules for monitoring something and taking actions (restart instance by webhook on fatal errors or send email to responsible engineer). Build timeline diagrams to find out bottlenecks Monitor the performance GDPR and Privacy Policy Do not forget that private information should be private, : DO NOT LOG OR STORE Credit cards information Cookies Login and passwords Access tokens Conclusion At least some logs are better than none. Logs in raw view are now so useful as metrics. A single will help you find problems more efficiently in a service that consists of multiple applications. CorelationiId Structured logs will allow you to query them more conveniently than ordinary text logs. And don't forget to follow the schedules and monitor. We hope you enjoyed reading this article and share our opinion on the importance of logging. If you are interested in more topics on software development, feel free to visit or contact our technical team to chat about logging :) Leobit Blog

Stories, News & Mentions about LOGDNA

Logdna’s Stories on HackerNoon

Logdna's News & Mentions