Get the most recent info and news about Bitnami on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

With over 1 million apps deployed per month, Bitnami makes it incredibly easy to deploy apps with native installers, as virtual machines, docker containers or in the cloud.

Bitnami

bitnami

Wind River Systems surveyed more than 200 auto industry companies to learn where should the auto industry focus its software-centric efforts.

Wind River Webinar: What Focus for Software-Centric Business Models?

We've tested more than 100 anti-malware apps to help you find the top malware protection and removal software for all your devices. When the IBM PC was new, I served as the president of the San ...

The Best Malware Removal and Protection Software for 2023

Trading options is the process of buying and selling stocks at a pre-negotiated price by a certain date. Options are a type of contract that gives the trader the right to buy or sell a particular ...

Best Options Trading Software

The best video editing software gives you post-production control over your footage. With visual content constantly growing in popularity, these apps are essential for marketing a business ...

Best video editing software (2023): Video editors for Windows, Mac and more

However, there are many different types of scanning software available, and different document types available. The best scanning software will be able to cater for a range of different needs and ...

Best scanning software of 2023

there's an almost mythical concept called the “10X developer,” meaning a software engineer who's such a high performer that she or he is perfectly capable of doing the work of 10 average-abled ...

Beyond The 10X Software Engineer: Focusing On The Bigger Picture

Despite economic fears, Gartner predicts spending on software will increase 11% in 2023, up to $879 billion from $790 billion in 2022. Why? The answer is simple. No business can survive in today ...

Four Crucial Steps For Successful Software Implementation

Recently software engineering has been one of the most trending fields. Chat GPT and the new technologies in many tools we use daily in different fields like graphic design and video editing make ...

Software Engineering 101

Who is the Master's in Software Engineering for? Drexel’s College of Computing & Informatics' Master of Science in Software Engineering (MSSE) is designed for students who hold a four-year bachelor's degree or master’s degree from a regionally ...

Master's (MS) in Software Engineering

When deploying multiple resources to Kubernetes, it is often challenging to keep track of all the resources, image tags, and environment variables distributed across multiple manifest files. To address this issue, manifest templating tools such as Kustomize and Helm exist, simplifying large, complex manifest files to templates with customization available through values.yaml. This provides flexibility with a single template across multiple environments (e.g., QA, staging, production), with changes configured via the values.yaml file. Additionally, it offers ease-of-use for end-users to deploy into any new environment with just a single command. However, solving one set of problems often introduces new challenges. Creating individual Helm charts can become difficult to manage as applications increase. It becomes essential to track changes across all Helm charts used by an organization, consuming significant engineering hours. Using community charts poses the challenge of pulling and customizing new charts, maintaining separate copies in the Helm repository, and consuming considerable time to pull and run these charts. Just as we've created a Helm chart for each application, bundling all required resources into a single chart, it would be advantageous to bundle all these micro-services into a single chart. Kubernetes Introducing Helm Sub-charts: Helm sub-charts are like mini-charts residing inside a larger Helm chart, akin to different players in your favorite video game. It's Helm's way of implementing inheritance. Charts can have dependencies, called , each with its values and templates. subcharts Why Do We Need Sub-charts? Sub-charts help break down large applications into more manageable parts, akin to splitting your favorite game into different levels. Breaking Down Big Stuff: Sub-charts can be reused in different projects, similar to using a favorite weapon or skill in various game levels. Reuse the Good Stuff: Sub-charts enable working on different parts of an app without accidentally affecting other parts, similar to cooking different dishes in separate pots. Keeping Things Separate: Things to Consider When Creating Helm Sub-charts: Before creating sub-charts, consider the following: A sub-chart is considered "stand-alone," meaning it can never explicitly depend on its parent chart. Consequently, a sub-chart cannot access the values of its parent, but a parent chart can override values for sub-charts. Helm sub-charts work well when the charts you want to deploy are used in combination with each other but are not directly dependent on each other's resources. Helm sub-charts work well when the charts you want to deploy are used in combination with each other but are not directly dependent on each other's resources. When a Helm chart has multiple Helm sub-charts, specifying the specific order can be challenging. Let's Build Some Helm Sub-charts: To create a master Helm with sub-charts, consider a master Helm chart example deploying Nginx and PostgreSQL. The folder structure is as follows: To create Helm sub-charts within the master Helm chart, add the following to the values.yaml file, describing the name of the Helm charts along with their versions and repositories. apiVersion: v2
name: my-master-chart
description: A Helm chart to deploy Nginx and PostgreSQL
version: 0.1.0
dependencies:
  - name: nginx
    version: 15.4.3
    repository: https://charts.bitnami.com/bitnami
  - name: postgresql
    version: 13.2.16
    repository: https://charts.bitnami.com/bitnami After adding the above to the values, run the following command to pull and add these dependencies to the charts folder. helm dependency update Once pulled, the folder structure should have .tgz files in the charts folder. To untar the .tgz file, use the command: tar -zxvf nginx-15.4.3.tgz After extraction, your file structure would resemble the one below: Just as we can manipulate inputs from the values file in individual Helm charts, we can use the global values.yaml file to manipulate the values of each of these Helm charts. By doing so, we can use the base Helm charts as templates and make all modifications from the global values.yaml file. This reduces complexity and simplifies the management of different input values in various Helm charts. Consider the following example to illustrate the use case. The values for Nginx mention the image tag, and PostgreSQL mentions the image tag along with the port number at which the PostgreSQL deployment runs. Here are the default values passed in the Nginx and PostgreSQL values.yaml files: Now, change the values in the master values.yaml file at the top level. Change the Nginx image from to and the PostgreSQL image from to . Also, change the PostgreSQL default port from to . 1.25.3-debian-11-r1 1.24.0-debian-11-r153 16.1.0-debian-11-r9 15.5.0-debian-11-r9 5432 5111 The values for both Nginx and images have changed, and the PostgreSQL port has been updated. PostgreSQL Liked my content? Feel free to reach out to my for interesting content and productive discussions. LinkedIn

Helm Sub-Charts Demystified: A Guide to Efficient Deployments

Your Definitive GitOps Guide For DevOps Experts And Newbies

Hi Hackernoon! Tech experts warmly welcomed my . Even a top-level expert from Weaveworks contacted me after reading it. And I was greatly surprised. After some warm words, we agreed that they would wait for the second part of my article. Well, the time has come. previous article on GitOps Let’s dive in! Content Overview The Multiple Environments Problem Values From Secrets Issue CI Ops vs GitOps So, Who Really Needs GitOps? Conclusion The Multiple Environments Problem We've already dealt with Flux workflow in the previous part. Now let's move on. Here is our Helm release. Suppose we rolled out changes to the development environment, tested everything, and wanted to roll out to the stage. How can we do that? We have GitOps, so obviously, we want to maximize the Git approach (merge code from dev to stage). But in this Helm release configuration, we can't do that because (for example) we have a different base in the stage environment (lines 20...23). What does GitOps say about managing environments? , GitOps says "Why don't you just have one environment, and you won't have to do anything stupid". I, for one, am not sure how many teams have only one environment. GitOps ideologues don't seem to be sure about that either. So they say, "Well, if you really need multistage, do it somewhere outside of the GitOps scope. Within your system, for example." CI/CD Values From So we think, review the options, and find a few. The first one is (lines 15, 16, 17). valuesFiles We can describe separate values files for individual environments and plug them into HelmRelease. But this scheme doesn't fit well to git merge either. Next, we try valuesFrom. Our environment namespace may contain one or more config maps and/or secrets (lines 22...25). From them, Helm-controller can take Values and customize our Helm chart. And I like that. I even make a separate Helm chart where I configure all microservices, link it to the application charts with the dependsOn key (line 16), and proudly call it a single point of platform configuration. And then, the following happens: I have several Kubernetes clusters. My colleagues come and say: "Let's write another config map, and there will be cluster-specific Values (lines 26...28). For example, cluster name, Ingress suffix, region, and access rights. And we will create this config map with , together with the cluster. Terraform When this happens, I need to take a weightier argument and consistently explain the depth of their downfall to my colleagues. Because in this particular case, our application configuration will depend on 4 repositories (application chart repository, platform setup repository, flux infrastructure repository, terraform file repository) and two deployment systems (Flux and terraform). This is where I can give you some advice. If you want to work with , you should have only one value in your variables and config map. It must be the name of the environment you are deploying to. Everything else should be done inside the Helm chart using templating mechanisms. valuesFrom Secrets Issue Next point. We have a Helm release with a line that should not be there - the database password (line 23). What does GitOps tell us about managing secrets? , It says Never store your passwords in clear text in a Git repository. What can we say? Thanks, Cap! The next thing GitOps offers us is: Let's generate your passwords randomly when you create your environment. And you will make all objects that require password protection based on those passwords. That way, the password will never leave the environment, everything will be secured, and everyone will be happy. Nice, sure. But how many people, for example, have their SQL database in Kubernetes? Or is a cross-plane installed there? As a rule, we don't create heavy, persistent services inside or from inside a Kubernetes cluster. : Read on for the following alternative approaches Let's use tools like Mozilla SOPS or Bitnami Sealed Secrets when we do public and private keys. The private key is written to the cluster. With the public key, we encrypt our passwords and commit to a Git repository. At the time of deployment, some tool inside Kubernetes decrypts this and gives it to applications. And a specially trained Ops team member will write this key to the cluster. Let him carry a printout of it in his jacket pocket and then carefully enter it by hand. OK, that seems like the best option. Now let's think about it. In , old but still relevant, there is, for example, the 31st question: "Can a user's account be disabled on all systems in 1 hour?". Your Ops guy quit. And if your Kubernetes clusters are not under a single sign-on system, then you must run across all clusters and delete his account from everywhere. The Limoncelli Test: 32 Questions for Your Sysadmin Team The next question is, "Can you change all privileged (root) passwords in 1 hour?". That secret key that your Ops guy compromised by getting fired is that root password. And to change it, you have to do certain things: Extract the private key from the cluster Decrypt what we have in the Git repository with it Make new keys Write the private key to the cluster Encrypt the passwords Commit them to git Run it through the release cycle and deprecate it And you have to do it in a particular order because we have an asynchronous pool model. Any careless rollout can turn your environment into a pumpkin. And rollback won't help here. A couple of words for fans of external secret stores like Hashicorp Vault. I haven't found a consensus on them, but from a common-sense point of view, using external secret stores makes Git an even more non-unique source of truth and thus goes against the GitOps approach. CI Ops vs GitOps As part of the GitOps concept's PR, WaveWorks has often referred to classic CI Ops as an outdated approach and anti-pattern. Why? Because with GitOps, we can separate the stage of the deployment, protect it, and manage it more clearly. Within CI Ops, the deployment stage becomes a part of the monolithic pipeline. And because of this, several disadvantages arise. We will now analyze this fact. But! To get a fair comparison, we must choose a "fighter" from the CI-systems side. Let's take the popular and familiar to many people Gitlab, which implements the concept of pipelines as code. Your CI code is in the same repository as your application and is actually part of it. This allows for greater CI/CD transparency, self-documenting pipelines, and knowledge transfer within the team... CI Ops vs GitOps: Security The first problem is poor security. It's like we have a CI system within CI Ops, and developers have full access to Kubernetes through it, and it's non-secure. And within GitOps, we have an asynchronous executor pool that we don't have direct access to, greatly increasing the system's security. Let's put Flux in and see what the actual security is. By default, Flux gives cluster administrator rights to all of its controllers. Obviously, it's not very secure. Let's try to improve this situation. I have found an for Flux multiuser use. We create a tenant and see that namespace and ServiceAccount are generated. article about multi-tenancy Flux controllers get administrator rights to this namespace, not all of them, just Kustomize controllers. Earlier, using to limit the Flux Helm controller was recommended. Now, they offer some strange mechanism with patching of controller's deployments so that it starts to pick up the account service from the namespace where it is deployed. Kyverno But let's just think about it, abstracting from the toolkit. We have two schemes: the synchronous push and asynchronous pull models. Which one is safer, assuming they both have admin rights? That's a rhetorical question because admin rights are admin rights. But even if we restrict infrastructure rights, these schemes have roughly the same access to infrastructure, passwords, and persistent data. They can harm. If you look at the problem a bit from the outside, you can see that in both cases, you have a Git repository. And it contains both application code and infrastructure code. And probably the only way to stop all the abuse here is to control what gets into your Git. Gitlab has a mechanism for (though only in the paid version for now, but as we know, Gitlab's paid features are quickly moving to the community version). Merge request approvals This mechanism prevents you from merging your changes into the working branch of your repository until the right number of peers have reviewed your changes and confirmed that they are OK. This rather simple mechanism increases security many times over—much more than the difference between pull and push models. CI Ops vs GitOps: Rollback Procedure Another disadvantage that GitOps ideologues cite is the difficulty of rolling back CI. Their guides say that the CI system is not meant to be the only source of truth. This means that when working with the CI system, we cannot determine what we have clustered and what we will roll back to. But with GitOps, we have an infrastructure repository synchronizing with the infrastructure, so we do a git revert. And everything is fine. Let's think. GitOps only covers the deployment stage. But since git cannot be the only source of truth, we should think about rollback at the stage of building our artifacts. Besides, Flux has everything in place to make rollbacks difficult or impossible. For example, we can awfully tag a docker image (line 19). Also, we can use the Helm chart version range (line 11). And if we use Helm chart from the application repository, we may need to roll back the application repository to roll back our environment. In addition, don't forget about possible dependencies in a separate repository, which we may also need to roll back. valuesFrom Flux also has recommendations for organizing infrastructure repositories. For example, Monorepo. It is when we have the infrastructure for all our environments in one branch in different folders. Or, for example, Repo per team. It is when each team has its own folder in a shared infrastructure repository. This is where you must be very careful when rolling back so you don't delete extra. As for CI Ops, Gitlab may be the only source of truth. Gitlab has a so-called environment mechanism, where we specify the name of an environment in a Pipeline Deploy step and see its state in the environment tracking. It immediately becomes transparent which pipelines, from which branch, and from which commits are got. We also have a visual display of the pipelines, where we can see what rolled in there and whether those stages passed. We can roll back to the right stage. It is decisively impossible when looking at the commit history in an infrastructure repository in the GitOps concept. Alternatively, we can , thus making the pipeline atomic. And this would really be the only source of truth. save the docker image as an artifact in the pipeline As a result, to rollback within the implementation described above, you would just need to restart the Rollout Job in the Pipeline History in the Gitlab visual interface. CI Ops vs GitOps: The Problem of Multiple Clusters According to the GitOps concept developers, it is difficult to propagate changes to multiple clusters in CI Ops. Allegedly, in a CI system, we need to script, script, and script. But in the beautiful GitOps, we just put Flux in a new cluster and give it an infrastructure repository. One moment and our application is already in the new cluster. Let's start analyzing. Here is our custom resource Helm-release in the infrastructure repository. I have 2 questions when looking at this Helm release: Where did this application come from? Where is its repository? Where are the build logs and test run results? From which branch and from which commit is it deployed? Where is the application deployed to? How many fluxes in which clusters look at this infrastructure repository and pull configuration from there? "Out of GitOps scope" can be the answer to the first question. We just need to write a proper HelmRelease push script that will write all the information we need in the commit message. For the second question, Flux offers a with the creation of an infrastructure repository (flux-fleet) to manage Flux specifically. Before installing Flux in a cluster, we first commit all YAMLs describing the installation of flux controllers and related custom resources. Then we apply them to the target cluster. When we need to connect a git repository and Kustomize to a new infrastructure repository, we commit to that infrastructure repository again. bootstrap mechanism So we have the entire list of our clusters and all the accesses in one repository. And it works fine until some manual operations happen. For example, your Ops expert quits. Let's say there's a Kubernetes cluster somewhere where he put Flux and gave him manual bypass flux-fleet access to your infrastructure repositories. You may never know if you didn't do the access revocation and key regeneration. So I'd recommend writing down step zero on your checklist as creating a flux-fleet repository and bootstrap Flux. Otherwise, you will have secret knowledge that may one day be lost with the employee. What do we have in CI-ops? In Gitlab, we can declare reusable pieces of code (lines 1...9). We can inherit from these pieces of code (lines 13, 19, 25) and declare some additional variables (lines 15, 21, 27) when we make a new environment. In this case, it is the name of the cluster relative to which we will change the functioning of our scripts. In this way, we will deploy to a new environment and get its automatic tracking. So, Who Really Needs GitOps? There are a few cases where I think a GitOps approach can be justified: You don't have a CI/CD system, or it can't provide normal deployment for some reason. Improved security. GitOps can really add to the mix of measures to improve the security of your infrastructure. The lion's share of security issues are "out of GitOps scope" and should be addressed before or in conjunction with GitOps implementation. You have a large, multi-component platform that is tested and deployed. In this sense, a GitOps infrastructure repository is a good way to make a "stable" version of the platform and operate it as a single entity. You have processes where the pull model has advantages over the push model (e.g., canary deployments). So what is GitOps? For those who like precise formulas, GitOps = Continuous Delivery - Continuous Deployment. So it's not a CD concept. It's just (one more) way to organize automatic deployment. By and large, this methodology brought almost nothing new. The methodology authors took the ideas of Infrastructure as code (IaC), added a PULL-based approach with a reconciliation loop from SCM of the last generation (puppet, chef), and made it mandatory to keep the configuration in GIT. Indeed, a new and key concept of GitOps is the concept of Git as a single source of truth, but this concept is clearly mediocre and has some flaws. In fact, it's an attempt to take a deployment step from an application delivery pipeline and put it into Git. One of the key principles of DevOps is that there are no bad actors. There are bad processes. Hence the blameless culture and the understanding that you need to organize processes in such a way that the problem becomes impossible. But because of the limited concept of GitOps, it has too many gaps and potential problems, too many things to do "out of GitOps scope". All this leads to misunderstanding of how to organize processes properly and, consequently, to workarounds with all the consequences. To build a good GitOps process, a company needs high IT maturity because the authors of the concept left out too many important things. So, GitOps does not provide any significant advantage over a well-organized CI Ops process, although it significantly increases the final system complexity and labor costs. And in my opinion, many GitOps tools understand this by implementing GitOps in addition to image building, e.g., Gitkube and JenkinsX. Conclusion What else are we missing to make us happy? We obviously don't have enough notifications. We have an asynchronous pull model. In this model, we don't understand what exactly this pull executor is doing on its side if it doesn't explicitly notify us. Let's add the installation and configuration of the notification controller to our checklist and match it. So, to work with GitOps, we need to do: Flux-fleet repository Bootstrap flux in a Kubernetes cluster using the flux-fleet repository Script for building and pushing images in the Docker registry Infrastructure Git repository Account for CI system access to the infrastructure GIT repository Script to generate and push the HelmRelease file Helm Repository Account for CI system access to the Helm repository Script to build and publish the Helm chart Flux account for the infrastructure repository Flux account for Helm chartʼs repository Flux account for the application repository Flux account for cluster access for an engineer on the Ops team GIT repository for "single point of platform configuration" Flux account to access the GIT repository Slack/Teams channel for Notification Controller Most of these items are "out of GitOps scope". And it's not a methodology problem. But as an experienced engineer, I understand that I can't do GitOps without these items. What about CI Ops? As part of CI Ops on Gitlab, we don't need to do many extra actions to make GitOps work. We need to: Make the CI script to build and publish a docker image Add Helm release deploy to the script via Helm apply Create an account for the CI system to access Kubernetes With that, deprived of the illusion that it's secure, we'll be more responsible in setting up access to the CI system and the cluster. In addition, we get a built-in integrated docker registry, built-in environment handling, built-in secrets handling, built-in notification of the execution of the pipeline stages, and some integrations.

What Is GitOps And Why Is It (Almost) Useless? Part 2

is an excellent key-value distributed database used internally by Kubernetes and managed by the CNCF. It's a great option, and that's the reason why Apache APISIX uses it too. Yet, it's not devoid of issues. etcd First, some mention scalability, but one can expect this from a distributed data store that values consistency. Another issue may be the need for more familiarity with etcd. It's relatively new, so your Ops team may need help operating it correctly while having decades of operating MySQL or Postgres. Finally, only a few etcd users are aware that it lacks maintainers: In the last few months, primary maintainers G.L. (Amazon, announcement) and S.B. (Red Hat) have stopped actively participating in the project. This leaves the project with only one active and two occasionally-reviewing maintainers, M.S. (Google),  P.T. (Google), both are relatively new to the project (1 month and 1 year of tenure) and S.P.Z. (IBM). Other maintainers are either dormant or have very minimal activity over the last six months. . The project is effectively unmaintained (emphasis mine) -- Google Groups of Kubernetes Steering Committee, March 2022 For all those reasons, you may prefer to use a standard SQL database with Apache APISIX. In this post, I'll show how you can use MySQL. The kine project It would be a lot of effort if each product had to introduce an abstraction layer and different adapters for both etcd and other databases. kine is a project that aims to offer a translation step between etcd calls and other implementations: Kine is an etcdshim that translates etcd API to: SQLite Postgres MySQL NATS Jetstream Features Can be ran standalone so any k8s (not just K3s) can use Kine Implements a subset of etcdAPI (not usable at all for general purpose etcd) Translates etcdTX calls into the desired API (Create, Update, Delete) -- Kine (Kine is not etcd) In essence, kine is a Go library that translates etcd calls to the datastore you want (among those implemented). Yet, using kine directly is a non-trivial effort. Fortunately, , the company that gave Apache APISIX to the Apache Software Foundation, provides a component already focused on APISIX usage. api7 ETCD adapter ETCD adapter wraps kine to be APISIX-specific: ETCD Adapter mimics the ETCD V3 APIs best effort. It incorporates the kine as the Server side implementation, and it develops a totally in-memory watchable backend. Not all features in ETCD V3 APIs supported, this is designed for Apache APISIX, so it's inherently not a generic solution. -- ETCD adapter Two things of note: At the moment of this writing, the adapter supports either local in-memory storage or MySQL It's available as an embeddable library but also as a standalone component Therefore, we can design our architecture as the following: Demo Let's implement the above architecture with an additional admin UI over MySQL. I'll use Docker Compose: version: "3"

services:
  apisix:
    image: apache/apisix:3.4.0-debian                         #1
    volumes:
      - ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
    ports:
      - "9080:9080"
      - "9180:9180"
    depends_on:
      - etcd-adapter
    restart: always                                           #2
  etcd-adapter:
    build: ./etcd-adapter                                     #3
    volumes:
      - ./adapter.yml:/etcd-adapter/conf/config.yaml:ro       #4
    depends_on:
      - mysql
    restart: always                                           #2
  mysql:
    image: bitnami/mysql:8.0                                  #5
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: etcd
      MYSQL_PASSWORD: etcd
      MYSQL_DATABASE: apisix
  adminer:
    image: adminer:standalone                                 #6
    ports:
      - "8080:8080"
    environment:
      ADMINER_DEFAULT_SERVER: mysql
    depends_on:
      - mysql Latest version of Apache APISIX, yeah! To avoid any failure with dependencies between containers, restart until it works. Kubernetes's manifests would involve health checks still needs to provide a container. We need to build from the source code api7.ai Override the default configuration file with a context-specific one The regular MySQL image didn't work for me. Let's take the one from Bitnami Adminer, formerly known as PHP myAdmin, will help to visualize the database state ETCD-adapter's configuration looks like this: server:
  host: 0.0.0.0                 #1
  port: 12379

log:
  level: info

datasource:
  type: mysql                   #2
  mysql:
    host: mysql                 #3
    port: 3306                  #3
    username: etcd              #3
    password: etcd              #3
    database: apisix Bind any IP since Docker will assign a random one Implementation type. The default is ; we need to change it. btree As configured in the file docker-compose.yml Finally, here's Apache APISIX configuration: deployment:
  admin:
    allow_admin:
      - 0.0.0.0/0
  etcd:
    host:
      - "http://etcd-adapter:12379"   #1 Use this etcd instance, which is the adapter Testing Now that we are set let's test our system by creating a route: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{                                          
  "methods": ["GET"],
  "uris": ["/get"],
  "upstream": {
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}' http://localhost:9180/apisix/admin/routes/1 We can now get it: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' http://localhost:9180/apisix/admin/routes/1 We can also check via the Adminer interface that it has been persisted via MySQL: Unfortunately, we need to stop at this point. Getting all routes doesn't work: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' http://localhost:9180/apisix/admin/routes {"header":{"revision":"1689689596"},"message":"Key not found"} Worse, using the route fails: curl localhost:9080/get {"error_msg":"404 Route Not Found"} Conclusion etcd is an excellent piece of infrastructure Kubernetes uses, but there might be better choices in some contexts. Worse, it might become a security threat in the future - or is already one, because of the lack of maintenance. Being able to move away from etcd is a considerable benefit. kine offers an etcd-compatible facade and multiple implementations. Using kine with Apache APISIX requires some adaptation effort, already done in ETCD-Adapter. Currently, ETCD-Adapter is not feature-complete (to say the least) and requires more love. That's why it was not donated to the Apache Foundation yet. If you're a Go developer and are interested in the project, feel free to subscribe to the to offer your help. Apache APISIX mailing list and/or join our Slack The complete source code for this post can be found on . GitHub To go further: Kine ETCD Adapter "Goodbye etcd, Hello PostgreSQL: Running Kubernetes with an SQL Database" Originally published at on July 23rd, 2023 A Java Geek

Apache APISIX Without etcd?

Creating a Google Kubernetes Engine (GKE) Cluster with Terraform in a Custom VPC

Efficient AKS Cluster Provisioning in a Virtual Network Using Terraform

When learning a new technology, sometimes I just want to see it work. It gives me a baseline to extend my ideas, to see what is possible, and to imagine what it can become. While reading through documentation or following tutorials, I am often greeted with numerous options and configurations. After seeing all the dazzling capabilities, I always want to see the core, the fundamental. Didn’t you ever just want to experience a simple working copy? Hence, I just want to see it work. This series aims at minimizing the possibility of having a missing link and encourages you to build your next innovative solution based on what you learned here. What is Kafka Stream? According to , Kafka Streams is a client library for building applications and where the input and output data are stored in Kafka clusters. Kafka microservices, What do we want to achieve? Run Kafka with KRaft in docker. Run a producer on your local machine (not in docker) that publishes messages to a topic. Run a consumer on your local machine (not in docker) that pulls a message whenever a message is published in a topic.] Resources You can find the source code at my GitHub repository Docker image: Bitnami Kafka Run Kafka in Docker Navigate to where the docker-compose.yml is. Run the below command in cmd to start a docker container of Kafka in the background. docker-compose up -d Docker-compose.yml version: "3"
services:
  kafka:
    image: "bitnami/kafka:3.2.3"
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 It controls whether the Kafka broker allows connections over plaintext listeners. A plaintext listener allows connections over an unencrypted and non-authenticated channel. It means the data exchanged between the client and the Kafka broker is sent in plain text. ALLOW_PLAINTEXT_LISTENER: Enable KRaft mode. KAFKA_ENABLE_KRAFT: The hostname a client should use to connect to the Kafka broker. You need this for both producer and consumer to communicate with Kafka. KAFKA_CFG_ADVERTISED_LISTENERS: Let’s publish a message to Kafka Navigate to where is at. Start a Powershell and run the following command. KafkaStream.Producer.csproj dotnet run When you see the message: “Message delivered to input-topic”, that means you did it! What is in the Producer? We first setup the configuration for the producer. Build a message Produce the message var config = new ProducerConfig
{
	BootstrapServers = "localhost:9092",
	ClientId = "producer-1"
};

using (var producer = new ProducerBuilder<string, string>(config).Build())
{
	var message = new Message<string, string>
	{
		Key = null, // Set the key if you want to partition the messages
		Value = "Hello, Kafka!"
	};

	try
	{
		var deliveryResult = await producer
                                    .ProduceAsync("input-topic", message);

		Console.WriteLine($"Message delivered to 
                            {deliveryResult.TopicPartitionOffset}");
	}
	catch (ProduceException<string, string> ex)
	{
		Console.WriteLine($"Delivery failed: {ex.Error.Reason}");
	}
} What is bootstrap-server? It serves as an entry point for a client to connect to a Kafka broker. Let’s setup the consumer Navigate to where KafkaStream.Consumer.csproj is at. Start a Powershell and run the following command. dotnet run The initialization will take a few seconds. Then you should see the following messages. Consuming messages from topic: input-topic
Press any key to exit
Received message: Hello, Kafka! from partition [0] offset 0 There you go! What is in the Consumer? The main thread sets up the Kafka consumer and starts a background thread to handle message consumption. In the background thread, the code runs in a continuous loop, until a cancellation is requested. A cancellation can be requested by pressing any key while the program is running. const string Topic = "input-topic";

// Configure the consumer
var config = new ConsumerConfig
{
	BootstrapServers = "localhost:9092",
	GroupId = "consumer-group-1",
	AutoOffsetReset = AutoOffsetReset.Earliest,
	EnableAutoCommit = false // Disable auto commit to have more control over offsets
};

using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
	// Subscribe to the topic
	consumer.Subscribe(Topic);

	Console.WriteLine($"Consuming messages from topic: {Topic}");

	// Start consuming messages in a background thread
	var cancellationTokenSource = new CancellationTokenSource();
	var cancellationToken = cancellationTokenSource.Token;

	var consumerThread = new Thread(() =>
	{
		try
		{
			while (true)
			{
				try
				{
					var consumeResult = 
                          consumer.Consume(cancellationToken);

					Console.WriteLine($"Received message: 
                          {consumeResult.Message.Value} from partition 
                          {consumeResult.Partition} offset 
                          {consumeResult.Offset}";

					// Manually commit the offset to mark the message as consumed
					consumer.Commit(consumeResult);
				}
				catch (ConsumeException ex)
				{
					Console.WriteLine($"Error occurred: 
                        {ex.Error.Reason}");
				}
			}
		}
		catch (OperationCanceledException)
		{
			// This exception will be thrown when cancellation is requested.
			Console.WriteLine("Cancellation requested");
		}
		finally
		{
			consumer.Close();
		}
	});

	// Start the consumer thread
	consumerThread.Start();

	// Wait for a key press to exit
	Console.WriteLine("Press any key to exit");
	Console.ReadKey();

	// Request cancellation and wait for the consumer thread to stop
	cancellationTokenSource.Cancel();
	consumerThread.Join();

	Console.WriteLine("End Consumer");
} What does offset do? In this context, it tells the consumer where it left off. Gotcha While developing your own producer/consumer, if you get an error message indicating the hostname cannot be resolved, check KAFKA_CFG_ADVERTISED_LISTENERS. If you still run into the same issue, consider updating the host file. Conclusion In this article, we successfully run Kafka in a docker container, produce a message to a topic, and consume a message from the topic. There are limitless applications of using Kafka. Go ahead and play with the project, and build something interesting of your own!

I Just Want to See It Work: Kafka Stream, C# Producer and Consumer

How to Create Amazon EKS Cluster Within its VPC using Terraform

PHP Classes Use Case for Database Management in a Document Builder

How to Configure TLS for Apache APISIX 

Grafana Loki: Architecture Summary and Running in Kubernetes

To tighten security on your WordPress website, you want a 2-factor authentication (2FA) mechanism to provide identity approximation, and a Captcha to prevent notorious bots from rampaging the login page (or any other equivalent). With that said, there is , a top player in WordPress security, which can provide a 2-factor authentication feature for your WordPress website. wp-admin Wordfence Along the line, you decided to utilize , a recently released but promising Captcha service due to its user-friendly and more privacy focus approach. You enabled both of them as instructed, and decided to log out of your admin account to test out the services, but then . Cloudflare Turnstile The problem On first look, the services seem to work. Cloudflare Turnstile appears on the login page and the login attempt leads to a 2-factor authentication on your screen. You type in the 2FA code and expect to be redirected to the WordPress dashboard, but the redirection turned out to be unexpected. You are redirected to the login page again, with an error message from Cloudflare Turnstile: Please verify that you are human. You think the 2FA code was mistyped, or somehow the Cloudflare Turnstile accidentally misrecognize the request as a bot and decide to try again, but unfortunately, the result is the same. At this point, you are stuck in a login loop and completely locked out of the WordPress admin dashboard. What happens when Cloudflare Turnstile is disabled? As the error message is from Cloudflare Turnstile, perhaps, disabling this Captcha service can solve the issue. However, with no access to the WordPress dashboard, you decide to disable it on the Cloudflare Dashboard instead. In the Cloudflare Turnstile Analytics dashboard, you see a noticeable spike in - unsolved Captcha requests. The time stamp aligns with when you start facing the problem. blue values The thing is, there is no trivial way to disable Cloudflare Turnstile here, so you decide to After the rotation, you still face the same loop on the WordPress login site. It might take a bit to propagate the change, or so you thought. That's correct, but after about 15 minutes, the change propagates and you see this Cloudflare Turnstile error message upon refreshing the WordPress login page. rotate the secret key. Invalid domain. Contact the Site Administrator if this problem persists. Now, you cannot even go to the 2FA code screen. Your username and password cannot be submitted as Cloudflare Turnstile completely blocks the login process after secret key rotation due to security reasons. Somehow, the problem is now even worse than before. Perform WordPress administrative operation but not via admin dashboard, is it possible? In the large picture, this can potentially be a problem with other Captcha services too. At the moment, it is certain that the Wordfence 2FA and Cloudflare Turnstile are incompatible and causing havoc. Disabling these services and reverting the WordPress site to a known and good state is the goal, but that seems to be impossible when being locked out of the admin dashboard, or is it? WordPress backup and restoration is trivial from the admin dashboard with the help of plugins, All-in-One WP Migration for example. This path is not available at the moment. Infrastructure snapshot restoration is another choice, but what if you don't have one? How about directly accessing and modifying WordPress content at the file system level? If your WordPress site is fully managed by a third party, you might have limited control of the web server. However, if an option is available from your WordPress hosting provider, or you are self-hosting, you can SSH directly to the server and modify your WordPress installation via an admin dashboard. Resolve the login loop by disabling WordPress plugins via SSH Following the previous article: , let's say your WordPress site is hosted on Google Cloud using WordPress Certified by Bitnami and Automattic on a Compute Engine, then there is an out-of-the-box and secure way to SSH to the web server via Google Cloud Compute Engine Dashboard (marked yellow in the picture below). Host a free WordPress site with Google Cloud and Cloudflare The secure SSH session is fully managed by Google Cloud, so you don't have to worry about any form of configuration. After a bit, the terminal of your virtual machine (Compute Engine). Your terminal may resemble or vice versa depending on the virtual machine configuration. For an out-of-the-box setup from Bitnami, it should be the same. The goal is to find out where WordPress plugins (Wordfence and Cloudflare Turnstile) reside on your web server and disable them from there. With that said, run the following commands: # Move to where the plugin folder locates
cd ../bitnami/stack/wordpress/wp-content
# Rename an existing, "broken" folder to disable the plugins
sudo mv plugins plugins-old At this point, your plugins are all disabled, but no worries, they can be restored later. Don't close the SSH window, but let's put that aside for now. Then, go back to your WordPress admin login page ( or any other equivalents). As both Wordfence 2FA and Cloudflare Turnstile are disabled, you can log in to the WordPress admin dashboard without any problem. Once you are in the dashboard, your browser stores a login session and it survives page refresh, so do not log out of this step. wp-admin Now, return to the Compute Engine SSH window, and run the following command. sudo mv plugins-old plugins By renaming the folder back to the original one, now your WordPress site detects the existence of plugins and applies them accordingly, . A refresh of an admin dashboard can reflect the changes. The problem is solved, and the login loop is now gone! plugins-old including the problematic login loop bug That said, as your login session is still there, you can simply go ahead and disable Wordfence 2FA and Cloudflare Turnstile in the WordPress admin dashboard. Conclusion Wordfence and Cloudflare Turnstile are great services to up your WordPress security game. That said, with Wordfence 2FA specifically, there seem to be problems regarding its interaction with Captcha service in general, or at least, in a specific set of conditions. Incompatible with Wordfence 2FA Login Security 2FA conflicts with reCAPTCHA plugins Whether you can SSH to the web server is an obstacle, as, you are using hosting on a fully managed provider. If you do have access, using SSH and directly modifying files on the web server, the issue can be resolved with relative ease. Hopefully, this tutorial help you recover the website in a matter of minutes. Interested in web development, IT content, and more? My other articles might be helpful to you! How to back up Microsoft 365 with Synology Active Backup? Is AMD GPU better than Nvidia on Linux? It depends 🐞 Also, let's connect! LinkedIn GitHub Twitter

How to Fix WordPress Admin Login Issue

Docker Compose is a great tool for development but I've never found a good example of how to use it for an ideal development workflow. Containers are immutable and thus by default are not an ideal development environment because development is fundamentally about changing things and then running the change. Most Docker Compose blogs talk about mounting your code repository as a volume which gets you around the immutability problem and mapping a port, but they don't go much past that. There are many more little tricks you can do to make your development experience better. Here I gathered all the tricks I've used to make Docker Compose a great development workflow. Create a Development Container Your development container will probably diverge from your production and CI containers so you can handle having two diverging containers by either having a multi-staged build or having a separate Dockerfile for development. I normally fall into having a separate Dockerfile because your production and CI containers are often optimized for caching and being efficient to ship the container around. Trying to fit a development workflow into that can become clumsy and difficult to do cleanly. The downside is that you may duplicate some Dockerfile code between the two files. However, this should be small since the development container is often very few Dockerfile lines. If you do use the staged build approach, use the attribute in your Docker Compose file. target Sync User Permissions It's good practice to not run as root but also for you development container it's nice to sync up the container user with your local user. This ensures your files mounted outside the container will have the correct permissions. To do this: 1. Add some build args to your Dockerfile ARG userid
ARG groupid 2. Create a user in your Dockerfile with sudo access # User setup
RUN addgroup mygroup --gid $groupid
RUN useradd -ms /bin/bash -u $userid -g $groupid myuser
RUN echo 'myuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers 3. When building, pass in your build args docker-compose build --build-arg userid=$(id -u) --build-arg groupid=$(id -g) Share Secrets There is a attribute you can use in Docker Compose, however, since this is for development we don't need to secure a production environment, I found it easier to just mount some shared files and directories. Below is an example of some "secrets" I like to share. secrets services:
  myapp:
    volumes:
      - .:/opt/myapp
      - $HOME/.git:/home/myuser/.git
      - $HOME/.ssh:/home/myuser/.ssh
      - $HOME/.netrc:/home/myuser/.netrc
      - $HOME/.pypirc:/home/myuser/.pypirc Share Git and SSH See above. If you share your and directories with your development container you can execute git commands in your container without needing to jump in and out. This makes your development workflow much more enjoyable with fewer terminals open or fewer keystrokes. .git .ssh Install Some Development Tools Especially if you use a separate Dockerfile and thus don't need to worry about the container size, install some extra development tools. Some apt packages I like to install: procps, nano, vim, sudo, htop, tmux, redis-tools, autojump, openssh-client, wget, unzip And one of my other favorite tools: thef*ck Share Shell Environments Since you have a development container you can now really easy make a share development environment setup. I like to include a shared in my repository and then in the Dockerfile copy that to the user's home directory. .bashrc COPY .bashrc /home/myuser/ Remember Your Command History Since containers are immutable, they will lose your command history on rebuild. You can solve this by linking in a file in your Dockerfile. .bash_history RUN ln -s /opt/myapp/.bash_history /home/myuser/.bash_history Full Setup Below is the full setup, , , and additions. Dockerfile.dev compose.yaml .gitignore Dockerfile.dev FROM ubuntu:bionic

ARG userid
ARG groupid

WORKDIR /build

# apt layer
RUN apt update
RUN apt install -y python3.8 python3.8-venv libpython3.8

# dev tools
RUN apt install -y procps nano vim sudo htop tmux redis-tools autojump openssh-client wget unzip

# pip, pdm setup
RUN python3.8 -m pip install pip --upgrade
RUN pip3.8 install --upgrade pip
RUN pip3.8 install pdm thefuck

# User setup
RUN addgroup mygroup --gid $groupid
RUN useradd -ms /bin/bash -u $userid -g $groupid myuser
RUN echo 'myuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

WORKDIR /opt/myapp
USER myuser

COPY .bashrc /home/myuser/
RUN ln -s /opt/myapp/.bash_history /home/myuser/.bash_history compose.yaml version: "3.9"

services:
    myapp:
        network_mode: host
        build:
            context: .
            dockerfile: Dockerfile.dev
        command: /bin/bash
        environment:
            - DEV_CONTAINER=1
            - PYTHONUNBUFFERED=1
            - XDG_DATA_HOME=/opt/myapp/.local/share
        volumes:
            - .:/opt/myapp
            - $HOME/.git:/home/myuser/.git
            - $HOME/.ssh:/home/myuser/.ssh
            - $HOME/.netrc:/home/myuser/.netrc
            - $HOME/.pypirc:/home/myuser/.pypirc
        ports:
            - 8888:8888
            - 8000:8000
            - 8080:8080
            - 8081:8081
            - 3000:3000
            - 5000:5000
        shm_size: 4gb
        depends_on:
            - redis

    redis:
        image: 'bitnami/redis:latest'
        ports:
            - 6379:6379
        environment:
            - ALLOW_EMPTY_PASSWORD=yes .gitignore additions .bash_history
.local

Using Docker Compose for Development without Going Crazy

Introduction is a streaming platform that is used to process data in real-time. In a world where data is king, Kafka is a valuable tool for developers and data engineers to learn. However, setting up Kafka locally can be frustrating, which can discourage learning. In this article, I will show you the fastest way to set up Kafka for development using Docker, and also show you how this setup can support connecting locally and from other local Docker containers. Kafka This tutorial assumes that you have some knowledge of using Docker and for development. If you are new to Docker, I recommend reading first. If you need to install Docker, follow the instructions . docker-compose this article here Kafka Overview The below diagram shows a high level overview of what Kafka is for beginners. (there's lot more to Kafka, like Zookeeper, Consumer Groups, Partitions, etc. but we'll leave that for another time.) Kafka categorizes data into topics. A topic is a category or feed name to which records are published. Producers publish messages to a specific topic. The messages can be in any format, with JSON and Avro being popular options. For example, a social media platform might use a producer to publish messages to a topic called posts whenever a user creates a post. Consumers subscribe to a topic to consume the records published by producers. In the social media example, there might be a consumer set up to consume the posts topic to perform safety checks on the post before it is published to the global feed, and another consumer may asynchronously send notifications to the user's followers. Setting up Kafka on Docker We will make use of the bitnami images for Kafka and Zookeeper. I prefer this over the wurstmeister images because it is easier to setup and more actively maintained. We will also make use of the tool to manage our containers. docker-compose Create a file called and add the following content: docker-compose.yml # docker-compose.yml
version: "3.7"
services:
  zookeeper:
    restart: always
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper-volume:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    restart: always
    image: docker.io/bitnami/kafka:3.3
    ports:
      - "9093:9093"
    volumes:
      - "kafka-volume:/bitnami"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper
volumes:
  kafka-volume:
  zookeeper-volume: Notice that we set a few environment variables for the Kafka container. For more information, see the image documentation. See that we have also defined a service. This is because Kafka depends on Zookeeper to store metadata about the topics and partitions. For development purposes, you don't need to interact with it and
you can safely ignore it for the time being. bitnami/kafka zookeeper To run this, simply run and you should see the following output: docker-compose up -d $ docker-compose up -d
Creating network "kafka-on-docker_default" with the default driver
Creating volume "kafka-on-docker_kafka-volume" with default driver
Creating volume "kafka-on-docker_zookeeper-volume" with default driver
Creating kafka-on-docker_zookeeper_1 ... done
Creating kafka-on-docker_kafka_1     ... done Interacting with the Kafka container from your local machine kafkacat offers a simple command line interface to interact with Kafka. It is a great tool to use to sanity check that Kafka is running. To install kafkacat, follow instructions from depending on your operating system. https://github.com/edenhill/kcat To sanity check that Kafka is running, run the following command to list all topics currently in Kafka: $ kcat -b localhost:9093 -L  # list all topics currently in kafka
Metadata for all topics (from broker 1: localhost:9093/1):
 1 brokers:
  broker 1 at localhost:9093 (controller)
 0 topics: Note that we are using instead of the default 9092 port. This is because we are using the port that is exposed to our local machine. localhost:9093 To test out the producer, run the following command: $ kcat -b localhost:9093 -t test-topic -P  # producer
one line per message
another line The default delimiter between messages is a newline. When you are done, press to send the messages.
(FYI - clicking will not work, you will have to try again.) ctrl-d ctrl+c To read the messages you have produced, run the following command to start a consumer: $ kcat -b localhost:9093 -t test-topic -C  # consumer
one line per message
another line
% Reached end of topic test-topic [0] at offset 2 Publishing arbitrary text is not quite what we want though, so let's try publishing JSON messages instead. To make this easier
for extension, we will write some Python scripts to produce and consume messages. To start, you must install the library. You can do this by running , or if you are
more advanced, use a and install from instead. kafka-python pip install kafka-python virtualenv requirements.txt # producer.py
from kafka import KafkaProducer
from datetime import datetime
import json
producer = KafkaProducer(
    bootstrap_servers=['localhost:9093'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
producer.send('posts', {'author': 'choyiny', 'content': 'Kafka is cool!', 'created_at': datetime.now().isoformat()}) Below is an example of a Python consumer that subscribes to and prints out every value. post # consumer.py
from kafka import KafkaConsumer
import json
consumer = KafkaConsumer(
    'posts',
    bootstrap_servers=['localhost:9093'],
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)
# note that this for loop will block forever to wait for the next message
for message in consumer:
    print(message.value) Connecting to Kafka from another Docker container If you produce messages to a Kafka topic, you most likely have planned a destination for the messages. For example, the above
snippet shows how you can write your own python consumer. Sometimes though, you would want to transport your messages to another
database such as Clickhouse or Elasticsearch for further processing or visualizations. Although you can insert messages directly into Clickhouse, for a more scalable solution, you can make use of bulk insertions to
minimize transactions. (Again, this is a topic for another time... our goal really is to get a POC up and running.) Let's show an example integration of our Dockerized Kafka to Clickhouse, an OLAP database. (What
Clickhouse is will be an article for another time...) To learn more about Clickhouse's table engine, see . Using the Kafka table engine Create a file called and add the following content: docker-compose.yml # docker-compose.yml
version: "3.7"
services:
  clickhouse:
    restart: always
    image: clickhouse/clickhouse-server
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - "clickhouse-volume:/var/lib/clickhouse/"
volumes:
  clickhouse-volume: Now we are ready to connect to Clickhouse. Run the following command to start the Clickhouse container: $ docker-compose -f clickhouse.docker-compose.yml exec clickhouse clickhouse-client
ClickHouse client version 22.11.2.30 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 22.11.2 revision 54460.

Warnings:
 * Linux is not using a fast clock source. Performance can be degraded. Check /sys/devices/system/clocksource/clocksource0/current_clocksource

0cd6f3269407 :) This is an interactive shell that allows you to run SQL commands. To create a table, run the following command: -- create messages queue
CREATE TABLE default.message_queue
(
  created_at DateTime,
  content String,
  author String
)
ENGINE = Kafka(
  'kafka:9092',
  'posts',
  'clickhouse',
  'JSONEachRow'
) settings kafka_thread_per_consumer = 1, kafka_num_consumers = 1; Note how we are using as the connection string instead of when connecting to Kafka locally. This
is because we are connecting to it through the internal Docker network. kafka:9092 localhost:9093 Let's create a few more tables to visualize the data: -- create messages table
CREATE TABLE default.messages
(
  created_at DateTime,
  content String,
  author String
)
ENGINE = MergeTree
ORDER BY created_at;

-- create materialized view
CREATE MATERIALIZED VIEW default.messages_mv
TO default.messages
AS SELECT * FROM default.message_queue; I'll leave it as an exercise to the reader to come up with the SQL queries to select the data in the table. messages Linking it all together For our demo app, we will create an API endpoint in Python Flask, and instead of saving it to a database directly, we are going to produce it to the Kafka topic . Follow along below or by cloning . On top of that, since we have setup Clickhouse to consume messages from Kafka, we will be able to see the messages in the Clickhouse UI. POST /posts posts this demo repo Here's the straight forward code for the API endpoint: # app.py
# fun fact: This snippet was generated entirely by Copilot
from flask import Flask, request
from kafka import KafkaProducer
from datetime import datetime
import json

app = Flask(__name__)
producer = KafkaProducer(
    bootstrap_servers=['kafka:9093'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

@app.route('/posts', methods=['POST'])
def create_post():
    post = request.get_json()
    # clickhouse can only parse strings without milliseconds
    post['created_at'] = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
    producer.send('posts', post)
    return 'ok' To run this, you can use the following command: $ flask run After running, you can try out the endpoint by executing a curl command: $ curl -X POST -H "Content-Type: application/json" -d '{"author": "choyiny", "content": "Kafka is cool!"}' http://localhost:5000/posts It would be cool if it worked first try, however, these are commands you can use to pull out the logs from Clickhouse to inspect errors: $ docker-compose exec clickhouse tail -f /var/log/clickhouse-server/clickhouse-server.log To remove corrupted messages from a kafka topic, it is sufficient to just delete the topic entirely with this command: $ docker-compose exec kafka /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic posts Ideally, there are no errors. I'll leave it as an exercise to the reader to connect to Clickhouse again and inspect that data from Kafka is being consumed. And there you have it! An end to end solution with Kafka on Docker in a local environment. Again, all the code is available at . If you have any questions, feel free to drop a
comment below! this demo repo Also published . here

Setting up Kafka on Docker for Local Development

Read/write splitting is a technique to route reads and writes to multiple database servers, allowing you to perform query-based load balancing. Implementing this at the application level is hard because it couples code or configuration parameters to the underlying database topology. For example, you might have to define different for each server in the database cluster. connection pools MariaDB MaxScale is an advanced database proxy that can be used as a read/write splitter that routes statements to replica nodes and / / statements to primary nodes. This happens automatically without having to change your application code or even configuration—with MaxScale, the database looks like a single-node database to your application. SELECT INSERT UPDATE DELETE In this hands-on tutorial, you’ll learn how to configure MariaDB database replication with one primary and two replica nodes, as well as how to set up MaxScale to hide the complexity of the underlying . The best part: you’ll learn all this without leaving your web browser! topology The Play With Docker Website (PWD) is a website that allows you to create virtual machines with preinstalled and interact with them directly in your browser. Log in and start a new session. Play With Docker Docker You will use a total of 5 nodes: : Primary server node1 : Replica server A node2 : Replica server B node3 : MaxScale database proxy node4 : Test machine (equivalent to a web server, for example) node5 : Even though are a good fit for the most simple scenarios and for development environments, it might not be the best option for production environments. MariaDB Corporation does not currently offer support for Docker deployments in production environments. For production environments, it is recommended to use MariaDB Enterprise (on the cloud or on-premise) or MariaDB SkySQL (currently available on AWS and GCP). Note databases on Docker containers Running the Primary Server Add a new instance using the corresponding button: On , run a MariaDB primary server as follows: node1 docker run --name mariadb-primary \
  -d \
  --net=host \
  -e MARIADB_ROOT_PASSWORD=password \
  -e MARIADB_DATABASE=demo \
  -e MARIADB_USER=user \
  -e MARIADB_PASSWORD=password \
  -e MARIADB_REPLICATION_MODE=master \
  -e MARIADB_REPLICATION_USER=replication_user \
  -e MARIADB_REPLICATION_PASSWORD=password \
  bitnami/mariadb:latest This configures a container running MariaDB Community Server with a database user for replication ( ). Replicas will use this user to connect to the primary. replication_user Running the Replica Servers Create two new instances ( and ) and run the following command on both of them: node2 node3 docker run --name mariadb-replica \
  -d \
  --net=host \
  -e MARIADB_MASTER_ROOT_PASSWORD=password \
  -e MARIADB_REPLICATION_MODE=slave \
  -e MARIADB_REPLICATION_USER=replication_user \
  -e MARIADB_REPLICATION_PASSWORD=password \
  -e MARIADB_MASTER_HOST=<PRIMARY_IP_ADDRESS> \
  bitnami/mariadb:latest Replace with the IP address of . You can find the IP address in the instances list. <PRIMARY_IP_ADDRESS> node1 Now you have a cluster formed by one primary node and two replicas. All the writes you perform on the primary node ( ) are automatically replicated to all replica nodes ( and ). node1 node1 node2 Running MaxScale MaxScale is a that understands . This allows it to route write operations to the master node and read operations to the replicas in a load-balanced fashion. Your application can connect to MaxScale using a single endpoint as if it was a one-node database. database proxy SQL Create a new instance ( ) and run as follows: node4 MaxScale docker run --name maxscale \
  -d  \
  --publish 4000:4000 \
  mariadb/maxscale:latest You can configure MaxScale through config files, but in this tutorial, we’ll use the command line to make sure you understand each step. In less ephemeral environments you should use config files, especially in orchestrated deployments such as and . Docker Swarm Kubernetes Launch a new shell in : node4 docker exec -it maxscale bash You need to create objects in MaxScale. These are the MariaDB databases to which MaxScale routes reads and writes. Replace , , and with the IP addresses of the corresponding nodes ( , , and ) and execute the following: server <NODE_1_IP_ADDRESS> <NODE_2_IP_ADDRESS> <NODE_3_IP_ADDRESS> node1 node2 node3 maxctrl create server node1 <NODE_1_IP_ADDRESS>
maxctrl create server node2 <NODE_2_IP_ADDRESS>
maxctrl create server node3 <NODE_3_IP_ADDRESS> Next, you need to create a MaxScale to check the state of the cluster. Run the following command: monitor maxctrl create monitor mdb_monitor mariadbmon \
    --monitor-user root --monitor-password 'password' \
    --servers node1 node2 node3 Don’t use the user in production environments! It’s okay in this ephemeral lab environment, but in other cases and give it the appropriate grants. Note: root create a new database user for MaxScale Now that MaxScale is monitoring the servers and making this information available to other modules, you can create a MaxScale . In this case, the service uses a MaxScale router to make reads and writes go to the correct type of server in the cluster (primary or replica). Run the following to create a new service: service maxctrl create service query_router_service readwritesplit \
    user=root \
    password=password \
    --servers node1 node2 node3 Finally, you need to create a MaxScale . This kind of object defines a port that MaxScale uses to receive requests. You have to associate the listener with the router. Run the following to create a new listener: listener maxctrl create listener \
    query_router_service query_router_listener 4000 \
    --protocol=MariaDBClient Notice how the listener is configured to use port . This is the same port you published when you run the Docker container. 4000 Check that the servers are up and running: maxctrl list servers You should see something like the following: Testing the Setup To test the cluster, create a new instance ( ) and start an Ubuntu container: node5 docker run --name ubuntu -itd ubuntu This container is equivalent to, for example, a machine that hosts a web application that connects to the database. Run a new Bash session in the machine: docker exec -it ubuntu bash Update the package catalog: apt update Install the MariaDB SQL client so you can run SQL code: apt install mariadb-client -y Connect to the database, or more precisely, to the MaxScale database proxy: mariadb -h 192.168.0.15 --port 4000 -u user -p As you can see, it’s as if MaxScale was a single database. Create the following table: MariaDB SQL CREATE TABLE demo.message(content TEXT); We want to insert rows that contain the unique server ID of the MariaDB instance that actually performs the insert operation. Here’s how: MariaDB SQL INSERT INTO demo.message VALUES \
    (CONCAT("Write from server ", @@server_id)), \
    (CONCAT("Write from server ", @@server_id)), \
    (CONCAT("Write from server ", @@server_id)); Now let’s see which MariaDB server performed the write and read operations: MariaDB SQL SELECT *, CONCAT("Read from server ", @@server_id) FROM demo.message; Run the previous query several times. You should get a result like this: In my cluster, all the writes were performed by server ID which is the primary node. Reads were executed by server IDs and which are the replica nodes. You can confirm the ID values by running the following on the primary and replica nodes: 367 908 308 docker exec -it mariadb-primary mariadb -u root -p \
    --execute="SELECT @@server_id"
 
docker exec -it mariadb-replica mariadb -u root -p \
    --execute="SELECT @@server_id" What’s Next? We focused on basic read/write splitting in this tutorial, but MaxScale can do much more than this. For example, enforce to your backend database topology, perform , perform load balancing, import and export data from and into , and even convert NoSQL/MongoDB API commands to SQL. MaxScale also includes a and for operations. Check the to learn more about MaxScale. security automated failover connection-based Kafka REST API web-based GUI documentation Also Published Here

How to do Database Read/Write Splitting With Your Browser

Creating a full-fledged API requires resources, both time and money. You need to think about the model, the design, the REST principles, etc., without writing a single line of code. Most of the time, you don't know whether it's worth it: you'd like to offer a Minimum Viable Product and iterate from there. I want to show how you can achieve it without writing a single line of code. The solution The main requirement of the solution is to use the . It's a well-established Open Source SQL database. PostgreSQL database Instead of writing our REST API, we use the PostgREST component: PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations. -- PostgREST Let's apply it to a simple use case. Here's a table that I want to expose via a CRUD API: product Note that you can find the whole source code on to follow along. GitHub PostgREST's is pretty complete and works out of the box. Yet, I didn't find any ready-made Docker image, so I created my own: Getting Started guide FROM debian:bookworm-slim                                                   #1

ARG POSTGREST_VERSION=v10.1.1                                               #2
ARG POSTGREST_FILE=postgrest-$POSTGREST_VERSION-linux-static-x64.tar.xz     #2

RUN mkdir postgrest

WORKDIR postgrest

ADD https://github.com/PostgREST/postgrest/releases/download/$POSTGREST_VERSION/$POSTGREST_FILE \
    .                                                                       #3

RUN apt-get update && \
    apt-get install -y libpq-dev xz-utils && \
    tar xvf $POSTGREST_FILE && \
    rm $POSTGREST_FILE                                                      #4 Start from the latest Debian Parameterize the build Get the archive Install dependencies and unarchive The Docker image contains a executable in the folder. We can "deploy" the architecture via Docker Compose: postgrest /postgrest version: "3"
services:
  postgrest:
    build: ./postgrest                                   #1
    volumes:
      - ./postgrest/product.conf:/etc/product.conf:ro    #2
    ports:
      - "3000:3000"
    entrypoint: ["/postgrest/postgrest"]                 #3
    command: ["/etc/product.conf"]                       #4
    depends_on:
      - postgres
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: "root"
    volumes:
      - ./postgres:/docker-entrypoint-initdb.d:ro       #5 Build the above Dockerfile Share the configuration file Run the executable postgrest With the configuration file Initialize the schema, the permissions, and the data At this point, we can query the table: product curl localhost:3000/product We immediately get the results: [{"id":1,"name":"Stickers pack","description":"A pack of rad stickers to display on your laptop or wherever you feel like. Show your love for Apache APISIX","price":0.49,"hero":false}, 
 {"id":2,"name":"Lapel pin","description":"With this \"Powered by Apache APISIX\" lapel pin, support your favorite API Gateway and let everybody know about it.","price":1.49,"hero":false}, 
 {"id":3,"name":"Tee-Shirt","description":"The classic geek product! At a conference, at home, at work, this tee-shirt will be your best friend.","price":9.99,"hero":true}] That was a quick win! Improving the solution Though the solution works, it has a lot of room for improvement. For example, the database user cannot change the data, but everybody can actually access it. It might not be a big issue for product-related data, but what about medical data? The PostgREST documentation is aware of it and explicitly advises using a reverse proxy: PostgREST is a fast way to construct a RESTful API. Its default behavior is great for scaffolding in development. When it’s time to go to production it works great too, as long as you take precautions. PostgREST is a small sharp tool that focuses on performing the API-to-database mapping. We rely on a reverse proxy like Nginx for additional safeguards. -- Hardening PostgREST Instead of nginx, we would benefit from a full-fledged API Gateway: enters . We shall add it to our Docker Compose: Apache APISIX version: "3"
services:
  apisix:
    image: apache/apisix:2.15.0-alpine                              #1
    volumes:
      - ./apisix/config.yml:/usr/local/apisix/conf/config.yaml:ro
    ports:
      - "9080:9080"
    restart: always
    depends_on:
      - etcd
      - postgrest
  etcd:
    image: bitnami/etcd:3.5.2                                       #2
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2397"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2397" Use Apache APISIX APISIX stores its configuration in etcd We shall first configure Apache APISIX to proxy calls to : postgrest curl http://apisix:9080/apisix/admin/upstreams/1 -H 'X-API-KEY: 123xyz' -X PUT -d ' #1-2
{
  "type": "roundrobin",
  "nodes": {
    "postgrest:3000": 1                                                             #1-3
  }
}'

curl http://apisix:9080/apisix/admin/routes/1 -H 'X-API-KEY: 123xyz' -X PUT -d '    #4
{
  "uri": "/*",
  "upstream_id": 1
}' Should be run in one of the Docker nodes, so use the Docker image's name. Alternatively, use but be sure to expose the ports localhost Create a reusable upstream Point to the PostgREST node Create a to the created route upstream We can now query the endpoint via APISIX: curl localhost:9080/product It returns the same result as above. DDoS protection We haven't added anything, but we're ready to start the work. Let's first protect our API from Distributed Denial of Service attacks. Apache APISIX is designed around a plugin architecture. To protect from DDoS, we shall use a plugin. We can set plugins on a specific when it's created or on every ; in the latter case, it's a . We want to protect every route by default, so we shall use one. route route global rule curl http://apisix:9080/apisix/admin/global_rules/1 -H 'X-API-KEY: 123xyz' -X PUT -d '
{
  "plugins": {
    "limit-count": {                 #1
      "count": 1,                    #2
      "time_window": 5,              #2
      "rejected_code": 429           #3
    }
  }
}' limits the number of calls in a time window limit-count Limit to 1 call per 5 seconds; it's for demo purposes Return ; the default is 429 Too Many Requests 503 Now, if we execute too many requests, Apache APISIX protects the upstream: curl localhost:9080/product <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html> Per-route authorization PostgREST also offers an Open API endpoint at the root. We thus have two routes: for the Open API spec and for the products. Suppose we want to disallow unauthorized people to access our data: Regular users can access products, while admin users can access both the Open API spec products. / /product and APISIX offers several . We will use the simplest one possible, . It relies on the abstraction. requires a specific header: the plugin does a reverse lookup on the value and finds the consumer whose key corresponds. authentication methods key-auth Consumer key-auth Here's how to create a consumer: curl http://apisix:9080/apisix/admin/consumers -H 'X-API-KEY: 123xyz' -X PUT -d '    #1
{
  "username": "admin",                                                               #2
  "plugins": {
    "key-auth": {
      "key": "admin"                                                                 #3
    }
  }
}' Create a new consumer Consumer's name Consumer's key value We do the same with consumer and key . Now, we can create a dedicated route and configure it so that only requests from pass through: user user admin curl http://apisix:9080/apisix/admin/routes -H 'X-API-KEY: 123xyz' -X POST -d ' #1
{
  "uri": "/",
  "upstream_id": 1,
  "plugins": {
    "key-auth": {},                                                             #2
    "consumer-restriction": {                                                   #2
      "whitelist": [ "admin" ]                                                  #3
    }
  }
}' Create a new route Use the and plugins key-auth consumer-restriction Only -authenticated requests can call the route admin Let's try the following: curl localhost:9080 It doesn't work as we are not authenticated via an API key header. {"message":"Missing API key found in request"} curl -H "apikey: user" localhost:9080 It doesn't work as we are authenticated as , but the route is not authorized for but for . user user admin {"message":"The consumer_name is forbidden."} curl -H "apikey: admin" localhost:9080 This time, it returns the Open API spec as expected. Monitoring A much-undervalued feature of any software system is monitoring. As soon as you deploy any component in production, you must monitor its health. Nowadays, many services are available to monitor. We will use as it's Open Source, battle-proven, and widespread. To display the data, we will rely on for the same reasons. Let's add the components to the Docker Compose file: Prometheus Grafana version: "3"
services:
  prometheus:
    image: prom/prometheus:v2.40.1                                    #1
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml    #2
    depends_on:
      - apisix
  grafana:
    image: grafana/grafana:8.5.15                                     #3
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning              #4
      - ./grafana/dashboards:/var/lib/grafana/dashboards              #4
      - ./grafana/config/grafana.ini:/etc/grafana/grafana.ini         #4-5
    ports:
      - "3001:3001"
    depends_on:
      - prometheus Prometheus image Prometheus configuration to scrape Apache APISIX. See the full file here Grafana image Grafana configuration. Most of it comes from the configuration by APISIX. provided Change the default port from to to avoid conflict with the PostgREST service 3000 3001 Once the monitoring infrastructure is in place, we only need to instruct APISIX to provide the data in a format that Prometheus expects. We can achieve it through configuration and a new global rule: plugin_attr:
  prometheus:
    export_addr:
      ip: "0.0.0.0"             #1
      port: 9091                #2 Bind to any address Bind to port . Prometheus metrics are available on on the Docker network 9091 http://apisix:9091/apisix/prometheus/metrics We can create the global rule: curl http://apisix:9080/apisix/admin/global_rules/2 -H 'X-API-KEY: 123xyz' -X PUT -d '
{
  "plugins": {
    "prometheus": {}
  }
}' Send a couple of queries and open the Grafana dashboard. It should look similar to this: Conclusion Creating a full-fledged REST(ful) API is a huge investment. One can quickly test a simple API by exposing one's database in a CRUD API via PostgREST. However, such an architecture is not fit for production usage. To fix it, you need to set a façade in front of PostgREST, a reverse proxy, or even better, an API Gateway. Apache APISIX offers a wide range of features, from authorization to monitoring. With it, you can quickly validate your API requirements at a low cost. The icing on the cake: when you've validated the requirements, you can keep the existing façade and replace PostgREST with your custom-developed API. The source code is available on . GitHub To go further: PostgREST Getting started with Apache APISIX Apache APISIX plugins Originally published at on November 20th, 2022 A Java Geek

Creating a Poor Man's API Using PostgREST 

DevOps has generated so many opportunities for software organizations to improve their operations. Software development best practices have given rise to many platforms and tools that make it easier to deploy software quickly. Kubernetes and Helm can do so much together to help developers deploy software with less effort. Kubernetes is the most popular open-source platform for running container-based apps in the enterprise. In fact, many organizations aim to run all their production deployments on Kubernetes. These are exciting times for those in the Kubernetes community because we’re seeing more and more users leverage its extended capabilities for even deeper operational insight and control. One such capability is the use of Helm to package and deploy individual Kubernetes applications. What is Helm? Image credit: IBM Developer Helm is a popular open-source tool for managing Kubernetes clusters. It simplifies the process of installing, updating, and removing applications on your cluster by packaging them as “charts.” These charts are then installed on your cluster with one command. Helm relies on the concept of charts. A chart is an archive of all the resource definitions that are needed to deploy an application to Kubernetes, including the configuration files and scripts needed for deployment. To get started with Helm, you need to create a chart for your application or find one online that suits your needs. Below, we will go through a brief tutorial on how to create a simple Helm chart. You can quickly install software onto your Kubernetes cluster, and the public Helm charts are readily available in ArtifactHub. Prerequisites for this tutorial: Install Helm from the official Helm website Install or (with Kubernetes enabled) Rancher Desktop Docker Desktop Sign up for the to deploy to any Kubernetes environment Harness Software Delivery Platform Tutorial: Create a Simple Helm Chart Use your terminal to create a new directory to store our code: mkdir helm-exampl   ecd helm-example Create a new chart and give it an appropriate name: This creates a simple chart that will deploy an Nginx pod. helm create new-chart You should see the following output on your terminal: Creating new-chart Change the directory into the folder of the new chart that we just created: cd new-chart With command, you can see the contents of the new chart folder: ls Chart.yaml   charts   templates   values.yaml Keeping everything as it is, we will deploy our first chart: helm install new-chart ./helm-example/new-chart Note: Use this command outside of your directory to set the path correctly as shown. ‍With the command , you can see your deployed helm chart: helm ls ‍ You can also confirm that the pods are up and running with the command : kubectl get pods In this example, we are using Rancher Desktop so it is easy to access the deployed chart. You can download from the official website. Rancher Desktop And when I check my localhost:56253, I can see the Nginx container running. As stated previously, if you are using Rancher Desktop, it’s very easy to access your newly deployed app. ‍ Next, you can package this chart to share it with anyone with the command: helm package ./new-chart Deploying with Harness CD It is very easy to use Helm charts and do deployments with Harness CD. Further in this tutorial, you’ll see how to deploy a publicly available Docker image to your Kubernetes cluster using Helm charts and a rolling deployment strategy through Harness CD. Assuming you have a Kubernetes cluster set up with 3 nodes, 4vCPU machine type, 16GB memory, and 100GB disk. Head to and sign up if this is your first time using the Harness platform. Once you log in, switch to the NextGen mode, which is the newest, most advanced version. Harness Create a project with any name you want, select continuous delivery and start creating the pipeline to deploy a MongoDB chart. Every stage is self-explanatory:  Add the required fields. Then specify the service, select deployment type as Kubernetes, add the manifest details as Helm chart, and then create a new HTTP Helm repo connector. Note that I have already created the HTTP Helm Repo Connector in this example. You have to create a new one if you are new to this process. Once you click on + HTTP Helm Repo Connector, it will ask for a few details and you need to provide them. Provide Helm Repository URL as as we are using the official chart. https://charts.bitnami.com/bitnami In the infrastructure setup, give cluster details: In the execution step, select the Rolling deployment strategy: Once everything is set up, save the pipeline and run it. ‍ You should see a successful deployment of the MongoDB image. Confirm the deployment by going to your cluster setup. I have my cluster created on GCP and I can confirm that the MongoDB image has been successfully deployed through the Helm chart. You have successfully learned how to create a Helm chart and deploy it with the help of Congratulations! Harness. Also published here.

Creating and Deploying a Simple Helm Chart

The programming language and the main framework are often crucial for startups. The speed and quality of resolving tasks depend on them. At the same time, tools have to be modern to save time and be flexible for the quick rebuilding of the solution. Python with Django is a perfect example of a typical solution to such a problem. They provide an ideal opportunity for the development team to create a product with any level of stability. From quick-and-dirty prototypes to stable, scalable applications. The wide range of benefits makes Django a powerful solution as the main technology for startups. And we at Daiquiri Team are versed in this. Popularity According to the , Python is the most popular language in April 2022. At the same time, Django has the largest number of stars on Github of all Python frameworks — . The closest competitors are Flask with and the rising star FastAPI with . TIOBE index 63.6K 58.7K 44.3K Why is this important? The more popular the technology is, the more developers have created their projects using it. The more such projects were created, the more problems the developers faced and successfully solved. Developers' loyalty to the technology shows they are confident that there are no significant problems in solving business problems using this technology. Recruiting the developer It is vital for startups to be capable of hiring developers quickly. Django's popularity plays into the hands of product owners. The developer on Django is really relatively easy to find. Most Python developers with commercial experience have at least encountered Django. Okay, among experienced developers Django is popular, what about newbies? The situation is definitely not worse. Django has a lot of educational materials (for example, here ), which reduces the entry threshold. courses on Udemy Django is also a cost-effective framework to work with. According to , the Django developer's annual salary is $45.3K, while the FastAPI developer's salary is $54K, for Flask — $54.8K. StackOverflow What makes Django so appealing to developers? The answer lies in the following reason. Simplicity and speed of development Django positions itself as a . So, it’s a framework where the biggest part of technical problems has ready-to-go and customizable solutions. It’s especially important for startups to have the ability to spend the developer’s time just on considerable things. Battery-included framework The developers can focus on resolving business tasks, not on developing things that were already built. In distinction to the above-mentioned Flask and FastAPI web frameworks, Django has built-in: , ORM and mechanisms of migration for a lot of DBMS , System of users and access rights , Administrative panel , , etc. Cache framework internationalization and localization modules A critical indicator for early-stage startups is . That’s why the speed of hypothesis testing has to be high. Django has a ready-made project structure that fits well for a quick start. time-to-market Also, Django has a lot of integration with 3rd-party, many of which are official ones. That's why Django developers spend less time working on solutions due to ready-made ones. These solutions are tested well, and their readiness to use in commercial projects is proven in practice. You can also of developing an IoT system using Django. In that story, we had built a prototype in 3 weeks from scratch. check out our case Quality of documentation Django’s documentation is cool; everyone knows this. You can check it on , , and . Reddit Quora YCombinator Moreover, Django is open-source. The source code of Django has a large number of comments, which are elaborated extensively and in detail. That’s why every Django developer knows that if there are any doubts about the implementation of some functionality, you can just read the source code. If it’s not enough for you, here’s one more argument — the size of the community knowledge base. There are questions on StackOverflow tagged . In comparison: Flask has , and FastAPI — issues. The vast knowledge base from the community simplifies the resolving of almost every development task. You can hardly worry that you will encounter a unique problem that the developers will have to resolve for a very long time. 288.7K django 49K 2.8K Performance For startups, the speed of hypothesis testing and time-to-market is more important than readiness for high load. Nevertheless, the ability to scale in the sense of performance is quite important for startups; otherwise, moving to a qualitatively different level can be very expensive. Django authors didn't have a particular idea to make it very fast, but its performance is enough for most startups' products. According to s, Django is even quicker than Flask. one of the most trustworthy benchmark With the proper setup, Django can withstand enormous loads; this has been repeatedly proven by the experience of both startups and industry leaders. Deployment Due to the maturity of Django, there are a large number of ready-made solutions for PaaS services for Django hosting. For example, , , , and . Heroku GCP Bitnami Clever Cloud Also, Django can be easily set up for work on any VPS, for example, on . Digital Ocean Convenient and easy-to-set-up deployment is the key to a productive workflow for developers and solid security at the production stage. With Django, you will have a minimum of problems on this matter. The positive experience of big companies As we already said, Django is used by , such as Instagram, Dropbox, YouTube, Spotify, The Washington Post, etc. Also, it is used by many middle and small startups to build their products. That’s why around Django, there are many solutions that especially startups need. big companies Django is a powerful framework both for the early and mature. It is perfect for experiments and for a stable product. Django has a huge number of ready-made solutions that are easy to customize for most real-life tasks. We at have developed many products on Django. If you are interested in assessing whether this is a good choice for your product technologically — write to us, and we will be glad to discuss it. Daiquiri Team Also published . here

Stories, News & Mentions about BITNAMI

Bitnami’s Stories on HackerNoon

Bitnami's News & Mentions