Wind River Systems surveyed more than 200 auto industry companies to learn where should the auto industry focus its software-centric efforts.

Wind River Webinar: What Focus for Software-Centric Business Models?

Introduction Software development stands as the backbone of innovation. In today’s fast-paced world, where digital platforms, mobile applications, artificial intelligence, and the internet of things are advancing at an unprecedented pace,

Exploring the Future of Software Development: Pursuing a BS in Computer Software Technology

We've tested more than 100 anti-malware apps to help you find the top malware protection and removal software for all your devices. When the IBM PC was new, I served as the president of the San ...

The Best Malware Removal and Protection Software for 2023

Aimed squarely at modern DevOps and SRE teams, Landscape Explorer is designed to guide users through the maze of their system landscapes, making the planning and execution of Chaos Engineering ... Director of Software Development at Salesforce.com.

Steadybit Unveils the "Landscape Explorer": Revolutionizing Chaos Engineering with Enhanced Visibility

Trading options is the process of buying and selling stocks at a pre-negotiated price by a certain date. Options are a type of contract that gives the trader the right to buy or sell a particular ...

Best Options Trading Software

there's an almost mythical concept called the “10X developer,” meaning a software engineer who's such a high performer that she or he is perfectly capable of doing the work of 10 average-abled ...

Beyond The 10X Software Engineer: Focusing On The Bigger Picture

Despite economic fears, Gartner predicts spending on software will increase 11% in 2023, up to $879 billion from $790 billion in 2022. Why? The answer is simple. No business can survive in today ...

Four Crucial Steps For Successful Software Implementation

Recently software engineering has been one of the most trending fields. Chat GPT and the new technologies in many tools we use daily in different fields like graphic design and video editing make ...

Software Engineering 101

Who is the Master's in Software Engineering for? Drexel’s College of Computing & Informatics' Master of Science in Software Engineering (MSSE) is designed for students who hold a four-year bachelor's degree or master’s degree from a regionally ...

Master's (MS) in Software Engineering

April 2023 Siemens and Microsoft are harnessing the collaborative power of generative artificial intelligence (AI) to help industrial companies drive

Siemens and Microsoft drive industrial productivity with generative … – Microsoft

Bitnami

bitnami

Axion Processors: Google's First Arm-based CPUs

How to Deploy MinIO and Trino with Kubernetes

কুবারনেটসের সাথে MinIO এবং Trino কীভাবে স্থাপন করবেন

So stellen Sie MinIO und Trino mit Kubernetes bereit

Cómo implementar MinIO y Trino con Kubernetes

Comment déployer MinIO et Trino avec Kubernetes

कुबेरनेट्स के साथ मिनिओ और ट्रिनो को कैसे तैनात करें

Kubernetes を使用して MinIO と Trino をデプロイする方法

Kubernetes를 사용하여 MinIO 및 Trino를 배포하는 방법

Como implantar MinIO e Trino com Kubernetes

Как развернуть MinIO и Trino с помощью Kubernetes

MinIO ve Trino'nun Kubernetes ile Dağıtılması

Cách triển khai MinIO và Trino với Kubernetes

如何使用 Kubernetes 部署 MinIO 和 Trino

When deploying multiple resources to Kubernetes, it is often challenging to keep track of all the resources, image tags, and environment variables distributed across multiple manifest files. To address this issue, manifest templating tools such as Kustomize and Helm exist, simplifying large, complex manifest files to templates with customization available through values.yaml. This provides flexibility with a single template across multiple environments (e.g., QA, staging, production), with changes configured via the values.yaml file. Additionally, it offers ease-of-use for end-users to deploy into any new environment with just a single command. However, solving one set of problems often introduces new challenges. Creating individual Helm charts can become difficult to manage as applications increase. It becomes essential to track changes across all Helm charts used by an organization, consuming significant engineering hours. Using community charts poses the challenge of pulling and customizing new charts, maintaining separate copies in the Helm repository, and consuming considerable time to pull and run these charts. Just as we've created a Helm chart for each application, bundling all required resources into a single chart, it would be advantageous to bundle all these micro-services into a single chart. Kubernetes Introducing Helm Sub-charts: Helm sub-charts are like mini-charts residing inside a larger Helm chart, akin to different players in your favorite video game. It's Helm's way of implementing inheritance. Charts can have dependencies, called , each with its values and templates. subcharts Why Do We Need Sub-charts? Sub-charts help break down large applications into more manageable parts, akin to splitting your favorite game into different levels. Breaking Down Big Stuff: Sub-charts can be reused in different projects, similar to using a favorite weapon or skill in various game levels. Reuse the Good Stuff: Sub-charts enable working on different parts of an app without accidentally affecting other parts, similar to cooking different dishes in separate pots. Keeping Things Separate: Things to Consider When Creating Helm Sub-charts: Before creating sub-charts, consider the following: A sub-chart is considered "stand-alone," meaning it can never explicitly depend on its parent chart. Consequently, a sub-chart cannot access the values of its parent, but a parent chart can override values for sub-charts. Helm sub-charts work well when the charts you want to deploy are used in combination with each other but are not directly dependent on each other's resources. Helm sub-charts work well when the charts you want to deploy are used in combination with each other but are not directly dependent on each other's resources. When a Helm chart has multiple Helm sub-charts, specifying the specific order can be challenging. Let's Build Some Helm Sub-charts: To create a master Helm with sub-charts, consider a master Helm chart example deploying Nginx and PostgreSQL. The folder structure is as follows: To create Helm sub-charts within the master Helm chart, add the following to the values.yaml file, describing the name of the Helm charts along with their versions and repositories. apiVersion: v2
name: my-master-chart
description: A Helm chart to deploy Nginx and PostgreSQL
version: 0.1.0
dependencies:
  - name: nginx
    version: 15.4.3
    repository: https://charts.bitnami.com/bitnami
  - name: postgresql
    version: 13.2.16
    repository: https://charts.bitnami.com/bitnami After adding the above to the values, run the following command to pull and add these dependencies to the charts folder. helm dependency update Once pulled, the folder structure should have .tgz files in the charts folder. To untar the .tgz file, use the command: tar -zxvf nginx-15.4.3.tgz After extraction, your file structure would resemble the one below: Just as we can manipulate inputs from the values file in individual Helm charts, we can use the global values.yaml file to manipulate the values of each of these Helm charts. By doing so, we can use the base Helm charts as templates and make all modifications from the global values.yaml file. This reduces complexity and simplifies the management of different input values in various Helm charts. Consider the following example to illustrate the use case. The values for Nginx mention the image tag, and PostgreSQL mentions the image tag along with the port number at which the PostgreSQL deployment runs. Here are the default values passed in the Nginx and PostgreSQL values.yaml files: Now, change the values in the master values.yaml file at the top level. Change the Nginx image from to and the PostgreSQL image from to . Also, change the PostgreSQL default port from to . 1.25.3-debian-11-r1 1.24.0-debian-11-r153 16.1.0-debian-11-r9 15.5.0-debian-11-r9 5432 5111 The values for both Nginx and images have changed, and the PostgreSQL port has been updated. PostgreSQL Liked my content? Feel free to reach out to my for interesting content and productive discussions. LinkedIn

Helm Sub-Charts Demystified: A Guide to Efficient Deployments

Your Definitive GitOps Guide For DevOps Experts And Newbies

Hi Hackernoon! Tech experts warmly welcomed my . Even a top-level expert from Weaveworks contacted me after reading it. And I was greatly surprised. After some warm words, we agreed that they would wait for the second part of my article. Well, the time has come. previous article on GitOps Let’s dive in! Content Overview The Multiple Environments Problem Values From Secrets Issue CI Ops vs GitOps So, Who Really Needs GitOps? Conclusion The Multiple Environments Problem We've already dealt with Flux workflow in the previous part. Now let's move on. Here is our Helm release. Suppose we rolled out changes to the development environment, tested everything, and wanted to roll out to the stage. How can we do that? We have GitOps, so obviously, we want to maximize the Git approach (merge code from dev to stage). But in this Helm release configuration, we can't do that because (for example) we have a different base in the stage environment (lines 20...23). What does GitOps say about managing environments? , GitOps says "Why don't you just have one environment, and you won't have to do anything stupid". I, for one, am not sure how many teams have only one environment. GitOps ideologues don't seem to be sure about that either. So they say, "Well, if you really need multistage, do it somewhere outside of the GitOps scope. Within your system, for example." CI/CD Values From So we think, review the options, and find a few. The first one is (lines 15, 16, 17). valuesFiles We can describe separate values files for individual environments and plug them into HelmRelease. But this scheme doesn't fit well to git merge either. Next, we try valuesFrom. Our environment namespace may contain one or more config maps and/or secrets (lines 22...25). From them, Helm-controller can take Values and customize our Helm chart. And I like that. I even make a separate Helm chart where I configure all microservices, link it to the application charts with the dependsOn key (line 16), and proudly call it a single point of platform configuration. And then, the following happens: I have several Kubernetes clusters. My colleagues come and say: "Let's write another config map, and there will be cluster-specific Values (lines 26...28). For example, cluster name, Ingress suffix, region, and access rights. And we will create this config map with , together with the cluster. Terraform When this happens, I need to take a weightier argument and consistently explain the depth of their downfall to my colleagues. Because in this particular case, our application configuration will depend on 4 repositories (application chart repository, platform setup repository, flux infrastructure repository, terraform file repository) and two deployment systems (Flux and terraform). This is where I can give you some advice. If you want to work with , you should have only one value in your variables and config map. It must be the name of the environment you are deploying to. Everything else should be done inside the Helm chart using templating mechanisms. valuesFrom Secrets Issue Next point. We have a Helm release with a line that should not be there - the database password (line 23). What does GitOps tell us about managing secrets? , It says Never store your passwords in clear text in a Git repository. What can we say? Thanks, Cap! The next thing GitOps offers us is: Let's generate your passwords randomly when you create your environment. And you will make all objects that require password protection based on those passwords. That way, the password will never leave the environment, everything will be secured, and everyone will be happy. Nice, sure. But how many people, for example, have their SQL database in Kubernetes? Or is a cross-plane installed there? As a rule, we don't create heavy, persistent services inside or from inside a Kubernetes cluster. : Read on for the following alternative approaches Let's use tools like Mozilla SOPS or Bitnami Sealed Secrets when we do public and private keys. The private key is written to the cluster. With the public key, we encrypt our passwords and commit to a Git repository. At the time of deployment, some tool inside Kubernetes decrypts this and gives it to applications. And a specially trained Ops team member will write this key to the cluster. Let him carry a printout of it in his jacket pocket and then carefully enter it by hand. OK, that seems like the best option. Now let's think about it. In , old but still relevant, there is, for example, the 31st question: "Can a user's account be disabled on all systems in 1 hour?". Your Ops guy quit. And if your Kubernetes clusters are not under a single sign-on system, then you must run across all clusters and delete his account from everywhere. The Limoncelli Test: 32 Questions for Your Sysadmin Team The next question is, "Can you change all privileged (root) passwords in 1 hour?". That secret key that your Ops guy compromised by getting fired is that root password. And to change it, you have to do certain things: Extract the private key from the cluster Decrypt what we have in the Git repository with it Make new keys Write the private key to the cluster Encrypt the passwords Commit them to git Run it through the release cycle and deprecate it And you have to do it in a particular order because we have an asynchronous pool model. Any careless rollout can turn your environment into a pumpkin. And rollback won't help here. A couple of words for fans of external secret stores like Hashicorp Vault. I haven't found a consensus on them, but from a common-sense point of view, using external secret stores makes Git an even more non-unique source of truth and thus goes against the GitOps approach. CI Ops vs GitOps As part of the GitOps concept's PR, WaveWorks has often referred to classic CI Ops as an outdated approach and anti-pattern. Why? Because with GitOps, we can separate the stage of the deployment, protect it, and manage it more clearly. Within CI Ops, the deployment stage becomes a part of the monolithic pipeline. And because of this, several disadvantages arise. We will now analyze this fact. But! To get a fair comparison, we must choose a "fighter" from the CI-systems side. Let's take the popular and familiar to many people Gitlab, which implements the concept of pipelines as code. Your CI code is in the same repository as your application and is actually part of it. This allows for greater CI/CD transparency, self-documenting pipelines, and knowledge transfer within the team... CI Ops vs GitOps: Security The first problem is poor security. It's like we have a CI system within CI Ops, and developers have full access to Kubernetes through it, and it's non-secure. And within GitOps, we have an asynchronous executor pool that we don't have direct access to, greatly increasing the system's security. Let's put Flux in and see what the actual security is. By default, Flux gives cluster administrator rights to all of its controllers. Obviously, it's not very secure. Let's try to improve this situation. I have found an for Flux multiuser use. We create a tenant and see that namespace and ServiceAccount are generated. article about multi-tenancy Flux controllers get administrator rights to this namespace, not all of them, just Kustomize controllers. Earlier, using to limit the Flux Helm controller was recommended. Now, they offer some strange mechanism with patching of controller's deployments so that it starts to pick up the account service from the namespace where it is deployed. Kyverno But let's just think about it, abstracting from the toolkit. We have two schemes: the synchronous push and asynchronous pull models. Which one is safer, assuming they both have admin rights? That's a rhetorical question because admin rights are admin rights. But even if we restrict infrastructure rights, these schemes have roughly the same access to infrastructure, passwords, and persistent data. They can harm. If you look at the problem a bit from the outside, you can see that in both cases, you have a Git repository. And it contains both application code and infrastructure code. And probably the only way to stop all the abuse here is to control what gets into your Git. Gitlab has a mechanism for (though only in the paid version for now, but as we know, Gitlab's paid features are quickly moving to the community version). Merge request approvals This mechanism prevents you from merging your changes into the working branch of your repository until the right number of peers have reviewed your changes and confirmed that they are OK. This rather simple mechanism increases security many times over—much more than the difference between pull and push models. CI Ops vs GitOps: Rollback Procedure Another disadvantage that GitOps ideologues cite is the difficulty of rolling back CI. Their guides say that the CI system is not meant to be the only source of truth. This means that when working with the CI system, we cannot determine what we have clustered and what we will roll back to. But with GitOps, we have an infrastructure repository synchronizing with the infrastructure, so we do a git revert. And everything is fine. Let's think. GitOps only covers the deployment stage. But since git cannot be the only source of truth, we should think about rollback at the stage of building our artifacts. Besides, Flux has everything in place to make rollbacks difficult or impossible. For example, we can awfully tag a docker image (line 19). Also, we can use the Helm chart version range (line 11). And if we use Helm chart from the application repository, we may need to roll back the application repository to roll back our environment. In addition, don't forget about possible dependencies in a separate repository, which we may also need to roll back. valuesFrom Flux also has recommendations for organizing infrastructure repositories. For example, Monorepo. It is when we have the infrastructure for all our environments in one branch in different folders. Or, for example, Repo per team. It is when each team has its own folder in a shared infrastructure repository. This is where you must be very careful when rolling back so you don't delete extra. As for CI Ops, Gitlab may be the only source of truth. Gitlab has a so-called environment mechanism, where we specify the name of an environment in a Pipeline Deploy step and see its state in the environment tracking. It immediately becomes transparent which pipelines, from which branch, and from which commits are got. We also have a visual display of the pipelines, where we can see what rolled in there and whether those stages passed. We can roll back to the right stage. It is decisively impossible when looking at the commit history in an infrastructure repository in the GitOps concept. Alternatively, we can , thus making the pipeline atomic. And this would really be the only source of truth. save the docker image as an artifact in the pipeline As a result, to rollback within the implementation described above, you would just need to restart the Rollout Job in the Pipeline History in the Gitlab visual interface. CI Ops vs GitOps: The Problem of Multiple Clusters According to the GitOps concept developers, it is difficult to propagate changes to multiple clusters in CI Ops. Allegedly, in a CI system, we need to script, script, and script. But in the beautiful GitOps, we just put Flux in a new cluster and give it an infrastructure repository. One moment and our application is already in the new cluster. Let's start analyzing. Here is our custom resource Helm-release in the infrastructure repository. I have 2 questions when looking at this Helm release: Where did this application come from? Where is its repository? Where are the build logs and test run results? From which branch and from which commit is it deployed? Where is the application deployed to? How many fluxes in which clusters look at this infrastructure repository and pull configuration from there? "Out of GitOps scope" can be the answer to the first question. We just need to write a proper HelmRelease push script that will write all the information we need in the commit message. For the second question, Flux offers a with the creation of an infrastructure repository (flux-fleet) to manage Flux specifically. Before installing Flux in a cluster, we first commit all YAMLs describing the installation of flux controllers and related custom resources. Then we apply them to the target cluster. When we need to connect a git repository and Kustomize to a new infrastructure repository, we commit to that infrastructure repository again. bootstrap mechanism So we have the entire list of our clusters and all the accesses in one repository. And it works fine until some manual operations happen. For example, your Ops expert quits. Let's say there's a Kubernetes cluster somewhere where he put Flux and gave him manual bypass flux-fleet access to your infrastructure repositories. You may never know if you didn't do the access revocation and key regeneration. So I'd recommend writing down step zero on your checklist as creating a flux-fleet repository and bootstrap Flux. Otherwise, you will have secret knowledge that may one day be lost with the employee. What do we have in CI-ops? In Gitlab, we can declare reusable pieces of code (lines 1...9). We can inherit from these pieces of code (lines 13, 19, 25) and declare some additional variables (lines 15, 21, 27) when we make a new environment. In this case, it is the name of the cluster relative to which we will change the functioning of our scripts. In this way, we will deploy to a new environment and get its automatic tracking. So, Who Really Needs GitOps? There are a few cases where I think a GitOps approach can be justified: You don't have a CI/CD system, or it can't provide normal deployment for some reason. Improved security. GitOps can really add to the mix of measures to improve the security of your infrastructure. The lion's share of security issues are "out of GitOps scope" and should be addressed before or in conjunction with GitOps implementation. You have a large, multi-component platform that is tested and deployed. In this sense, a GitOps infrastructure repository is a good way to make a "stable" version of the platform and operate it as a single entity. You have processes where the pull model has advantages over the push model (e.g., canary deployments). So what is GitOps? For those who like precise formulas, GitOps = Continuous Delivery - Continuous Deployment. So it's not a CD concept. It's just (one more) way to organize automatic deployment. By and large, this methodology brought almost nothing new. The methodology authors took the ideas of Infrastructure as code (IaC), added a PULL-based approach with a reconciliation loop from SCM of the last generation (puppet, chef), and made it mandatory to keep the configuration in GIT. Indeed, a new and key concept of GitOps is the concept of Git as a single source of truth, but this concept is clearly mediocre and has some flaws. In fact, it's an attempt to take a deployment step from an application delivery pipeline and put it into Git. One of the key principles of DevOps is that there are no bad actors. There are bad processes. Hence the blameless culture and the understanding that you need to organize processes in such a way that the problem becomes impossible. But because of the limited concept of GitOps, it has too many gaps and potential problems, too many things to do "out of GitOps scope". All this leads to misunderstanding of how to organize processes properly and, consequently, to workarounds with all the consequences. To build a good GitOps process, a company needs high IT maturity because the authors of the concept left out too many important things. So, GitOps does not provide any significant advantage over a well-organized CI Ops process, although it significantly increases the final system complexity and labor costs. And in my opinion, many GitOps tools understand this by implementing GitOps in addition to image building, e.g., Gitkube and JenkinsX. Conclusion What else are we missing to make us happy? We obviously don't have enough notifications. We have an asynchronous pull model. In this model, we don't understand what exactly this pull executor is doing on its side if it doesn't explicitly notify us. Let's add the installation and configuration of the notification controller to our checklist and match it. So, to work with GitOps, we need to do: Flux-fleet repository Bootstrap flux in a Kubernetes cluster using the flux-fleet repository Script for building and pushing images in the Docker registry Infrastructure Git repository Account for CI system access to the infrastructure GIT repository Script to generate and push the HelmRelease file Helm Repository Account for CI system access to the Helm repository Script to build and publish the Helm chart Flux account for the infrastructure repository Flux account for Helm chartʼs repository Flux account for the application repository Flux account for cluster access for an engineer on the Ops team GIT repository for "single point of platform configuration" Flux account to access the GIT repository Slack/Teams channel for Notification Controller Most of these items are "out of GitOps scope". And it's not a methodology problem. But as an experienced engineer, I understand that I can't do GitOps without these items. What about CI Ops? As part of CI Ops on Gitlab, we don't need to do many extra actions to make GitOps work. We need to: Make the CI script to build and publish a docker image Add Helm release deploy to the script via Helm apply Create an account for the CI system to access Kubernetes With that, deprived of the illusion that it's secure, we'll be more responsible in setting up access to the CI system and the cluster. In addition, we get a built-in integrated docker registry, built-in environment handling, built-in secrets handling, built-in notification of the execution of the pipeline stages, and some integrations.

What Is GitOps And Why Is It (Almost) Useless? Part 2

is an excellent key-value distributed database used internally by Kubernetes and managed by the CNCF. It's a great option, and that's the reason why Apache APISIX uses it too. Yet, it's not devoid of issues. etcd First, some mention scalability, but one can expect this from a distributed data store that values consistency. Another issue may be the need for more familiarity with etcd. It's relatively new, so your Ops team may need help operating it correctly while having decades of operating MySQL or Postgres. Finally, only a few etcd users are aware that it lacks maintainers: In the last few months, primary maintainers G.L. (Amazon, announcement) and S.B. (Red Hat) have stopped actively participating in the project. This leaves the project with only one active and two occasionally-reviewing maintainers, M.S. (Google),  P.T. (Google), both are relatively new to the project (1 month and 1 year of tenure) and S.P.Z. (IBM). Other maintainers are either dormant or have very minimal activity over the last six months. . The project is effectively unmaintained (emphasis mine) -- Google Groups of Kubernetes Steering Committee, March 2022 For all those reasons, you may prefer to use a standard SQL database with Apache APISIX. In this post, I'll show how you can use MySQL. The kine project It would be a lot of effort if each product had to introduce an abstraction layer and different adapters for both etcd and other databases. kine is a project that aims to offer a translation step between etcd calls and other implementations: Kine is an etcdshim that translates etcd API to: SQLite Postgres MySQL NATS Jetstream Features Can be ran standalone so any k8s (not just K3s) can use Kine Implements a subset of etcdAPI (not usable at all for general purpose etcd) Translates etcdTX calls into the desired API (Create, Update, Delete) -- Kine (Kine is not etcd) In essence, kine is a Go library that translates etcd calls to the datastore you want (among those implemented). Yet, using kine directly is a non-trivial effort. Fortunately, , the company that gave Apache APISIX to the Apache Software Foundation, provides a component already focused on APISIX usage. api7 ETCD adapter ETCD adapter wraps kine to be APISIX-specific: ETCD Adapter mimics the ETCD V3 APIs best effort. It incorporates the kine as the Server side implementation, and it develops a totally in-memory watchable backend. Not all features in ETCD V3 APIs supported, this is designed for Apache APISIX, so it's inherently not a generic solution. -- ETCD adapter Two things of note: At the moment of this writing, the adapter supports either local in-memory storage or MySQL It's available as an embeddable library but also as a standalone component Therefore, we can design our architecture as the following: Demo Let's implement the above architecture with an additional admin UI over MySQL. I'll use Docker Compose: version: "3"

services:
  apisix:
    image: apache/apisix:3.4.0-debian                         #1
    volumes:
      - ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
    ports:
      - "9080:9080"
      - "9180:9180"
    depends_on:
      - etcd-adapter
    restart: always                                           #2
  etcd-adapter:
    build: ./etcd-adapter                                     #3
    volumes:
      - ./adapter.yml:/etcd-adapter/conf/config.yaml:ro       #4
    depends_on:
      - mysql
    restart: always                                           #2
  mysql:
    image: bitnami/mysql:8.0                                  #5
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: etcd
      MYSQL_PASSWORD: etcd
      MYSQL_DATABASE: apisix
  adminer:
    image: adminer:standalone                                 #6
    ports:
      - "8080:8080"
    environment:
      ADMINER_DEFAULT_SERVER: mysql
    depends_on:
      - mysql Latest version of Apache APISIX, yeah! To avoid any failure with dependencies between containers, restart until it works. Kubernetes's manifests would involve health checks still needs to provide a container. We need to build from the source code api7.ai Override the default configuration file with a context-specific one The regular MySQL image didn't work for me. Let's take the one from Bitnami Adminer, formerly known as PHP myAdmin, will help to visualize the database state ETCD-adapter's configuration looks like this: server:
  host: 0.0.0.0                 #1
  port: 12379

log:
  level: info

datasource:
  type: mysql                   #2
  mysql:
    host: mysql                 #3
    port: 3306                  #3
    username: etcd              #3
    password: etcd              #3
    database: apisix Bind any IP since Docker will assign a random one Implementation type. The default is ; we need to change it. btree As configured in the file docker-compose.yml Finally, here's Apache APISIX configuration: deployment:
  admin:
    allow_admin:
      - 0.0.0.0/0
  etcd:
    host:
      - "http://etcd-adapter:12379"   #1 Use this etcd instance, which is the adapter Testing Now that we are set let's test our system by creating a route: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{                                          
  "methods": ["GET"],
  "uris": ["/get"],
  "upstream": {
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}' http://localhost:9180/apisix/admin/routes/1 We can now get it: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' http://localhost:9180/apisix/admin/routes/1 We can also check via the Adminer interface that it has been persisted via MySQL: Unfortunately, we need to stop at this point. Getting all routes doesn't work: curl -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' http://localhost:9180/apisix/admin/routes {"header":{"revision":"1689689596"},"message":"Key not found"} Worse, using the route fails: curl localhost:9080/get {"error_msg":"404 Route Not Found"} Conclusion etcd is an excellent piece of infrastructure Kubernetes uses, but there might be better choices in some contexts. Worse, it might become a security threat in the future - or is already one, because of the lack of maintenance. Being able to move away from etcd is a considerable benefit. kine offers an etcd-compatible facade and multiple implementations. Using kine with Apache APISIX requires some adaptation effort, already done in ETCD-Adapter. Currently, ETCD-Adapter is not feature-complete (to say the least) and requires more love. That's why it was not donated to the Apache Foundation yet. If you're a Go developer and are interested in the project, feel free to subscribe to the to offer your help. Apache APISIX mailing list and/or join our Slack The complete source code for this post can be found on . GitHub To go further: Kine ETCD Adapter "Goodbye etcd, Hello PostgreSQL: Running Kubernetes with an SQL Database" Originally published at on July 23rd, 2023 A Java Geek

Apache APISIX Without etcd?

Creating a Google Kubernetes Engine (GKE) Cluster with Terraform in a Custom VPC

Efficient AKS Cluster Provisioning in a Virtual Network Using Terraform

When learning a new technology, sometimes I just want to see it work. It gives me a baseline to extend my ideas, to see what is possible, and to imagine what it can become. While reading through documentation or following tutorials, I am often greeted with numerous options and configurations. After seeing all the dazzling capabilities, I always want to see the core, the fundamental. Didn’t you ever just want to experience a simple working copy? Hence, I just want to see it work. This series aims at minimizing the possibility of having a missing link and encourages you to build your next innovative solution based on what you learned here. What is Kafka Stream? According to , Kafka Streams is a client library for building applications and where the input and output data are stored in Kafka clusters. Kafka microservices, What do we want to achieve? Run Kafka with KRaft in docker. Run a producer on your local machine (not in docker) that publishes messages to a topic. Run a consumer on your local machine (not in docker) that pulls a message whenever a message is published in a topic.] Resources You can find the source code at my GitHub repository Docker image: Bitnami Kafka Run Kafka in Docker Navigate to where the docker-compose.yml is. Run the below command in cmd to start a docker container of Kafka in the background. docker-compose up -d Docker-compose.yml version: "3"
services:
  kafka:
    image: "bitnami/kafka:3.2.3"
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 It controls whether the Kafka broker allows connections over plaintext listeners. A plaintext listener allows connections over an unencrypted and non-authenticated channel. It means the data exchanged between the client and the Kafka broker is sent in plain text. ALLOW_PLAINTEXT_LISTENER: Enable KRaft mode. KAFKA_ENABLE_KRAFT: The hostname a client should use to connect to the Kafka broker. You need this for both producer and consumer to communicate with Kafka. KAFKA_CFG_ADVERTISED_LISTENERS: Let’s publish a message to Kafka Navigate to where is at. Start a Powershell and run the following command. KafkaStream.Producer.csproj dotnet run When you see the message: “Message delivered to input-topic”, that means you did it! What is in the Producer? We first setup the configuration for the producer. Build a message Produce the message var config = new ProducerConfig
{
	BootstrapServers = "localhost:9092",
	ClientId = "producer-1"
};

using (var producer = new ProducerBuilder<string, string>(config).Build())
{
	var message = new Message<string, string>
	{
		Key = null, // Set the key if you want to partition the messages
		Value = "Hello, Kafka!"
	};

	try
	{
		var deliveryResult = await producer
                                    .ProduceAsync("input-topic", message);

		Console.WriteLine($"Message delivered to 
                            {deliveryResult.TopicPartitionOffset}");
	}
	catch (ProduceException<string, string> ex)
	{
		Console.WriteLine($"Delivery failed: {ex.Error.Reason}");
	}
} What is bootstrap-server? It serves as an entry point for a client to connect to a Kafka broker. Let’s setup the consumer Navigate to where KafkaStream.Consumer.csproj is at. Start a Powershell and run the following command. dotnet run The initialization will take a few seconds. Then you should see the following messages. Consuming messages from topic: input-topic
Press any key to exit
Received message: Hello, Kafka! from partition [0] offset 0 There you go! What is in the Consumer? The main thread sets up the Kafka consumer and starts a background thread to handle message consumption. In the background thread, the code runs in a continuous loop, until a cancellation is requested. A cancellation can be requested by pressing any key while the program is running. const string Topic = "input-topic";

// Configure the consumer
var config = new ConsumerConfig
{
	BootstrapServers = "localhost:9092",
	GroupId = "consumer-group-1",
	AutoOffsetReset = AutoOffsetReset.Earliest,
	EnableAutoCommit = false // Disable auto commit to have more control over offsets
};

using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
	// Subscribe to the topic
	consumer.Subscribe(Topic);

	Console.WriteLine($"Consuming messages from topic: {Topic}");

	// Start consuming messages in a background thread
	var cancellationTokenSource = new CancellationTokenSource();
	var cancellationToken = cancellationTokenSource.Token;

	var consumerThread = new Thread(() =>
	{
		try
		{
			while (true)
			{
				try
				{
					var consumeResult = 
                          consumer.Consume(cancellationToken);

					Console.WriteLine($"Received message: 
                          {consumeResult.Message.Value} from partition 
                          {consumeResult.Partition} offset 
                          {consumeResult.Offset}";

					// Manually commit the offset to mark the message as consumed
					consumer.Commit(consumeResult);
				}
				catch (ConsumeException ex)
				{
					Console.WriteLine($"Error occurred: 
                        {ex.Error.Reason}");
				}
			}
		}
		catch (OperationCanceledException)
		{
			// This exception will be thrown when cancellation is requested.
			Console.WriteLine("Cancellation requested");
		}
		finally
		{
			consumer.Close();
		}
	});

	// Start the consumer thread
	consumerThread.Start();

	// Wait for a key press to exit
	Console.WriteLine("Press any key to exit");
	Console.ReadKey();

	// Request cancellation and wait for the consumer thread to stop
	cancellationTokenSource.Cancel();
	consumerThread.Join();

	Console.WriteLine("End Consumer");
} What does offset do? In this context, it tells the consumer where it left off. Gotcha While developing your own producer/consumer, if you get an error message indicating the hostname cannot be resolved, check KAFKA_CFG_ADVERTISED_LISTENERS. If you still run into the same issue, consider updating the host file. Conclusion In this article, we successfully run Kafka in a docker container, produce a message to a topic, and consume a message from the topic. There are limitless applications of using Kafka. Go ahead and play with the project, and build something interesting of your own!

Stories, News & Mentions about BITNAMI

Bitnami’s Stories on HackerNoon

Bitnami's News & Mentions