Confluent shows rising price performance, earning an upgrade to its IBD Relative Strength Rating from 79 to 82.

Confluent Stock Scores Relative Strength Rating Upgrade; Hits Key Benchmark

The stock of Confluent Inc (NASDAQ: CFLT) has increased by 1.25 when compared to last closing price of 27.22.Despite this, the company has seen a loss of -2.06% in its stock price over the last five trading days.

Confluent Inc (CFLT) Shares Up Despite Recent Market Volatility

Confluent Inc’s filing revealed that its Chief Technology Officer Verbowski Chad unloaded Company’s shares for reported $0.25 million on Sep 21. In the deal valued at $29.80 per share,8,519 shares were sold.

Recent Insider Activity Could Benefit Confluent Inc (CFLT)

Data infrastructure software company, Confluent (NASDAQ:CFLT) will be reporting earnings tomorrow after market hours. Here's what investors should know.

Confluent (CFLT) To Report Earnings Tomorrow: Here Is What To Expect

video British Pound Price Chart – GBP/USD Daily: chart The British Pound has been defending confluent support all month long at 1.2084/89- a region defined by the objective ...

Robin Hood NYC 2023 Fireside Chat Stan Druckenmiller and Paul Tudor Jones

CFLT stock has gone down by -9.52%, with a monthly decline of -7.76% and a quarterly plunge of -21.68%. The volatility ratio for the week is 6.07%, and the volatility levels for the last 30 days are 5.

Confluent Inc (CFLT) Stock: A Review of the Recent Movement

Neo4j today announced major new capabilities enabling cloud or self managed customers to increase analytical queries

Neo4j Advances Data Processing: Combine Transactional and Analytical Workloads in One Database

Graph database market leader Neo4j Inc. today announced new features that it says can improve the speed of analytical queries up to 100-fold and enable users to run both transactional and analytical workloads within a single database while tracking real-time data changes.

Neo4j boosts query speed and support for hybrid transaction/analytical uses

Neo4j®, the world's leading graph database and analytics company, announced major new capabilities enabling cloud or self managed customers

Neo4j Enhances Cloud Database Performance Enabling up to 100X Faster Analytics and Real-Time Decision Making

Confluent Inc. is a company that has been making waves in the world of data streaming with its innovative approach. Founded in 2014 by the creators of Apache Kafka, Confluent has quickly become a leader in the industry, providing a platform that enables ...

Introduction to Confluent Inc

Confluent

confluent

We recently published a series of tutorial videos and on the Apache Kafka® platform as we see it. After you hear that there’s a thing called Kafka but before you put hands to keyboard and start writing code, you need to form a mental model of what the thing is. tweets give you the basics you need to know to have the broad grasp on Kafka necessary to continue learning and eventually start coding. This post summarizes them. These videos Events Pretty much all of the programs you’ve ever written respond to events of some kind: the mouse moving, input becoming available, web forms being submitted, bits of JSON being posted to your endpoint, the sensor on the pear tree detecting that a partridge has landed on it. Kafka encourages you to see the world as sequences of events, which it models as key-value pairs. The key and the value have some kind of structure, usually represented in your language’s type system, but fundamentally they can be anything. Events are immutable, as it is (sometimes tragically) impossible to change the past. Kafka topics Because the world is filled with so many events, Kafka gives us a means to organize them and keep them in order: topics. A topic is an ordered log of events. When an external system writes an event to Kafka, it is appended to the end of a topic. By default, messages aren’t deleted from topics until a configurable amount of time has elapsed, even if they’ve been read. Topics are properly , not queues; they are durable, replicated, fault-tolerant records of the events stored in them. logs Logs are a very handy data structure that are efficient to store and maintain, but it’s worth noting that reading them is not too exciting. You can really only scan a log, not query it, so we’ll have to add functionality on a future day to make this more pleasant. Partitioning Topics are stored as log files on disk, and disks are notoriously finite in size. It would be no good if our ability to store events were limited to the disks on a single server, or if our ability to publish new events to a topic or subscribe to updates on that topic were limited to the I/O capabilities of a single server. To be able to scale out and not just up, Kafka gives us the option of breaking topics into partitions. are a systematic way of breaking the one topic log file into many logs, each of which can be hosted on a separate server. Partitions This gives us the ability in principle to scale topics out forever, although practical second-order effects and the finite amount of matter and energy available in the known universe to perform computation do place some upper bounds on scalability. Kafka brokers Kafka is distributed data infrastructure, which implies that there is some kind of node that can be duplicated across a network such that the collection of all of those nodes functions together as a single Kafka cluster. That node is called a . broker A broker can run on bare metal hardware, a cloud instance, in a container managed by Kubernetes, in Docker on your laptop, or wherever JVM processes can run. Kafka brokers are intentionally kept very simple, maintaining as little state as possible. They are responsible for writing new events to partitions, serving reads on existing partitions, and replicating partitions among themselves. They don’t do any computation over messages or routing of messages between topics. Replication As a responsible data infrastructure component, Kafka provides replicated storage of topic partitions. Each topic has a configurable replication factor that determines how many of these copies will exist in the cluster in total. One of the replicas is elected to be the leader, and it is to this replica that all writes are produced and from which all reads are probably consumed. (There are some advanced features that allow some reads to be done on non-leader partitions, but let’s not worry about those here on day five.) The other replicas are called , and it is their job to stay up to date with the leader and be eligible for election as the new leader if the broker hosting the current leader goes down. followers Kafka producers Once the Kafka cluster is up and running with its minimal feature set, we need to be able to talk to it from the outside world. A is an external application that writes messages to a Kafka cluster, communicating with the cluster using Kafka’s network protocol. is a publicly documented thing, but it would be an extraordinarily bad idea to write your own interface library when so many excellent ones are available. producer That network protocol Out of the box, Apache Kafka provides a Java library, and Confluent supports libraries in Python, C/C++, .NET languages, and Go. The producer library manages all of the non-trivial network plumbing between your client program and the cluster and also makes decisions like how to assign new messages to topic partitions. The producer library is surprisingly complex in its internals, but the API surface area for the basic task of writing a message to a topic is very simple indeed. Kafka consumers The is an external application that reads messages from Kafka topics and does some work with them, like filtering, aggregating, or enriching them with other information sources. Like the producer, it relies on a client library to handle the low-level network interface in addition to some other pretty sophisticated functionality. A consumer can be just a single instance, or it can be many instances of the same program: a consumer group. consumer Consumer groups are elastically scalable by default, but the library only manages some of the challenges associated with scale-out and fault tolerance. For example, if your consumer is stateful (and it probably is), then you’ll have some extra work to do to manage that state during failover or scaling operations. The Kafka ecosystem Let’s pause on this day and set the stage for the rest of the series. With basic pub/sub, partitioning, producing, and consuming work in hand, other needs are going to arise. These things consistently emerge from organizations making serious use of Kafka. You need data integration, schema management, and options for stream processing. The Kafka community and have solved these problems in standard ways and are likely to continue solving new common problems as they arise. Confluent community Kafka Connect is a system for connecting non-Kafka systems to Kafka in a declarative way, without requiring you to write a bunch of non-differentiated integration code to connect to the same exact systems that the rest of the world is connecting to. Kafka Connect Connect runs as a scalable, fault-tolerant cluster of machines external to the Kafka cluster. Rather than write bespoke code to read data from a database or write messages to Elasticsearch, you deploy pre-built connectors from the extensive connector ecosystem, and configure them with a little bit of JSON. Connect then reads data from source systems and writes it to sink systems automatically. Schema Registry Schema change is a constant fact of life. Any time you serialize data, put it somewhere, and hope to go get it from that place later on, changes in the format of the data are a perennial challenge. We feel this problem most acutely in database schemas, but message formats in Kafka are no exception. The exists to help manage schema change over time. When you release a new producer or a new consumer application with a modified message format, the Schema Registry will help the client application determine whether the new schema is compatible with the old one, given the expectations of other clients that have yet to be versioned. It’s an indispensable tool for a complex deployment. Confluent Schema Registry Kafka Streams Producing messages to Kafka is often fairly simple: Messages come from some source, either read from some input or computed from some prior state, and they go into a topic. But reading gets complicated very quickly, and the consumer API really doesn’t offer much more abstraction than the producer. The exists to provide this layer of abstraction on top of the vanilla consumer. It’s a Java API that provides a functional view of the typical stream processing primitives that emerge in complex consumers: filtering, grouping, aggregating, joining, and more. It provides an abstraction not just for streams, but for streams turned into tables, and a mechanism for querying those tables as well. It builds on the consumer library’s native horizontal scalability and fault tolerance, while addressing the consumer’s limited support for state management. Kafka Streams API ksqlDB Writing stream processing applications in Java is a nice thing to do if you’re using Kafka Streams, and if you’re using Java, and if it makes sense to marry stream processing functionality with the application itself. But what if you didn’t want to do those things? Or what if you wanted a simpler approach in which you just used SQL to get your stream processing done? This is precisely what is: an application-oriented stream processing database for Kafka. A small cluster of ksqlDB nodes runs continuous stream processing queries written in SQL, constantly consuming input events and producing results back into Kafka. It exposes the same stream and table abstractions as Kafka Streams and makes tables queryable through a lightweight JSON API. ksqlDB Coda I hope this video series has helped you see the breadth of the Kafka ecosystem. I want you to have a basic mental model of how Kafka works and what other components have grown up around it to help you solve the kinds of problems that will inevitably present themselves as you build event-driven systems. If you’ve got some time in the next couple of weeks, and if a bit of study helps you relax as it does for me, be sure to . check them out Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the senior director of developer advocacy. He can frequently be found at speaking at conferences in the U.S. and all over the world. He is the co-presenter of various O’Reilly training videos on topics ranging from Git to distributed systems, and is the author of . He lives in Littleton, CO, U.S., with the wife of his youth, their three children having grown up. Gradle Beyond the Basics

Apache Kafka: How Does It Work?

Kafka Schema Evolution: A Guide to the Confluent Schema Registry

Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit), CDK for Terraform, and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks, Confluent, and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent, Snowflake, Databricks, and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization. Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? Infrastructure as code (IaC) From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. Terraform CloudFormation On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. Resourcely 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit) , CDK for Terraform , and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. AWS CDK (Cloud Development Kit) CDK for Terraform Pulumi While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks , Confluent , and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. Databricks Confluent Elastic Kubernetes While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent , Snowflake , Databricks , and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Confluent Snowflake Databricks dbt Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization.

Platform Engineering Abstraction: How to Scale IaC for Enterprise

Step-by-step guide on how to use the .Net Aspire Kafka component in Visual Studio. .Net Aspire framework is used to develop cloud and production-ready distributed applications. It consists of components to handle cloud-native concerns such as Redis, Postgres etc. Prerequisites Install .Net 8
Install Visual Studio 2022 version 17 or higher
.Net Aspire Workload
Container runtime such as Docker Desktop 10 Day .Net Aspire Challenge Objectives Learn how to create a starter project using .Net Aspire with the Apache Kafka component. Github Sample: The solution structure is divided into the following projects: DotnetAspireChallenge.ApiService
DotnetAspireChallenge.AppHost
DotnetAspireChallenge.ServiceDefaults
DotnetAspireChallenge.Web Getting Started Step 1: Install the following NuGet package Install the following Nuget package into the subsequent project “DotnetAspireChallenge.AppHost” dotnet add package Aspire.Hosting.Kafka In the above project, register Kafka UI as shown below var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); Step 2: Add dependency of Kafka Producer Add the dependency in the Program.cs file of the project “**DotnetAspireChallenge.**ApiService” builder.AddKafkaProducer<string, string>("messaging"); and add a relevant minimal API endpoint using the following code. public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } The endpoint takes two parameters namely key and value as route values, and produces the message on the docker-hosted Kafka server. https://localhost:7313/send?key=key&value=1 Step 3: Add dependency of Kafka Consumer Now move “DotnetAspireChallenge.Web” project wherein register as a Kafka producer builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); Note: It's mandatory to provide a default group ID. Step 4: Create a Razor Page Create a basic razor page named “KafkaConsumer.razor” to show the consumed message from the Kafka server. @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } Step 5: Configure HttpCall to the ApiService public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } Kafka Produce Demo Kafka UI Demo Kafka Consume Demo Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge
Cheatsheet: Cheat Sheets — .Net Step-by-step guide on how to use the .Net Aspire Kafka component in Visual Studio. .Net Aspire framework is used to develop cloud and production-ready distributed applications. It consists of components to handle cloud-native concerns such as Redis, Postgres etc. Prerequisites Install .Net 8 Install Visual Studio 2022 version 17 or higher .Net Aspire Workload Container runtime such as Docker Desktop Install .Net 8 .Net 8 Install Visual Studio 2022 version 17 or higher Visual Studio 2022 .Net Aspire Workload Container runtime such as Docker Desktop Docker Desktop 10 Day .Net Aspire Challenge 10 Day .Net Aspire Challenge 10 Day .Net Aspire Challenge Objectives Learn how to create a starter project using .Net Aspire with the Apache Kafka component. Github Sample : The solution structure is divided into the following projects: Github Sample DotnetAspireChallenge.ApiService DotnetAspireChallenge.AppHost DotnetAspireChallenge.ServiceDefaults DotnetAspireChallenge.Web DotnetAspireChallenge.ApiService DotnetAspireChallenge.AppHost DotnetAspireChallenge.ServiceDefaults DotnetAspireChallenge.Web Getting Started Step 1: Install the following NuGet package Install the following Nuget package into the subsequent project “ DotnetAspireChallenge.AppHost ” DotnetAspireChallenge.AppHost dotnet add package Aspire.Hosting.Kafka dotnet add package Aspire.Hosting.Kafka In the above project, register Kafka UI as shown below var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); Step 2: Add dependency of Kafka Producer Add the dependency in the Program.cs file of the project “**DotnetAspireChallenge.**ApiService” builder.AddKafkaProducer<string, string>("messaging"); builder.AddKafkaProducer<string, string>("messaging"); and add a relevant minimal API endpoint using the following code. public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } The endpoint takes two parameters namely key and value as route values, and produces the message on the docker-hosted Kafka server. https://localhost:7313/send?key=key&value=1 https://localhost:7313/send?key=key&value=1 Step 3: Add dependency of Kafka Consumer Now move “ DotnetAspireChallenge.Web ” project wherein register as a Kafka producer DotnetAspireChallenge.Web builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); Note: It's mandatory to provide a default group ID. Note: It's mandatory to provide a default group ID. Step 4: Create a Razor Page Create a basic razor page named “KafkaConsumer.razor” to show the consumed message from the Kafka server. @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } Step 5: Configure HttpCall to the ApiService public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } Kafka Produce Demo Kafka UI Demo Kafka Consume Demo Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge Cheatsheet: Cheat Sheets — .Net Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge Cheatsheet: Cheat Sheets — .Net Cheat Sheets — .Net

10 Days of .Net Aspire: Day 5 — Integrating Apache Kafka 

Unlocking the Power of Data Lakes for Embedded Analytics in Multi-Tenant SaaS

মাল্টি-টেন্যান্ট SaaS-এ এমবেডেড অ্যানালিটিক্সের জন্য ডেটা লেকের পাওয়ার আনলক করা

Entfesseln Sie die Leistungsfähigkeit von Data Lakes für eingebettete Analysen in Multi-Tenant-SaaS

Liberando el poder de los lagos de datos para análisis integrados en SaaS multiinquilino

Libérer la puissance des lacs de données pour l'analyse intégrée dans les SaaS multi-locataires

मल्टी-टेनेंट SaaS में एम्बेडेड एनालिटिक्स के लिए डेटा लेक्स की शक्ति को अनलॉक करना

マルチテナント SaaS の組み込み分析のためのデータレイクのパワーを解き放つ

멀티 테넌트 SaaS에 내장된 분석을 위한 데이터 레이크의 강력한 기능 활용

Desbloqueando o poder dos data lakes para análises incorporadas em SaaS multilocatário

Раскрытие возможностей озер данных для встроенной аналитики в мультитенантном SaaS

Çok Kiracılı SaaS'ta Gömülü Analitik için Data Lakes'in Gücünü Ortaya Çıkarma

Khai thác sức mạnh của Hồ dữ liệu cho Phân tích nhúng trong SaaS nhiều người thuê

释放数据湖的力量，实现多租户 SaaS 中的嵌入式分析

Ockam teamed up with Redpanda to launch Redpanda Connect with Ockam: the first zero-trust streaming data platform. This is a natural partnership because both companies have the same ethos; to enable every developer to build distributed systems, at scale, with simple tools. Both companies' products are based on popular open-source projects and are built by veteran, high-performing teams. Redpanda Connect with Ockam is the only enterprise-scale, zero-trust, streaming data platform that's simultaneously easy to deploy, secure-by-design, and highly performant. “Enterprise executives tell me that their high-value streaming applications take a long time to build and to ship,” said Matthew Gregory, Ockam's CEO. “Their biggest obstacle is the construction of secure streaming data connections across their siloed enterprise, and between partner companies. Secure data streaming is now simple to set up, trivial to maintain, and almost impossible to mess up. We've unlocked data so application development can move fast.” Enterprise Kafka product vendors defer zero-trust implementation details to their customers. It's difficult, even for sophisticated engineering organizations, to pull together a team that can choose and assemble the dozens of parts required to build a secure-by-design streaming data platform. Redpanda Connect with Ockam empowers a single developer to easily create end-to-end encrypted streaming pipelines that can connect 100's of applications across clouds and hybrid infrastructure. It builds trust in data streams, eliminates sources of risk to critical data, and removes the operational complexities inherent in managing keys at scale. Redpanda provides a declarative data streaming service that enables developers and organizations to build simple, chained, and stateless processing steps for their data pipelines. Our partnership with Redpanda brings a truly zero-trust capability to these pipelines. Ockam's cryptographic protocols and data integrity guarantees remove the risk of MITM and supply chain attacks that could otherwise tamper with data inflight and lead to data poisoning; highlighted as one of the OWASP Top 10 risks for companies that leverage AI and Large Language Model (LLM) systems. The mutual authentication and encrypted data protocols that Ockam provides also mean organizations can transmit highly sensitive data with cryptographic guarantees so only the intended recipients will be able to decrypt and process it. This will unlock new high-value applications that businesses may have been unwilling to explore due to privacy and governance risk, or complexity. “We help the world's largest companies build and scale streaming applications," said Alex Gallego, Redpanda's CEO. "Redpanda Connect with Ockam was conceived to empower our customers to build streaming data applications with ease. Redpanda Connect comes with over 200 pre-built integrations that will quickly connect to and unlock your data silos. Ockam provides the mutual authentication and end-to-end encryption that zero-trust streaming data applications require. Together with Ockam, we've abstracted away all of the complex pieces that cause engineering teams to stumble.” Confluent's “2023 Data Streaming Report: Moving Up the Maturity Curve” recently surveyed 2,250 IT and engineering leaders from organizations that use streaming data. 74% cited that their streaming data applications are blocked because of fragmented projects, uncoordinated teams, and constrained engineering budgets. 32% claim that they successfully “reduced security risks” after implementing data streaming services across their applications. However, 94% have high-security expectations for these same applications. These are alarming numbers! Despite the obvious benefits that a real-time streaming data architecture provides, companies struggle to ship data pipelines to their applications. Often, they're blocked for several quarters. Once they get to production, they realize that their data doesn't have the security guarantees that they need or expect from their streaming data infrastructure. Redpanda Connect with Ockam aims to move these numbers to 0% and 100%, respectively. Let's go! Connect, Secure, and Stream—All In One Simple Platform Connect With Redpanda Connect (Formerly Known as Benthos) Redpanda Connect empowers engineers to build connectors and integrations to over 200 applications, including SQL, Mongo, Snowflake, Cassandra, Redis, Splunk, AWS Lambda, Azure CosmosDB, and GCP Cloud Storage. Redpanda Connect can process streaming data with transforms, mapping, filtering, hydration, and enrichment capabilities. Secure With Ockam Zero-trust in your data streaming infrastructure allows you to have absolute trust in your data and applications. Ockam adds the guarantees of data confidentiality, integrity, and authenticity, so you can enforce data governance and privacy policies at scale. Each producer creates its own unique cryptographically provable identity and encryption keys. The producers use their keys to establish trusted secure channels through Redpanda Cloud and to the consumers across the streaming platform. All the data that moves through your streaming data platform is encrypted in motion - even when it's passing through a broker. Keys, enrollments, and credentials are safely created, stored, rotated, and revoked automagically so there's almost nothing to manage. Ockam makes it simple to Build Trust across your entire data layer - in a way that's almost impossible to mess up. Stream With Redpanda Cloud Redpanda Cloud is a complete streaming data platform delivered as a fully managed service with automated upgrades and patching, data and partition balancing, and 24×7 support. It continuously monitors and maintains your clusters along with the underlying infrastructure to meet strict performance, availability, reliability, and security requirements. Give It a Try https://www.ockam.io/blog/redpanda_connect_with_ockam#give_it_a_try Ockam teamed up with Redpanda to launch Redpanda Connect with Ockam: the first zero-trust streaming data platform . This is a natural partnership because both companies have the same ethos; to enable every developer to build distributed systems, at scale, with simple tools. Both companies' products are based on popular open-source projects and are built by veteran, high-performing teams. Redpanda Redpanda Redpanda Connect with Ockam: the first zero-trust streaming data platform Redpanda Connect with Ockam is the only enterprise-scale, zero-trust, streaming data platform that's simultaneously easy to deploy, secure-by-design, and highly performant. Redpanda Connect with Ockam is the only enterprise-scale, zero-trust, streaming data platform that's simultaneously easy to deploy, secure-by-design, and highly performant. “Enterprise executives tell me that their high-value streaming applications take a long time to build and to ship,” said Matthew Gregory, Ockam's CEO. “Their biggest obstacle is the construction of secure streaming data connections across their siloed enterprise, and between partner companies. Secure data streaming is now simple to set up, trivial to maintain, and almost impossible to mess up. We've unlocked data so application development can move fast.” Enterprise Kafka product vendors defer zero-trust implementation details to their customers. It's difficult, even for sophisticated engineering organizations, to pull together a team that can choose and assemble the dozens of parts required to build a secure-by-design streaming data platform. Redpanda Connect with Ockam empowers a single developer to easily create end-to-end encrypted streaming pipelines that can connect 100's of applications across clouds and hybrid infrastructure. It builds trust in data streams, eliminates sources of risk to critical data, and removes the operational complexities inherent in managing keys at scale. Redpanda provides a declarative data streaming service that enables developers and organizations to build simple, chained, and stateless processing steps for their data pipelines. Our partnership with Redpanda brings a truly zero-trust capability to these pipelines. Ockam's cryptographic protocols and data integrity guarantees remove the risk of MITM and supply chain attacks that could otherwise tamper with data inflight and lead to data poisoning; highlighted as one of the OWASP Top 10 risks for companies that leverage AI and Large Language Model (LLM) systems. The mutual authentication and encrypted data protocols that Ockam provides also mean organizations can transmit highly sensitive data with cryptographic guarantees so only the intended recipients will be able to decrypt and process it. This will unlock new high-value applications that businesses may have been unwilling to explore due to privacy and governance risk, or complexity. “We help the world's largest companies build and scale streaming applications," said Alex Gallego, Redpanda's CEO. "Redpanda Connect with Ockam was conceived to empower our customers to build streaming data applications with ease. Redpanda Connect comes with over 200 pre-built integrations that will quickly connect to and unlock your data silos. Ockam provides the mutual authentication and end-to-end encryption that zero-trust streaming data applications require. Together with Ockam, we've abstracted away all of the complex pieces that cause engineering teams to stumble.” Confluent's “ 2023 Data Streaming Report: Moving Up the Maturity Curve ” recently surveyed 2,250 IT and engineering leaders from organizations that use streaming data. 2023 Data Streaming Report: Moving Up the Maturity Curve 2023 Data Streaming Report: Moving Up the Maturity Curve 74% cited that their streaming data applications are blocked because of fragmented projects, uncoordinated teams, and constrained engineering budgets. 74% cited that their streaming data applications are blocked because of fragmented projects, uncoordinated teams, and constrained engineering budgets. 32% claim that they successfully “reduced security risks” after implementing data streaming services across their applications. However, 94% have high-security expectations for these same applications. 32% claim that they successfully “reduced security risks” after implementing data streaming services across their applications. However, 94% have high-security expectations for these same applications. 32% claim that they successfully “reduced security risks” after implementing data streaming services across their applications. However, 94% have high-security expectations for these same applications. These are alarming numbers! Despite the obvious benefits that a real-time streaming data architecture provides, companies struggle to ship data pipelines to their applications. Often, they're blocked for several quarters. Once they get to production, they realize that their data doesn't have the security guarantees that they need or expect from their streaming data infrastructure. Redpanda Connect with Ockam aims to move these numbers to 0% and 100%, respectively. Let's go! Connect, Secure, and Stream—All In One Simple Platform Connect With Redpanda Connect (Formerly Known as Benthos) Redpanda Connect empowers engineers to build connectors and integrations to over 200 applications, including SQL, Mongo, Snowflake, Cassandra, Redis, Splunk, AWS Lambda, Azure CosmosDB, and GCP Cloud Storage. Redpanda Connect can process streaming data with transforms, mapping, filtering, hydration, and enrichment capabilities. Secure With Ockam Zero-trust in your data streaming infrastructure allows you to have absolute trust in your data and applications. Ockam adds the guarantees of data confidentiality, integrity, and authenticity, so you can enforce data governance and privacy policies at scale. Each producer creates its own unique cryptographically provable identity and encryption keys. The producers use their keys to establish trusted secure channels through Redpanda Cloud and to the consumers across the streaming platform. All the data that moves through your streaming data platform is encrypted in motion - even when it's passing through a broker. Keys, enrollments, and credentials are safely created, stored, rotated, and revoked automagically so there's almost nothing to manage. Ockam makes it simple to Build Trust across your entire data layer - in a way that's almost impossible to mess up. Stream With Redpanda Cloud Redpanda Cloud is a complete streaming data platform delivered as a fully managed service with automated upgrades and patching, data and partition balancing, and 24×7 support. It continuously monitors and maintains your clusters along with the underlying infrastructure to meet strict performance, availability, reliability, and security requirements. Give It a Try https://www.ockam.io/blog/redpanda_connect_with_ockam#give_it_a_try https://www.ockam.io/blog/redpanda_connect_with_ockam#give_it_a_try

Ockam and Redpanda Team Up: Launching the World's First Zero-Trust Streaming Data Platform

ওকাম এবং রেডপান্ডা টিম আপ: বিশ্বের প্রথম জিরো-ট্রাস্ট স্ট্রিমিং ডেটা প্ল্যাটফর্ম চালু করছে

Ockam und Redpanda schließen sich zusammen: Einführung der weltweit ersten Zero-Trust-Streaming-Datenplattform

Ockam y Redpanda se unen: lanzan la primera plataforma de transmisión de datos de confianza cero del mundo

Ockam et Redpanda s'associent : lancement de la première plateforme de données de streaming sans confiance au monde

ओक्कम और रेडपांडा ने मिलकर दुनिया का पहला जीरो-ट्रस्ट स्ट्रीमिंग डेटा प्लेटफॉर्म लॉन्च किया

Ockam と Redpanda が提携: 世界初のゼロトラスト ストリーミング データ プラットフォームを立ち上げ

Ockam과 Redpanda의 협력: 세계 최초의 제로 트러스트 스트리밍 데이터 플랫폼 출시

Ockam e Redpanda se unem: lançando a primeira plataforma de streaming de dados Zero Trust do mundo

Оккам и Redpanda объединяются: запускают первую в мире платформу потоковой передачи данных с нулевым доверием

Ockam ve Redpanda Güçlerini Birleştiriyor: Dünyanın İlk Sıfır Güven Veri Akışı Platformunu Başlatıyor

Ockam và Redpanda hợp tác: Ra mắt nền tảng dữ liệu phát trực tuyến không tin cậy đầu tiên trên thế giới

Ockam 与 Redpanda 合作：推出全球首个零信任流数据平台

Boosting Integration Test Performance with EmbeddedKafka and GraalVM

Elasticsearch Updates, Inserts, Deletes: Understanding How They Work and Their Limitations

ইলাস্টিক সার্চ আপডেট, সন্নিবেশ, মুছে ফেলা: তারা কীভাবে কাজ করে এবং তাদের সীমাবদ্ধতা বোঝা

Elasticsearch-Updates, -Einfügungen und -Löschungen: Funktionsweise und Einschränkungen

Actualizaciones, inserciones y eliminaciones de Elasticsearch: comprensión de cómo funcionan y sus limitaciones

Mises à jour, insertions, suppressions d'Elasticsearch : comprendre leur fonctionnement et leurs limites

इलास्टिकसर्च अपडेट, इंसर्ट, डिलीट: समझें कि वे कैसे काम करते हैं और उनकी सीमाएँ

Elasticsearch の更新、挿入、削除: 仕組みと制限を理解する

Elasticsearch 업데이트, 삽입, 삭제: 작동 방식과 제한 사항 이해

Atualizações, inserções e exclusões do Elasticsearch: entendendo como funcionam e suas limitações

Обновления, вставки и удаления Elasticsearch: понимание того, как они работают и их ограничения

Elasticsearch Güncellemeleri, Eklemeler, Silmeler: Nasıl Çalıştıklarını ve Sınırlamalarını Anlamak

Cập nhật, chèn, xóa Elaticsearch: Tìm hiểu cách chúng hoạt động và những hạn chế của chúng

Elasticsearch 更新、插入、删除：了解它们的工作原理及其局限性

Hi There! So, I believe each of us has encountered a situation when, after modifying an entity in the database, it's necessary to notify other systems about this change. For example, upon accepting an order, we need to inform all related services and teams about the arrival of a new order. The first solution that comes to mind is straightforward: have each service or database polled at regular intervals to check for new orders or status updates. But what if the logic becomes too complex and needs to be segmented? This is where we turn to the event-driven approach, where we broadcast what happened with our objects. is an architectural pattern that allows systems to asynchronously exchange messages or events. The main advantage of EDA is its ability to increase system scalability and flexibility by minimizing direct dependencies between its components. Event-driven architecture (EDA) This facilitates easier adaptation to changes and growth, ensures higher resilience, and simplifies the integration of diverse services. However, a challenge arises in synchronizing data between the database and the messaging system. If we first record data about a new order in the database and then attempt to send a message to Kafka, we cannot guarantee that the message will be sent. A potential producer error could lead to the loss of critical notifications. actually Attempting to wrap these operations in a single transaction (writing to the database and sending to Kafka, followed by a transaction commit) faces fundamental difficulties, as Kafka operates outside the transaction context of databases. This necessitates an alternative solution that ensures atomicity and delivery guarantees for messages without compromising the system's performance and reliability. Here is where the comes to the rescue. Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html?embedable=true The Outbox pattern involves creating a special table (outbox) in the database, where an event intended for the messaging system is recorded within the same transaction as the main data changes. Thus, both operations - changing the state in the main table and creating an event in the outbox table - are guaranteed to be executed together. Subsequently, a dedicated connector (e.g., Kafka Connect) asynchronously reads data from the outbox table and sends it to Kafka, thereby ensuring the delivery of events to the end recipients. This approach achieves high reliability and data consistency without adding undue complexity to the system's architecture or compromising its performance. Let’s review it. The Problem ACID is an acronym describing key transaction properties in databases: A transaction is either executed in full or not executed at all. Atomicity: A transaction transitions the database from one consistent state to another. Consistency: The execution of one transaction doesn't impact the execution of others. Isolation: Completed transactions persist in the database, even in case of system failures. Durability: In distributed systems, the requirement to synchronize data across multiple services is a common challenge. The conventional Two-Phase Commit (2PC) approach is often deemed suboptimal due to its sluggish and blocking nature. Moreover, dealing with data integrity recovery in the event of a system failure poses significant difficulties. The Outbox Pattern emerges as a remedy to address the need for atomicity in writing operations to the database and dispatching messages to a message queue. The core of this solution involves incorporating an additional table within the service's database, referred to as the "Outbox." This table acts as a buffer, storing messages destined for dispatch to a message queue. How the Outbox Pattern Works: Local Transaction: When performing a business operation, data changes are written to the respective database tables. A message intended for other services is placed in the Outbox table within the same transaction. Message Publication: Pollers periodically scan the Outbox table for new messages. Each message from the Outbox table is published to the corresponding message queue. After successful publication, the message is either deleted from the Outbox table or marked as sent. Let’s Code Let's break down this process step by step with a simple example, utilizing Go. You can find the source code and additional materials at the following . link Step 1: Setting Up the Order Service We'll start by creating a service for handling orders. This service will be responsible for receiving orders and recording them in the database. A key component will be implementing the "Outbox" pattern to ensure that every change in order status is reliably transmitted to Kafka. For our case, we need to create only one method: curl -X POST
  http://localhost:8080/api/v1/create-order
  -H 'Content-Type: application/json'
  -d '{
    "name": "John",
    "surname": "Doe",
    "details": {
      "code": 123,
      "operation_code": 456,
      "transaction_code": "sdfsdfsdfs"
    }
  }' The core concept of this service involves writing to both the orders table and the outbox table within a single transaction. For instance, in Go, this would be implemented as follows: func (r *OrderRepository) CreateOrder(userUUID string, order domain.Order) (*int32, error) {
	det := order.Details
	var id int32

	tx, err := r.db.Begin()
	if err != nil {
		return nil, err
	}

	err = tx.QueryRow(
		query,
		order.Name, order.Surname, det.Code, det.OperationCode, det.TransactionCode, StatusCreated).Scan(&id)

	if err != nil {
		tx.Rollback()
		return nil, err
	}

	// Save to outbox
	err = r.createOutbox(userUUID, order)
	if err != nil {
		tx.Rollback()
		return nil, err
	}

	return &id, tx.Commit()
} As a result, we’ll expect messages like this: Step 2: Configuring Kafka JDBC Connector Next, we'll set up the Kafka JDBC Connector to automatically transfer messages from the outbox table to Kafka. This allows other services to subscribe to these events and react to changes in order statuses in real-time. For the Kafka connector, I'll be using the JDBC connector. Find a good way for you to install it: https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/overview.html?embedable=true#install-the-jdbc-sink-connector After installation, just call connector API: curl http://localhost:8083/connector-plugins We’ll expect to see: [{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"10.7.5"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"10.7.5"},
{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"7.6.0-ccs"}] Next, we need to create settings for the source. Be careful at this stage, as we are providing credentials for our connector. I recommend creating separate credentials with read-only permissions for the outbox table so that this service can only observe changes. Overall, there are plenty of best practices in this regard, but we won't delve into them here. {
  "name": "jdbc-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://postgres:5432/postgres",
    "connection.user": "postgres",
    "connection.password": "postgres",
    "topic.prefix": "microshop.order-",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "table.whitelist": "outbox",
    "poll.interval.ms": "5000",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable": "false",
    "transforms": "createKey",
    "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields": "user_id"
  }
} Set - our table. "table.whitelist": "outbox" Set - our topic. "topic.prefix": "microshop.order-" Set - our message key. "transforms.createKey.fields": "user_id" More about settings and fields: … here And send it. curl -X POST -H "Content-Type: application/json" --data @jdbc-source-config.json http://localhost:8083/connectors And that's it! Now, you just need to create an order, and check if everything is working and if messages are being recorded in the tables. To view what we have in Kafka, I use Kafka UI for such tasks, which allows us to quickly see what we wrote there and if it's working. Step 3: Creating the Background Processing Service We'll create a background service that listens for messages from Kafka and executes the appropriate processing logic. For instance, this service might be responsible for updating order statuses or initiating delivery processes. Full schema: The last service is a background (order-background-service). In our case, it’s just a simple service that can consume messages from Kafka. ...
	err = c.SubscribeTopics([]string{"microshop.order-outbox"}, nil)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to subscribe to topics: %s\n", err)
		os.Exit(1)
	}

...

	ev := c.Poll(cfg.Poll)
	if ev == nil {
	  continue
	}

	switch e := ev.(type) {
		case *kafka.Message:

		// Create an instance of Outbox to store the unmarshaled data
		var outbox domain.Outbox

		// Unmarshal the JSON message into the Outbox struct
		err := json.Unmarshal(e.Value, &outbox)
		if err != nil {
			fmt.Println("Error decoding message:", err)
			return
		}
		fmt.Println("Payload:", outbox.Payload)

		_, err = c.CommitMessage(e)
		if err != nil {	
			fmt.Printf("Failed to commit message: %v\n", err)
		} Be Careful! It is essential to pause here and reflect on the following: as we can observe, entries are now being recorded in tables atomically, ensuring ACID compliance. However, our pipeline has been expanded to include both a Kafka connector and Kafka itself. It is crucial to remember the delivery guarantees that Kafka promises. Apache Kafka offers developers various levels of message delivery guarantees, enabling a balance between performance and reliability according to application requirements. These guarantees are based on the concepts of atomicity, durability, and isolation. Here are the primary delivery guarantees provided by Kafka: : This foundational guarantee ensures a message is delivered at least once. It implies that, in the event of failures or retries, a message may be delivered or processed more than once. This guarantee is facilitated by retry mechanisms in case of errors and acknowledgments from consumers. At Least Once : In this mode, messages can be lost but will not be delivered more than once. This is the fastest and least reliable mode, as it does not account for retries in case of errors. This approach may be acceptable for scenarios where the loss of some messages is not critical. At Most Once : This is the most complex and desirable guarantee, ensuring that each message is delivered and processed “exactly once”, eliminating duplication or loss. Kafka has supported "exactly once" delivery since version 0.11, using a combination of techniques such as idempotent producers and transactions to ensure atomicity across various topics and consumers. Exactly Once The choice among these guarantees depends on your application's requirements for reliability and performance. If you need more information about it just let me know. In conclusion, implementing an Outbox pattern alongside a reliable Kafka connector like the JDBC connector offers a robust solution for orchestrating data changes across microservices. This approach ensures atomicity and consistency when updating multiple systems, significantly reducing the chances of data inconsistencies. This method serves as a crucial first step toward an Event-Driven Architecture (EDA). By introducing a reliable messaging system, like Kafka, and seamlessly integrating it with a transactional database, we establish a foundation for distributing events across the microservices landscape. This approach becomes particularly valuable when considering the need for analytics, as all relevant events are readily available for consumption. Analytics teams can effortlessly retrieve and process these events, contributing to the generation of valuable insights and reports. In essence, the Outbox pattern, coupled with Kafka integration, not only addresses immediate concerns of data consistency but also lays the groundwork for a scalable, event-driven ecosystem that is essential for modern microservices architectures. This facilitates the seamless flow of information between services, unlocking the potential for enhanced analytics and real-time decision-making. If you're interested in learning more about advanced topics such as managing deletions from the Outbox table and further insights into working with this setup, consider subscribing for future updates. Your subscription will help gauge the interest in this topic and guide the direction of upcoming articles. Stay tuned for more in-depth explorations into the intricacies of microservices architectures and event-driven systems! Useful Links Kafka UI: click here Kafka JDBC connector: click here Outbox pattern: click here

Unlocking Microservices Reliability With ACID and the Outbox Pattern

, we discussed what event streaming windowing is, and we examined in detail the structure of a windowed aggregate in Kafka Streams and Flink SQL. In the first post of this series In this post, we'll dive into two specific windowing implementations: hopping and tumbling windows. Hopping Windows A hopping window has a fixed time length, and it moves forward or "hops" at a time interval smaller than the window's length. For example, a hopping window can be one minute long and advance every ten seconds. The following illustration demonstrates the concept of a hopping window: So, from the illustration above, hopping windows can produce overlapping results. A hop forward can include results contained in the previous window. Let's look at another illustration demonstrating this concept: Walking through the picture: Window one starts at 12:00:00 PM and will collect data until 12:01:00 PM (end time is exclusive). At 12:00:30 PM, due to the thirty-second advance, window two starts gathering data. Window one and window two will share data for thirty seconds from the start of window two until the end of window one. The process continues with each window advance. Let's show how you would implement a hopping window in Kafka Streams and Flink SQL. Kafka Streams Hopping Window For a hopping windowed aggregation in Kafka Streams, you'll use one of the factory methods in the class: TimeWindows KStream<String,Double> iotHeatSensorStream =
  builder.stream("heat-sensor-input",
    Consumed.with(stringSerde, doubleSerde));

iotHeatSensorStream.groupByKey()
      .windowedBy(
                  TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)) <1>
                             .advanceBy(Duration.ofSeconds(30)) <2>
                  )
        .aggregate(() -> new IotSensorAggregation(tempThreshold),
         aggregator,
         Materialized.with(stringSerde, aggregationSerde))
         .toStream().to("sensor-agg-output",
           Produced.with(windowedSerde, aggregationSerde)) By using sets the window size at one minute, the means Kafka Streams will drop any out-of-order records that would have been included in the window had they arrived in order. We'll get into grace periods more in the blog post on windowing time semantics. TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)) withNoGrace The call makes this a hopping window. It creates a window that is one minute in size and advances every ten seconds. .advanceBy(Duration.ofSeconds(30) Next, let's move on to hopping windows with Flink SQL. Flink SQL Hopping Window Note that Flink hopping windows can also be referred to as sliding windows. Kafka Stream offers a sliding window variant that behaves differently from its hopping window offering. So, for clarity, we'll only refer to Flink windows with an advance smaller than the window size as hopping windows. SELECT window_start,
       window_end,
       device_id,
       AVG(reading) AS avg_reading
FROM TABLE(HOP <1>
               (TABLE device_readings,   <2>
                     DESCRIPTOR(ts),    <3>
                     INTERVAL '30' SECONDS,  <4>
                     INTERVAL '1' MINUTES  <5>
               ))
GROUP BY window_start,
         window_end,
         device_id Specifying hopping windows by passing the function to the function. HOP TABLE The table you'll use as the source for the hopping window aggregation. The is the column with the time attribute used for the window. DESCRIPTOR This first is the amount of "hop" or advance of the window. INTERVAL The second is the size of the window. INTERVAL Now, let's move on to tumbling windows. Tumbling Windows A tumbling window has a fixed length in size and has an advance that is the same amount of time as the size. Tumbling windows are considered a specialized case of hopping windows due to the advance equalling the window size. Since a tumbling window starts a new one when the previous one ends, they don't share any data. You won't find records from one window in another one; the following illustration helps clarify this process: Stepping through this illustration Window one starts at 12:00:00 PM and will collect data until it ends. The end time of the window is exclusive. At 12:01:00 PM window two starts collecting data. Since each window starts collecting data after the previous window ends, there are no shared results. Kafka Streams Tumbling Window KStream<String,Double> iotHeatSensorStream =
  builder.stream("heat-sensor-input",
    Consumed.with(stringSerde, doubleSerde));

iotHeatSensorStream.groupByKey()
      .windowedBy(
                  TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)) <1>
                  )
        .aggregate(() -> new IotSensorAggregation(tempThreshold),
         aggregator,
         Materialized.with(stringSerde, aggregationSerde))
         .toStream().to("sensor-agg-output",
           Produced.with(windowedSerde, aggregationSerde)) Using without the clause automatically makes this a tumbling window. Since you didn't specify an advance, Kafka Streams will add one equal to the size. TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)) .advanceBy You could add the clause with the same amount of time if you choose to skip the shortened version. advanceBy Flink SQL Tumbling Window Tumbling windows in Flink SQL are defined similarly to the hopping variety with a couple of differences. SELECT window_start,
       window_end,
       device_id,
       AVG(reading) AS avg_reading
FROM TABLE(TUMBLE <1>
               (TABLE device_readings,   <2>
                     DESCRIPTOR(timestamp),    <3>
                     INTERVAL '1' MINUTES  <4>
               ))
GROUP BY window_start,
         window_end,
         device_id Specifying tumbling windows by passing the function to the function. TUMBLE TABLE The table you'll use as the source for the tumbling window aggregation. The is the column with the time attribute used for the window. DESCRIPTOR By passing a single parameter, Flink SQL will utilize this as the advance and the size. INTERVAL You define tumbling windows in Flink SQL similarly to Kafka Streams in that you only provide a one-time parameter. Use Cases Let's look at an illustration for the use case of hopping windows: So, from looking at this image, we could generalize hopping windows as "every <time-period of window advance> give me <aggregate> over the last <window size> period."  From our examples here, it would be "every 30 seconds, give me the average temp reading over the last minute." So, any problem domain where you want to closely monitor changes over time or compare changes relative to the previous reading could be a good fit for the hopping window. It’s worth noting that Kafka Streams will emit periodic results before a window closes, while Flink SQL will only emit results when it’s closed. I’ll go into more detail about this in the fourth installment of this blog series. For the tumbling window, we have another illustration: This illustration for tumbling windows can be summarized as "give me <aggregate> every <window size period>" or restated to fit the examples in this post: "give me the average temp reading over the last minute." Since tumbling windows don't share any records, any situation requiring a unique count of events per window period would be a reason to use a tumbling window. Resources Apache Flink® on Confluent Cloud Flink SQL Windows Kafka Streams windowing documentation Kafka Streams in Action 2nd Edition

Stories, News & Mentions about CONFLUENT

Confluent’s Stories on HackerNoon

Confluent's News & Mentions