How to Move End-to-end Encrypted Data Through Kafka

The Confluent add-on for Ockam Orchestrator enables tamper-proof and end-to-end encrypted message streams through Confluent Cloud, with zero-code changes. The drop-in solution enables companies to exceed common security and compliance requirements without the need for costly rearchitecture or development work. Kafka deployments typically combine authentication tokens and Transport Layer Security (TLS) to protect data moving into and out of Kafka topics. While this does secure data in transit over the Internet, it doesn't provide a complete solution to securing data as it travels Kafka. The Kafka broker will be able to temporarily see the plaintext data. through Encrypting communication both into and out of your Kafka broker combined with encryption of data at rest, inside Kafka, won't be sufficient protection from a data breach if the Kafka broker or the infrastructure it is running on is compromised, as the plaintext data and the keys to decrypt the data are available in memory. Ockam solves these problems, while providing additional risk mitigating benefits and data integrity assurances, via the Confluent add-on for Ockam Orchestrator. Trust your data-in-motion Your team has decided that integrating a stream processing platform into your systems is the best way to meet both the scale and agility needs of your business, and Apache Kafka was the obvious choice for the platform to use. You've also decided to take advantage of using an experienced managed platform in Confluent Cloud so that your team can focus on adding value rather than running infrastructure. The development is complete, the team has a working solution where producers can send data to Confluent Cloud, and consumers are able to receive it on the other side. Communication between your producers, consumers, and Confluent Cloud is encrypted and secure using Transport Layer Security (TLS) so you're about to proceed with moving things into production usage. A security and compliance review has surfaced a problem. While trust Confluent Cloud, the increased focus on reducing supply-chain risk means that having data move through their systems unencrypted no longer meets your security requirements. How do you mitigate the risk to your business should some other vendor in your supply chain have a security breach? You need to be able to ensure that: you There is no possibility for a third-party to read the data while it's in transit A third-party can not modify the data while it's in transit Only authenticated and authorized parties can produce data Without solutions to both of these there is a risk that a security breach in your own supply-chain can not only expose your sensitive data, but cause unexpected downstream escalations of their own. Integrate with no code changes Let me take you through a complete working example of how, in just a few minutes, you can configure and integrate an entire end-to-end solution that ensures that your data is always encrypted and tamper-proof while in motion through Kafka and Confluent. Note: The code examples below may change as we continue to improve the developer experience. Please check for the most up to date documentation. https://docs.ockam.io/ Initial setup of Ockam If you’ve previously set up Ockam you can skip this section and move straight to the two examples below. brew install build-trust/ockam/ockam (if you’re not using brew for package management we have in our documentation) installation instructions for other systems Once installed you need to enroll your local identity with Ockam Orchestrator, run the command below and follow the instructions provided: ockam enroll Configure the Confluent add-on We're assuming here that you've already got a working cluster on Confluent Cloud (and the Kafka Command Line Tools that come with the ), so the first step is to configure your Ockam project to use the Confluent add-on by pointing it to your bootstrap server address: latest Kafka distribution ockam project addon configure confluent \ --bootstrap-server YOUR_CONFLUENT_CLOUD_BOOTSTRAP_SERVER_ADDRESS We'll then need to save our Ockam project configuration so we can use it later to register our producers and consumer, so save the output to a file name : project.json ockam project information default --output json > project.json As the administrator of the Ockam project, you're able to control what other identities are allowed to enroll themselves into your project by issuing unique one-time use enrollment tokens. We'll start by creating one for our consumer: ockam project enroll --project-path project.json --attribute role=member > consumer.token ... and then one for the first producer: ockam project enroll --project-path project.json --attribute role=member > producer1.token The last configuration file we need to generate is , which will be where you store the username and password you use to access your cluster on Confluent Cloud: kafka.config cat > kafka.config <<EOF request.timeout.ms=30000 security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="YOUR_CONFLUENT_CLOUD_USER_NAME" \ password="YOUR_CONFLUENT_CLOUD_PASSWORD"; EOF Configure consumers to receive encrypted messages On your consumer node you'll start by creating a new identity (you'll need the Ockam Command installed, so repeat the install instructions if you're doing this on a separate host): ockam identity create consumer Copy the and files from the previous section, and then use them to authenticate and enroll this identity into your Ockam project: project.json consumer.token ockam project authenticate \ --project-path project.json \ --identity consumer \ --token $(cat consumer.token) An Ockam node is a way to connect securely connect different services to each other, so we'll create one here that we'll use to communicate through the Confluent Cloud cluster using the identity we just created: ockam node create consumer \ --project project.json --identity consumer Once that completes we can now expose our Kafka bootstrap server. This is like the remote Kafka bootstrap server and brokers have become virtually adjacent on : localhost:4000 ockam service start kafka-consumer \ --node consumer \ --project-route /project/default \ --bootstrap-server-ip 127.0.0.1 \ --bootstrap-server-port 4000 \ --brokers-port-range 4001-4100 Copy the file across, and use it to create a new topic that we'll use for sending messages between the producer and consumer in this demo (in this case we've called the topic ) kafka.config demo-topic kafka-topics.sh \ --bootstrap-server localhost:4000 \ --command-config kafka.config \ --create \ --topic demo-topic \ --partitions 3 The final step is to start our consumer script, pointing it to as our bootstrap server: localhost:4000 kafka-console-consumer.sh \ --topic demo-topic \ --bootstrap-server localhost:4000 \ --consumer.config kafka.config The consumer code will push all communication into the Ockam node process that is running on the local host. That local Ockam process will automatically manage the generation of cryptographic keys, establishing a secure channel for communication with any producer nodes, and then subsequently receiving, decrypting, and forwarding on any messages that are received from the broker running on our Confluent Cloud cluster. Send encrypted messages from multiple producers To have messages for our consumer to process, we need to have something producing them. We'll go through a very similar process now but instead create the parts necessary for a producer. We start once again by creating an identity on the producer's host (again, install the Ockam Command on that host if required): ockam identity create producer1 Copy over the and files from the earlier section and use it to authenticate and enroll into our Ockam project: project.json producer1.token ockam project authenticate \ --project-path project.json \ --identity producer1 \ --token $(cat producer1.token) Create a node and link it to both the project and identity we've created: ockam node create producer1 \ --project project.json \ --identity producer1 And expose our Kafka bootstrap server on port so we can start sending messages through Confluent Cloud: 5000 ockam service start kafka-producer \ --node producer1 \ --project-route /project/default \ --bootstrap-server-ip 127.0.0.1 \ --bootstrap-server-port 5000 \ --brokers-port-range 5001-5100 Make sure to copy the file across, and start your producer: kafka.config kafka-console-producer.sh \ --topic demo-topic \ --bootstrap-server localhost:5000 \ --producer.config kafka.config Your existing producer code will now be running, communicating with the broker via the secure portal we've created that has exposed the Kafka bootstrap server and Kafka brokers on local ports, and sending messages through to the consumer that was setup in the previous step. However, all message payloads will be transparently encrypted as they enter the node on the producer, and not decrypted until they exit the consumer node. At no point in transit can the broker see the plaintext message payload that was initially sent by the producer. To take this example further you could repeat this step for number of producers by generating a new enrollment token for each, and then repeating the steps in this section. N If you look at the encrypted messages inside Confluent Cloud, they will render as unrecognizable characters like in the following screen capture: End-to-end encrypted message streams An example that only takes a few minutes to implement means it can be easy to miss a number of the important improvements to security and integrity that comes from this approach: : each consumer and producer generates its own cryptographic keys, and is issued its own unique credentials. They then use these to establish a mutually trusted secure channel between each other. By removing the dependency on a third-party service to store or distribute keys you're able to reduce your vulnerability surface area and eliminate single points of failure. Unique keys per identity : by pushing control of keys to the edges of the system, where authenticated encryption and decryption occurs, no other parties in the supply-chain are able to modify the data in transit. You can be assured that the data you receive at the consumer is exactly what was sent by your producers. You can also be assured that only authorized producers can write to a topic ensuring that the data in your topic is highly trustworthy. If you have even more stringent requirements you can take control of your credential authority and enforce granular authorization policies. Tamper-proof data transfer : Ockam secure channels regularly rotate authentication keys and session secrets. This approach means that if one of those session secrets was exposed your total data exposure window is limited to the small duration that secret was in use. Rotating authentication keys means that even when the identity keys of a producer are compromised - no historical data is compromised. You can selectively remove the compromised producer and its data. With centralized shared key distribution approaches there is the risk that all current and historical data can’t be trusted after a breach because it may have been tampered with or stolen. Ockam's approach eliminates the risk of compromised historical data and minimizes the risk to future data using automatically rotating keys. Reduced exposure window All these benefits are possible, today, with no code changes and minimal configuration change to existing deployments. Next steps If you haven't already followed the example in this guide, you can for free by following the instructions in our documentation. get started using Ockam If you’d like to talk specifically about this or other potential use cases for Ockam, the team would be more than happy to . chat with you Alternatively you can join us, and an ever growing community of developers who want to build trust by making applications that are secure-by-design, in the or on . Build Trust Discord server Github Also published . here