A Twitter user, @Chacha_huu, voiced outrage over the exorbitant food prices at Mumbai Airport. He claimed to have paid Rs 490 for two samosas, a tea, and a water bottle, sparking a heated online debate.

Man says 'Hindu Khatre mai hai' after spending Rs 490 on samosa and tea at Mumbai airport. Netizens say this

Crowd behavior in some of the matches during the ongoing men’s Cricket World Cup tournament in India has revived the debate about whether the rise of jingoism during the rule of India’s Hindu nationalist Bharatiya Janata Party (BJP) has vitiated the atmosphere of sports in the country.

Are Visiting Teams Facing Crowd Hostility at the Cricket World Cup in India?

Navarro, along with her dog ChaCha, posed with Pascal as ‘The View' host showcased her appreciation for the actor, while also admitting that her "little Latino heart is bursting." While Navarro couldn't contain her excitement after meeting Pascal in New ...

'How lucky': 'The View' host Ana Navarro's jaw-dropping photo with Pedro Pascal leaves fans 'jealous'

High Court Judge Chacha Mwita, sitting at Nairobi's Milimani Law Courts, gave the extension even as he scheduled hearings on the matter to begin on November 9. - Kenya breaking news | Kenya news today | Capitalfm.

Court extends orders restraining Haiti Mission to November 9 when hearings start

The election in Devtalab Assembly constituency in Vindhya region of Madhya Pradesh will be more interesting because it will be

Chacha Vs Bhatija: Girish Gautam's Challenges Same As 20 Years Ago

Police in Lilongwe have arrested 22-year-old Precious Kasakatiza for operating various Facebook accounts and stealing money from Facebook users. Deputy national police spokesperson Alfred Chimthere has confirmed the arrest of the suspect.

Man arrested for duping Facebook users

By bringing in fresh faces for the November 17 Madhya Pradesh Assembly elections, the Congress has left several, more in the Vindhya region, surprised.As per the second list of 85 candidates released late on Thursday night,

MP Election 2023: New Congress Faces From Vindhya Region Makes It 'Chacha Vs Bhateeja' In Devtalab

In the 2022 Zila Panchayat election, Padmesh had defeated Girish Gautam's son Rahul Gautam, and is likely to give a tough contest in Devtalab seat. Rewa's Sirmour seat, which the BJP's Divyaraj Singh won for the last two consecutive elections (2013 and 2018), has Congress fielding Ramgareeb Adivasi.

(MENAFN- IANS) Bhopal, Oct 20 (IANS) By bringing in fresh faces for the November 17 Madhya Pradesh Assembly elections, the Congress has left several, more in the Vindhya region, surprised. As per the second list of 85 candidates released late on Thursday ...

MP Polls: New Cong Faces From Vindhya Region Makes It 'Chacha Vs Bhateeja' In Devtalab

By bringing in fresh faces for the November 17 Madhya Pradesh Assembly elections, the Congress has left several, more in the Vindhya region, surprised.

MP Polls: New Cong faces from Vindhya region makes it 'Chacha Vs Bhateeja' in Devtalab

ChaCha

chacha

Cipher suites in TLS 1.3 are a combination of algorithms used for encryption and integrity. TLS 1.3 has only five possible cipher suites, because it removed all unsecure cipher suites from TLS 1.2. TLS_AES_128_GCM_SHA256 TLS_AES_256_GCM_SHA384 TLS_AES_128_CCM_8_SHA256 TLS_CHACHA20_POLY1305_SHA256 In TLS 1.2, separate algorithms were used for encryption and integrity of the messages. However, in TLS 1.3, all cipher suites use AEAD algorithms. AEAD stands for Authenticated Encryption with Associated Data. AEAD algorithms provide both encryption and authentication in a single step, making the process a lot simpler. For example, the cipher suite TLS_CHACHA20_POLY1305_SHA256 uses ChaCha20-Poly1305, as an AEAD cipher, and SHA-256 as a hash function for the Key Derivation Function. (Reminder: Key Derivation Function is used in TLS handshake to derive many keys) Key Features About CHACHA20 It’s a symmetric key encryption algorithm Developed by Google Simple design and implementation, making it faster than AES Known for its security and high speed. Generates a stream of pseudo-random bits called the key-stream. This key-stream is then XORed with the plaintext to produce the cipher-text. Widely Supported Encryption & Integrity with ChaCha20 Okay, now the fun part. Let’s visualize how the elegant design of ChaCha20 integrates with Poly1305. First step is encrypt with CHACHA20. At a high level, ChaCha20 will take as inputs a shared secret key, a nonce, and a counter. Think of the nonce and counter as params used to increase the unpredictability and randomness of the cipher text. The "20" in "ChaCha20" refers to the number of rounds the algorithm goes through to process the data. These rounds involve various operations such as bit manipulations, addition, rotation, and XOR operations. There exists reduced round versions of ChaCha20 called ChaCha12 and ChaCha8. Based on these inputs, the ChaCha20 algorithm generates a pseudorandom stream of bits called the key-stream. Then, this key-stream is XORed with JayP’s plaintext message to produce the cipher text, in other words, the encrypted message. After that, the encrypted message, and unencrypted associated data such as addresses, ports, timestamps, together with the nonce and the secret key are inputs of POLY1305. The MAC (Message Authentication Code) POLY1305 will output a MAC. algorithm The MAC, also referred to as the authentication tag, will be sent along with the encrypted message. Verification with ChaCha20 Youtube, to decrypt and verify the message, would perform the reverse of the steps explained earlier. 1. Verification: - Youtube calculates the authentication tag from the received cipher text, the shared secret key, the associated data, and the nonce.

- The calculated authentication tag is compared to the received authentication tag.

- If they match, Youtube knows that the received message has not been altered. 2. Decryption: - The ChaCha20 stream cipher is used with all necessary params to generate the same pseudo random stream, which is XORed with the received cipher text to produce the plaintext. Also Published . here For the Animated Interactive Version: https://youtu.be/S1Awy242Vf8?embedable=true

Understanding Cipher Suites & AEAD — ChaCha20-Poly1305 Example

DJ Casper, creator of the world famous “Cha Cha Slide,” passed away recently at the age of 58. While his massively viral hit lives on, its rights remain largely with Universal Music rather than its creator. Casper’s story highlights the need for greater transparency and control for artists over their work. Blockchain technology offers a path forward. The Opacity of the Current System Under the current music industry system, creators lack visibility into the full earnings their work generates. Opaque accounting practices mean artists depend on intermediaries like record labels and collecting societies to distribute royalties. But convoluted processes make tracking compensation nearly impossible. Blockchain Creates Transparency Blockchain’s distributed ledger technology enables direct relationships between artists and fans. Musicians can register work on blockchain platforms to manage rights, track usage in real-time, and view royalty data transparently. Smart Contracts Automate Payments Smart contracts built into blockchain code can automate royalty distribution. Artists get paid quickly and accurately, sans middlemen skimming profits. Platforms are already using crypto tokens to facilitate direct artist support from fans. The Future Looks Brighter with Blockchain While adoption challenges remain, blockchain promises to empower artists with rights, profits, and legacies preserved on transparent, decentralized ledgers. DJ Casper’s legacy lives on, but blockchain could give future creators fuller control over their work. Sources: Fox News People Chicago Sun-Times “. Thank you for reading “DJ Casper, 'Cha Cha Slide' Icon, Dies at 58: A Blockchain Tribute for Artist Empowerment. Subscribe to our newsletter: Consensus Protocol Follow us on Twitter: Twitter Also published . here

DJ Casper, 'Cha Cha Slide' Icon, Passes Away at 58: A Blockchain Tribute for Artist Empowerment

Centralization as a Factor Constraining the Development of Secure Communications

Architecting a Modern Data Lake in a Post-Hadoop World

Mä Moderno Datos Quta Arquitectura ukax mä Post-Hadoop Mundo ukan luratawa

Post-Hadoop Dünyasında Müasir Məlumat Gölünün Memarlığı

Arhitektura modernog jezera podataka u post-Hadoop svijetu

Arkitektering af en moderne datasø i en post-hadoop verden

Arquitectura de un lago de datos moderno en un mundo post-Hadoop

ポストHadoopの世界における最新のデータレイクの設計

თანამედროვე მონაცემთა ტბის არქიტექტურა პოსტჰადოპის სამყაროში

Хадоопын дараах ертөнцөд орчин үеийн мэдээллийн нуурыг барих нь

Go aga Letsha la Datha la Sebjalebjale Lefaseng la Ka morago ga Hadoop

Kubaka Ikiyaga kigezweho muri nyuma yisi ya Hadoop

Arkitektimi i një liqeni modern të të dhënave në një botë post-Hadoop

Pag-arkitekto ng Modernong Data Lake sa isang Post-Hadoop World

Cryptographic Ransomware Encryption Detection: Survey: on Crypto-ransomware Behavior and Methodology

This tutorial shows you how to deploy and automate a Docker container using Pulumi in Google Cloud Platform's Cloud Run with minimum permissions. Here's the GitHub Repository for reference: https://github.com/superjose/deploy-to-cloud-run-go Get the CLIs: . Create a Google Cloud Account Install Google Cloud CLI Install Pulumi CLI - We need it to build the Docker container. Install Docker Bootstrap the project: Create a directory: pulumi mkdir pulumi Run to initialize a pulumi project with Go (It can be your language of choice pulumi new go Navigate to the directory. pulumi cd pulumi Navigate to and create a new project. cloud.google.com Take note of the (Usually the project's name). project-id Generate the necessary permissions using gcloud CLI Login with the Google Auth CLI: gcloud auth login (Optional) Set the project in google cloud CLI (Can be changed anytime). This saves you from passing into every command. --project [PROJECT-ID] gcloud If your machine has multiple GCP projects, skip this step and pass the flag into every command. --project gcloud Create a service account (The account that Pulumi will connect to): gcloud iam service-accounts create pulumi-gcp --description="Pulumi GCP" Download the credentials for the service accounts and store them locally (Remember to replace with your GCP Project Id): [PROJECT-ID] gcloud iam service-accounts keys create ~/keys/gcp/pulumi-service-account-key-file.json --iam-account=pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Set Pulumi's gcp credentials config path:
(This will connect the service account with Pulumi) pulumi config set gcp:credentials ~/keys/gcp/pulumi-service-account-key-file.json Set the GCP Project by doing: pulumi config set gcp:project [PROJECT-ID] Create a file (Inside the dir) and add the required permissions in : roles.gcp.yml pulumi includedPermissions # https://cloud.google.com/iam/docs/creating-custom-roles#creating
# Yaml to define the Pulumi GCP Roles that need to be created with gcloud CLI
title: Pulumi GCP Roles
description: |
  This policy ensures that all GCP roles are created using Pulumi.
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
  - serviceusage.services.list
  - serviceusage.services.enable
  - serviceusage.services.disable
  - serviceusage.services.get
  - serviceusage.services.use
    # Permissions for GCR
  - storage.objects.create
  - storage.objects.delete # Optional: only include if you need to delete images
  - storage.objects.get
  # Permissions for Google Artifact Registry
  - artifactregistry.repositories.create
  - artifactregistry.repositories.delete
  - artifactregistry.repositories.get
  - artifactregistry.repositories.list
  - artifactregistry.repositories.update
  - artifactregistry.repositories.downloadArtifacts
  - artifactregistry.repositories.uploadArtifacts
  - artifactregistry.repositories.deleteArtifacts
  # Permissions
  # Permissions
  - run.services.create
  - run.services.get
  - run.services.list
  - run.services.update
  - run.services.delete
  - run.services.getIamPolicy
  - run.services.setIamPolicy
  - iam.serviceAccounts.actAs
# NOTE: This should be removed the first time you're creating a role.
# This etag is to update the current active role (As GCP lets you manage multiple roles)
# I'm commenting it out so I can always replace the role
# etag: BwYS74Xx5y4= Create the with the file above:
(We assume we're running this code from the directory) pulumi_admin_role pulumi gcloud iam roles create pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' In case you need to make edits, change the file and use: gcloud iam roles update pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' We're also adding the role (I haven't found a better way) (Otherwise we'd get 403 errors when refreshing and updating in Pulumi)<sup>1</sup> serviceAccountAdmin gcloud projects add-iam-policy-binding [PROJECT-ID] --role roles/iam.serviceAccountAdmin   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com (The following will add the new role to the service account) gcloud projects add-iam-policy-binding [PROJECT-ID] --role projects/[PROJECT-ID]/roles/pulumi_admin_role   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Our in the directory:
(Check the code for comments!) main.go pulumi We enable the required services (Artifact Registry, and Cloud Run). Artifact Registry is used to host the docker container image. Cloud Runner will launch the Docker image from Artifact Registry. We build the docker image locally (We specify the platform in case you're using an ARM chip like M1, M2, Snapdragon SQ, X Elite, etc.) We create a chain of "DependsOn" to notify Pulumi:
5.1 Services need to be enabled first
5.2 We create the Artifact Repository
5.3 We build the docker image and push it to Artifact Registry.
5.4 We pull the Docker Image from Artifact Registry and run it.
5.5 We add IAM permissions so it can be accessed from anywhere. package main

import (
	"errors"
	"log"
	"os"

	"github.com/joho/godotenv"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/artifactregistry"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/cloudrun"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/projects"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

// This is the name you created in Google Cloud Platform (GCP).
const gcpProjectId = "deploy-to-cloud-run-go"

// The Docker Image Name
const dockerImageName = "my-app-docker"
const artifactRegistryServiceName = "artifact-registry-api"
const artifactRegistryRepoName = "my-app-artifact-repo"
const artifactRegistryRepoLocation = "us-east1"
const cloudRunAdminServiceName = "cloud-run-admin-service"
const cloudRunServiceName = "cloud-run-service"

// For more info: https://cloud.google.com/run/docs/locations
const cloudRunLocation = "us-east1"

// The tag for the Docker image
const imageTag = "latest"

// This is a url like: us-east1-docker.pkg.dev
// It is used to push the Docker image to Google Container Registry
// For more info: https://cloud.google.com/container-registry/docs/pushing-and-pulling
// The format is: <region>-docker.pkg.dev
var dockerGCPServer = cloudRunLocation + "-docker.pkg.dev"

// The full path to the Docker image
// It is used to deploy the Docker image to Google Cloud Run
// The format is: <region>-docker.pkg.dev/<project-id>/<repo-name>/<image-name>:<tag>
// For more info: https://cloud.google.com/run/docs/deploying
// Example: us-east1-docker.pkg.dev/deploy-to-cloud-run-go/my-app--artifact-repo/my-app-docker:latest
var dockerImageWithPath = dockerGCPServer + "/" + gcpProjectId + "/" + artifactRegistryRepoName + "/" + dockerImageName + ":" + imageTag

func main() {

	// Load the .env file
	err := godotenv.Load()

	if err != nil {
		log.Fatal("Error loading .env file")
	}

	pulumi.Run(func(ctx *pulumi.Context) error {

		enabledServices, serviceResultErr := enableServices(ctx)

		if serviceResultErr != nil {
			return serviceResultErr
		}

		artifactRegistryRepo, createArtifactErr := createArtifactRegistryNewRepository(ctx, &enabledServices)
		if createArtifactErr != nil {
			return createArtifactErr
		}

		dockerImage, buildAndPushErr := buildAndPushToContainerRegistry(ctx, &enabledServices, artifactRegistryRepo)

		if buildAndPushErr != nil {
			return buildAndPushErr
		}

		deployContainerErr := deployContainerToCloudRun(ctx, &enabledServices, dockerImage)

		if deployContainerErr != nil {
			return deployContainerErr
		}

		return nil
	})
}

type EnabledServices struct {
	CloudRunService         *projects.Service `pulumi:"cloudRunService"`
	ArtifactRegistryService *projects.Service `pulumi:"artifactRegistryService"`
}

func enableServices(ctx *pulumi.Context) (EnabledServices, error) {

	cloudRunService, cloudRunAdminErr := projects.NewService(ctx, cloudRunAdminServiceName, &projects.ServiceArgs{
		Service: pulumi.String("run.googleapis.com"),
		Project: pulumi.String(gcpProjectId),
	})

	if cloudRunAdminErr != nil {
		return EnabledServices{}, cloudRunAdminErr
	}

	artifactRegistryService, err := projects.NewService(ctx, artifactRegistryServiceName, &projects.ServiceArgs{
		Service: pulumi.String("artifactregistry.googleapis.com"),
	})

	if err != nil {
		return EnabledServices{}, err
	}
	return EnabledServices{
		CloudRunService:         cloudRunService,
		ArtifactRegistryService: artifactRegistryService,
	}, nil
}

func createArtifactRegistryNewRepository(ctx *pulumi.Context, enabledServices *EnabledServices) (*artifactregistry.Repository, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	dependingResources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
	}

	repo, err := artifactregistry.NewRepository(ctx, artifactRegistryRepoName, &artifactregistry.RepositoryArgs{
		Location:     pulumi.String(artifactRegistryRepoLocation),
		RepositoryId: pulumi.String(artifactRegistryRepoName),
		Format:       pulumi.String("DOCKER"),
		Description:  pulumi.String("The repository that will hold social-log Docker images."),
	}, pulumi.DependsOn(dependingResources))

	if err != nil {
		return nil, err
	}

	return repo, nil
}

func buildAndPushToContainerRegistry(ctx *pulumi.Context, enabledServices *EnabledServices, artifactRegistryRepo *artifactregistry.Repository) (*docker.Image, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	if artifactRegistryRepo == nil {
		return nil, errors.New("artifactRegistryRepo cannot be nil")
	}

	// Lookup GOOGLE_CREDENTIALS environment variable which should hold the path to the JSON key file
	jsonKeyPath, present := os.LookupEnv("GOOGLE_CREDENTIALS_FILE_PATH")
	if !present {
		return nil, errors.New("GOOGLE_CREDENTIALS_FILE_PATH environment variable is not set")
	}

	// Read the JSON key file
	jsonKey, err := os.ReadFile(jsonKeyPath)
	if err != nil {
		return nil, err
	}

	dependingSources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
		artifactRegistryRepo,
	}

	// Build and push Docker image to Google Container Registry using the JSON key
	image, err := docker.NewImage(ctx, dockerImageName, &docker.ImageArgs{
		Build: &docker.DockerBuildArgs{
			Context: pulumi.String("../"), // Adjust the context according to your project structure

			ExtraOptions: pulumi.StringArray{
				// This option is needed for devices running on ARM architecture, such as Apple M1/M2/MX CPUs
				pulumi.String("--platform=linux/amd64"),
			},
		},
		ImageName: pulumi.String(dockerImageWithPath),
		Registry: &docker.ImageRegistryArgs{
			Server:   pulumi.String(dockerGCPServer),
			Username: pulumi.String("_json_key"),     // Special username for GCP
			Password: pulumi.String(string(jsonKey)), // Provide the contents of the key file
		},
	}, pulumi.DependsOn(dependingSources))
	if err != nil {
		return nil, err
	}

	return image, nil
}

func deployContainerToCloudRun(ctx *pulumi.Context, enabledServices *EnabledServices, dockerImage *docker.Image) error {

	if enabledServices == nil || enabledServices.CloudRunService == nil {
		return errors.New("enabledServices cannot be nil")
	}

	if dockerImage == nil {
		return errors.New("dockerImage cannot be nil")
	}

	dependingSources := []pulumi.Resource{
		enabledServices.CloudRunService,
		dockerImage,
	}

	appService, err := cloudrun.NewService(ctx, cloudRunServiceName, &cloudrun.ServiceArgs{
		Location: pulumi.String(cloudRunLocation), // Choose the appropriate region for your service
		Template: &cloudrun.ServiceTemplateArgs{
			Spec: &cloudrun.ServiceTemplateSpecArgs{
				Containers: cloudrun.ServiceTemplateSpecContainerArray{
					&cloudrun.ServiceTemplateSpecContainerArgs{
						Image: dockerImage.ImageName,
						Resources: &cloudrun.ServiceTemplateSpecContainerResourcesArgs{
							Limits: pulumi.StringMap{
								"memory": pulumi.String("256Mi"), // Adjust the memory limit as needed
							},
						},
					},
				},
			},
		},
		Traffics: cloudrun.ServiceTrafficArray{
			&cloudrun.ServiceTrafficArgs{
				Percent:        pulumi.Int(100),
				LatestRevision: pulumi.Bool(true),
			},
		},
	}, pulumi.DependsOn(dependingSources))

	if err != nil {
		return err
	}

	_, iamErr := cloudrun.NewIamMember(ctx, "invoker", &cloudrun.IamMemberArgs{
		Service:  appService.Name,
		Location: appService.Location,
		Role:     pulumi.String("roles/run.invoker"),
		Member:   pulumi.String("allUsers"),
	})

	if iamErr != nil {
		return iamErr
	}

	ctx.Export("containerUrl", appService.Statuses.Index(pulumi.Int(0)).Url().ToOutput(ctx.Context()))
	return nil
} Update the in your Dockerfile:
There's a , and in which you need to export your environment variable to ENV known issue here HOME /root # Set the ENV HOME before your ENTRYPOINT.
ENV HOME=/root

# This is specific to your project. 
ENTRYPOINT ["/whatever-is-your-entrypoint"] Create a .env file in the pulumi directory: Set the * to the one you saved on full path 11) GOOGLE_CREDENTIALS_FILE_PATH="/Users/myusername/keys/gcp/pulumi-service-account-key-file.json" Run pulumi up You should be up and running! Extras The go.mod module social-log-go

go 1.21

toolchain go1.22.0

require (
	github.com/pulumi/pulumi-gcp/sdk/v7 v7.11.2
	github.com/pulumi/pulumi/sdk/v3 v3.108.1
)

require (
	dario.cat/mergo v1.0.0 // indirect
	github.com/Masterminds/semver v1.5.0 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
	github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
	github.com/agext/levenshtein v1.2.3 // indirect
	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
	github.com/atotto/clipboard v0.1.4 // indirect
	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
	github.com/blang/semver v3.5.1+incompatible // indirect
	github.com/charmbracelet/bubbles v0.16.1 // indirect
	github.com/charmbracelet/bubbletea v0.24.2 // indirect
	github.com/charmbracelet/lipgloss v0.7.1 // indirect
	github.com/cheggaaa/pb v1.0.29 // indirect
	github.com/cloudflare/circl v1.3.7 // indirect
	github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
	github.com/djherbis/times v1.5.0 // indirect
	github.com/emirpasic/gods v1.18.1 // indirect
	github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
	github.com/go-git/go-billy/v5 v5.5.0 // indirect
	github.com/go-git/go-git/v5 v5.11.0 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/glog v1.1.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-multierror v1.1.1 // indirect
	github.com/hashicorp/hcl/v2 v2.17.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
	github.com/joho/godotenv v1.5.1 // indirect
	github.com/kevinburke/ssh_config v1.2.0 // indirect
	github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
	github.com/mattn/go-isatty v0.0.19 // indirect
	github.com/mattn/go-localereader v0.0.1 // indirect
	github.com/mattn/go-runewidth v0.0.15 // indirect
	github.com/mitchellh/go-ps v1.0.0 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
	github.com/muesli/cancelreader v0.2.2 // indirect
	github.com/muesli/reflow v0.3.0 // indirect
	github.com/muesli/termenv v0.15.2 // indirect
	github.com/opentracing/basictracer-go v1.1.0 // indirect
	github.com/opentracing/opentracing-go v1.2.0 // indirect
	github.com/pgavlin/fx v0.1.6 // indirect
	github.com/pjbgf/sha1cd v0.3.0 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/pkg/term v1.1.0 // indirect
	github.com/pulumi/appdash v0.0.0-20231130102222-75f619a67231 // indirect
	github.com/pulumi/esc v0.6.2 // indirect
	github.com/pulumi/pulumi-docker/sdk/v3 v3.6.1 // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/rogpeppe/go-internal v1.11.0 // indirect
	github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
	github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/skeema/knownhosts v1.2.1 // indirect
	github.com/spf13/cobra v1.7.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/texttheater/golang-levenshtein v1.0.1 // indirect
	github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
	github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
	github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
	github.com/xanzy/ssh-agent v0.3.3 // indirect
	github.com/zclconf/go-cty v1.13.2 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	golang.org/x/crypto v0.17.0 // indirect
	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
	golang.org/x/mod v0.14.0 // indirect
	golang.org/x/net v0.19.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.15.0 // indirect
	golang.org/x/term v0.15.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	golang.org/x/tools v0.15.0 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
	google.golang.org/grpc v1.57.1 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/warnings.v0 v0.1.2 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	lukechampine.com/frand v1.4.2 // indirect
) If you include this Run , and this will fetch all the packages for you go.mod go tidy Footnotes I fought against permissions for 5 days. The predefined GCP role brought in the additional permissions needed. 1* serviceAccountAdmin

Automate and Deploy a Docker Container to Google Cloud Run from Scratch Using Pulumi and Go

In cybersecurity, it is inevitable to face questions like this. Even if one isn't directly related to a security field but is a developer or system administrator, they have probably pondered a similar question when configuring client-server communications. I am writing this because TLS keeps evolving, gracefully kicking out suites that are no longer considered safe, and I aim to create an article that will save someone time in their investigations. Although investigations can be beneficial for personal growth, I must say. The main question for today's discussion would be: What cipher suites are appropriate to include in TLS configuration, and which ones should be removed to enhance its security? The Audience I am assuming that the reader has intentionally sought out information on TLS cipher suites and found this article. Therefore, I will also assume that the reader is already familiar with concepts related to TLS/SSL, cipher suites, asymmetric and symmetric ciphers, block and stream ciphers, and I won't delve deeply into these topics. The scenario in which this article is helpful at most is when TLS has been configured for some time, and an individual is considering whether they should revise all the ciphers currently in use. The Context TLS is a protocol that defines how communication should occur to achieve a certain level of security. This entails the delivering of packages within the network ensuring that their confidentiality and integrity security aspects were not violated. The versions history can be found on , I will only mention that the product line was: Wikipedia SSL 1.0 > SSL 2.0 > SSL 3.0 > TLS 1.0 > TLS 1.1 > TLS 1.2 > TLS 1.3 and each new version is more secure than the previous one. Typically, one might support older TLS versions for compatibility purposes. However, it is now considered best practice to support versions TLS 1.2 and TLS 1.3 only, and for this article, we will exclusively focus on these two versions. The TLS process begins with the negotiation, also known as the "Handshake." During this phase, various aspects of the communication are defined. Still, our specific interest lies in the following: Client has a set of configured TLS ciphers. Server also has a set of configured TLS ciphers. When there is a match - information sharing between client and server happens. In cases where multiple matches exist, the most secure cipher is selected. Additionally, there is an option to enable support for a custom priority order of cipher suites if needed. If, for some reason, the server and client do not have any matching cipher suites, a connection will not be established. A common example is the communication between a browser and Tomcat web server. Both have configurable predefined cipher suites. If Tomcat's cipher suites have no common ciphers with those set by the browser attempting to establish a connection, it will result in a "Handshake Failed Error." The concept of a secure TLS configuration, among other settings, involves specifying the permitted cipher suites to prevent any connections from being established using outdated or vulnerable cryptographic algorithms. Moreover, once the initial setup is completed and verified, periodic revisions must be scheduled to ensure correspondence with continually improving security standards. Straight to the Facts Each TLS version supports only specific ciphers, and not all of them are considered secure. The general process when configuring TLS involves choosing: The version of TLS. The ciphers of that version will be supported. The table below is intended to provide an overview of TLSv1.2 and TLSv1.3, specifying which ciphers are supported and which among them are recommended for use. The version of the table presented here is a deflated version of the table from author’s Google drive. Please refer to the original one to see deprecated and prohibited ciphers. original It is based on: , August 2008 introduced TLSv1.2 specification (RFC 5246) 37 cipher suites. This number flows over the internet, claiming “TLS1.2 has 37 ciphers”. To be honest I was quite confused because the list of recommended suites for TLS 1.2 that we can find on IANA’s site has some ciphers that are not in this initial 37-items list. Where did they come from? As it turned out, the additional cipher suites beyond the original 37 can be attributed to the release of various RFCs over the years. Some of these RFCs introduced new cipher suites, while others removed older, less secure ones. As a result, the joined list of TLS 1.2 ciphers includes at least 111 cipher suites (if I didn’t miss any RFC, although I feel pretty confident about this number) encompassing all of them since the 2008. The most crucial updates for this article would be the ones that introduced new cipher suites: 8, August 2008 RFC 528 , August 2008 RFC 5289 , March 2009 RFC 5487 , July 2012 RFC 6655 , June 2016 RFC 7905 , August 2016 RFC 7919 , September 2018 RFC 8442 Another important updates are the ones that provide guidance on which ciphers are better to be avoided for security reasons: , February 2015 - prohibits the use of RC4 RFC 7465 - deprecates MD5 and SHA-1 RFC 9155 are not recommended since they do not provide confidentiality and used only for authenticity and integrity purposes null ciphers CBC mode is tend to be , so SSL scanners are likely to flag it as weak vulnerable , August 2018 TLSv1.3 specification (RFC 8446) Unlike its predecessor, TLS 1.3 version has a rather modest number of cipher suites: 5. from September 2023 IANA’s TLS’s Parameters This paper was updated last month, making it quite current. Today's recommendations for the proper configuration of TLS include 24 cipher suites, comprising 20 from TLS 1.2 and 4 from TLS 1.3. TLSv1.2 Recommended Ciphers TLSv1.3 Recommended Ciphers TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_AES_256_GCM_SHA384 TLS_DHE_PSK_WITH_AES_128_GCM_SHA256 TLS_CHACHA20_POLY1305_SHA256 TLS_DHE_PSK_WITH_AES_256_GCM_SHA384 TLS_AES_128_CCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_128_CCM TLS_DHE_RSA_WITH_AES_256_CCM TLS_DHE_PSK_WITH_AES_128_CCM TLS_DHE_PSK_WITH_AES_256_CCM TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 TLS_DHE_PSK_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256 TLS_ECDHE_PSK_WITH_AES_256_GCM_SHA384 TLS_ECDHE_PSK_WITH_AES_128_CCM_SHA256 Useful Tools It is common these days to encounter shorter TLS cipher suite notations compared to the classic format. Here are two examples of the same cipher suite described. Short Notation: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256 Classic Notation: TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256 OpenSSL provides that can assist in revealing the ciphers associated with the short notation. Particularly does the magic: commands openssl ciphers -V ‘CIPHER_SUITE‘ If one doesn’t need to manual analyse existing configuration, they can use Mozilla’s for creating a TLS sample setup. This tool offers a template config that takes into account existing environment and security requirements. They also have a helpful on how to use it effectively. awesome tool guide To dig down into the "Why" and "How" of TLS security, as usually OWASP comes to the rescue. TLS Cheat Sheet Testing for Weak TLS Ciphers To test the cipher suites that the server uses using web: Qualys SSL Server Test using command line: nmap --script ssl-enum-ciphers -p 443 target-ip Before implementing a TLS configuration recommendation, it's worth to verify the publication date of the source, whether it's an article or an RFC. This step helps to ensure that the most up-to-date information is used, which is of particular significance when it comes to cryptographic algorithms. Thanks for reading and happy TLS configuration! P.S. If you kept asking yourself why Chandler, this is because he knows how to do data reconfiguration.

TLS Cipher Suites: Which Ones to Let Go in 2023?

The has given a big boom in the era of the advent of the Internet. With the advent of , people massively using email began to think about greater security on the network and look for alternative solutions. development of email services Web3 Today, in our interview, I will tell you about the team that launched the Web3 DeMail Service from the initial idea to the full implementation of a working solution that will help solve current problems in existing email services and answer why people are moving to Web3. We will talk with the Team Lead and Project Owner, Nikita Tkach, and the main team members. The following interview has been edited for brevity and clarity. Editor’s Note: Let's start with the first question. Can you tell our readers about Qamon and how it differs from traditional email services? Oh, there are a lot of differences. It starts with the fundamental work principle. Qamon is basically a backed-less service. That means that we don’t have any kind of intermediary servers. We use blockchain itself, where one user sends a transaction to another user, where the payload of this transaction is email content itself (or the file).  So, the blockchain serves as the difference: No intermediary servers = 0 risk for privacy. Who knows what happens on the back-end side when the mail travels from one user to another? I think it's self-explanatory here. We use a distributed file system (IPFS) to store files (attachments). It’s censorship-resistant and redundant to attacks. Decentralized Availability: With IPFS, your content will be available through a network of nodes that store copies of that content. If one site becomes unavailable, users can still access content through other sites. This improves the overall availability of content even under failure conditions. No central point of failure: Since IPFS does not depend on a centralized infrastructure, the risk of losing access to your content due to service outages or shutdowns is greatly reduced. Your content will remain available even if specific nodes or a dead service are down. Well, first of all, Qamon doesn’t have any intermediary servers, which classy mails have. Why do we need to migrate from traditional emails? Moving from traditional email to services like Qamon removes the limitations of traditional email systems. Traditional emails are subject to privacy breaches, hacks, and centralized points of failure. Using blockchain technology and a decentralized infrastructure, Qamon brings privacy and security risks to 0, ensures uninterrupted communication with less risk of data leaks associated with centralized servers, and provides a more reliable email ecosystem. Moving from traditional email to services like Qamon removes the limitations of traditional email systems. Traditional emails are subject to privacy breaches, hacks, and centralized points of failure. Using blockchain technology and a decentralized infrastructure, Qamon brings privacy and security risks to 0, ensures uninterrupted communication with less risk of data leaks associated with centralized servers, and provides a more reliable email ecosystem. Could you tell us about your previous experience? What has your team achieved so far? Nice to hear! Having your application on multiple platforms along with a functional web service provides users with many options to access and use your decentralized email service. Offering an application that is compatible with macOS, iOS, iPadOS, and Android and also has a separate fully functional web service, users can interact with your platform on different devices and operating systems. What inspired the team to create a decentralized mail service based on the Everscale blockchain? The motivation for creating Qamon is related to the desire to eliminate the shortcomings of traditional mail services. The use of blockchain and a serverless approach can provide increased privacy, security, and availability for email communication. Decentralization is in line with the broader trend towards that emphasize user empowerment, privacy, and data ownership with the same set of features and conveniences that are available to businesses and customers. Web3 technologies How does Qamon use Everscale's decentralized infrastructure to enhance email security and privacy? The details will depend on the specific implementation of Qamon, but in general, the decentralized nature of the Everscale blockchain can contribute to email security and privacy by: Reduced dependency on centralized servers vulnerable to attack Enable end-to-end encrypted communication between users Zeroing the risk of exposing sensitive data during transmission Using blockchain immutability to maintain a tamper-proof record of transactions. How do you prevent censorship or surveillance in Qamon? Decentralization and detection technology to detect censorship and surveillance in Qamon. The transmission of emails over a distributed network in encrypted form makes it impossible for a single entity to control or censor communication. Encryption and the distributed nature of the system are raised to maximize user privacy, making surveillance difficult. How does Qamon deal with scalability and potential network congestion issues in a decentralized environment? We just use the most scalable blockchains as transport for our emails =). Those are Venom and Everscale. I’m 100% sure that this kind of retail service (which fully relies on the blockchain as a backend). How Qamon protect users from spam, phishing, and other email attacks? In the Qamon ecosystem, there is no shared address database susceptible to spam. Instead, the blockchain acts as a decentralized and immutable ledger that records email transactions. Each email transmission involves a cost, adding a layer of financial disincentive for sending unsolicited or malicious messages. This cost requirement significantly hinders spammers' ability to flood the network with unwanted content. Are there any special encryption or authentication protocols used by Qamon to improve security? Could you tell us more about it? Qamon's sophisticated security framework employs a combination of asymmetric and symmetric encryption (utilizing the chacha20Poly1305 algorithm), blockchain-based storage, recipient key exchange, and cryptographic signatures, establishing a comprehensive email security system that guarantees confidentiality, authenticity, integrity, and non-repudiation, all underpinned by meticulous implementation, vigilant key management, and ongoing threat surveillance. Do you have some integrations with other Web3 Apps/dApps? Of course, in addition to the security measures mentioned earlier, Qamon offers support for the ylide protocol. This protocol currently allows users to receive and decrypt messages if they previously used it as part of our service. Looking ahead, we have ambitious plans for full protocol integration, which will cover both sending messages and expanding support for EVM (Ethereum Virtual Machine) and TVM networks, reflecting our desire to improve the security and ease of communication in decentralized ecosystems. Can you discuss the user interface and accessibility aspects of Qamon? Is it compatible with existing centralized email clients? The user interface and accessibility of Qamon are driven by our design philosophy, prioritizing user-friendly familiarity akin to traditional email clients while seamlessly incorporating blockchain functionalities. In the upcoming quarter, we're set to embark on a journey to bridge the gap between . This will empower users to have human-readable addresses, much like popular email services, enabling them to send messages from centralized platforms and receive them within Qamon, thereby enhancing interoperability and convenience. Web2 and Web3 Are there any plans to expand Qamon's features or integrate with other decentralized platforms in the future? Qamon's future trajectory hinges on our development roadmap and visionary goals. This encompasses potential integration with further decentralized platforms, the expansion of functionalities, and forging collaborations with additional services. Our current integration with the ylide protocol is just the tip of the iceberg; we have even more ambitious plans in this realm, driven by our commitment to pushing the boundaries of secure and decentralized communication. Let's discuss the future. What is your vision for further mail services? Will it also be in demand in the future? Email services, especially those that offer enhanced privacy, security, and data ownership through decentralization, are likely to remain relevant in the future. As users become more conscious of their digital footprint and data privacy, demand for secure and censorship-resistant communication platforms could continue to grow. Will the government also use services like Qamon? What do you think? Undoubtedly, government entities stand to derive substantial benefits from embracing services akin to Qamon. The potential utility of such platforms extends to enhancing the security, efficiency, and reliability of governmental communications. We are, in fact, already in the midst of a pioneering phase, actively subjecting Qamon to rigorous testing within the administrative frameworks of select nations. The outcomes of these ongoing trials stand to offer tangible insights into the viability and adaptability of Qamon within governmental contexts. What else could you tell our readers? I'd like to emphasize that Qamon's unique approach to email services highlights the ongoing innovation in the tech industry. The use of blockchain, decentralization, and cryptography opens up new possibilities for enhancing digital communication security and privacy. As you explore these advancements, remember that the success and adoption of platforms like Qamon depend on a balance between innovation and practicality, ensuring that technology truly meets users' needs while safeguarding their digital lives. About the Qamon Team is an experienced IT professional with seven years of experience in Web3 since 2021. Responsible for the development of the entire project, has four years of experience as a team leader, and since 2023 as a project manager. Nikita's expertise notes the choice of technologies, the structure of the platform, and the development of the main application modules. Nikita Tkach is an experienced IT specialist with ten years of development experience, has a master's degree, and also has experience in managing teams and business processes. Ruslan advised on the development of the Qamon backend (ipfs) and architectural solutions. Ruslan Galimov With six years of experience, is a skilled front-end developer. He's primarily known for his work on the web component of the Qamon application, making user interfaces smooth and intuitive. He's also been involved in integrating blockchain technology, adding advanced security features to Qamon's capabilities. Ivan Vasiliev is an experienced Flutter developer with over four years of experience specializing in blockchain integration. It excels in the development of decentralized applications, providing a user-friendly experience. In addition to Flutter, Vadim also acts as a universal server developer, demonstrating a holistic understanding of the project from idea to implementation. His dedication and diverse skill set make him a valuable asset to our team. Vadim Reznichenko About Qamon DeMail Service is an encrypted, decentralized email service built on the Everscale blockchain. It provides an interface and email experience that is familiar to most users. Qamon Qamon email messages are end-to-end encrypted. That means that no one but the recipient can read the content of sent messages. Each message is encrypted one-way for the end user, and no one else has the key to decrypt it except for the recipient. A cipher is created on the sender’s device, and all messages enter the network already encrypted. Only the receiver can decrypt them. That’s the beauty of cryptography: centralized servers and companies do not have access or any backdoors to content. Qamon’s main mission is to provide strong end-to-end encryption and confidential communication for everyone. It also has several advantages over the existing solutions on the market. Source: https://blog.everscale.network/products/qamon-private-email-service-powered-by-everscale I hope you liked this interview. Write comments, share, and like it! If you have any questions or if you need help — you can write me on or . Twitter Linkedin

Interview: How to Launch a DeMail Service and Ensure Secure Communication for the Future of Email

Delta Lake is an open-source storage framework that is used to build data lakes on top of object storage in a Lakehouse architecture. supports ACID transactions, scalable metadata handling and unified streaming and batch data processing.  Delta Lake is commonly used to provide reliability, consistency, and scalability to Apache Spark applications. Delta Lake runs on the top of the existing data lake storage, such as MinIO, and is compatible with Apache Spark APIs. Delta Lake The original Delta Lake paper ( ) describes how it was built for cloud object storage. When for external tables, they relied on MinIO. HPE Ezmeral Runtime Enterprise customers . MinIO supports Delta Lake’s requirements for durability because MinIO follows strict read-after-write and list-after-write consistency models for all i/o operations both in distributed and standalone modes and is widely acknowledged to run Delta Lake workloads. Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores Vertica tested the use of Delta Lake run Delta Lake on MinIO Many organizations rely on cloud native object stores such as MinIO and AWS S3 to house large structured, semi-structured and unstructured datasets. Each table is stored as a set of objects that are Parquet or ORC, and arranged into partitions. Queries over large files are basically scans that execute quickly. Without Delta Lake, more complex Spark workloads, particularly those that modify, add or remove data, face challenges to performance and correctness under heavy multi-user/multi-app loads. Multi-object updates are not atomic and queries are not isolated, meaning that if a delete is conducted in one query then other concurrent queries will get partial results as the original query updates each object. Rolling back writes is tricky, and a crash in the middle of an update can result in a corrupted table. The real performance killer is metadata – for massive tables with millions of objects that are Parquet files holding billions or trillions of records, metadata operations can bring the applications built on a data lake to a dead stop. Delta Lake was designed to combine the transactional reliability of databases with the horizontal scalability of data lakes. Delta Lake was built to support OLAP-style workloads with an ACID table storage layer over cloud native object stores such as MinIO. As described in the paper , “the core idea of Delta Lake is simple: we maintain information about which objects are part of a Delta table in an ACID manner, using a write-ahead log that is itself stored in the cloud object store.” Objects are encoded in Parquet and can be read by an engine that understands Parquet. Multiple objects can be updated at once “in a serialized manner while still achieving high parallel read and write performance”. The log contains metadata such as min/max statistics for each file, “enabling order of magnitude faster metadata searches” than searching files in the object store directly. Delta lake: high-performance ACID table storage over cloud object stores Delta Lake provides the following: Delta Lake ensures that all changes to data are written to storage and committed for durability while being available to users and apps atomically. There are no partial or corrupted files sitting in your data lake anymore. ACID guarantees: All reads and writes using Spark (or another distributed processing engine) can scale to petabytes. Unlike most other storage formats and query engines, Delta Lake leverages Spark to scale out all the metadata processing and can efficiently handle metadata of billions of files for petabyte-scale tables. Scalable data and metadata handling: The Delta Lake transaction log records details about every modification made to data, including a full audit trail of the changes. Data snapshots enable developers to access and revert to earlier versions of data for audits, rollbacks or for any other reason. Audit history and time travel: Delta Lake automatically prevents the insertion of data that does not match the existing table schema. However, the table schema can be explicitly and safely evolved to accommodate changes to data structure and format. Schema enforcement and schema evolution: Most distributed processing frameworks do not support atomic data modification operations on data lakes. In contrast, Delta Lake supports merge, update and delete operations for complex use cases such as change-data-capture, slowly-changing-dimension operations and streaming upserts. Support for deletes, updates and merges: A Delta Lake table has the ability to work in batch mode and as a streaming source and sink. Delta Lake works across a wide variety of latencies including streaming data ingest and batch historic backfill to provide real-time interactive queries. Streaming jobs write small objects into the table at low latency, later transactionally combining them into larger objects for better performance. Streaming and batch unification: Relying on object storage means that the objects in a Delta table and its log are immutable and can be safely cached locally - wherever across the multicloud locally is. Caching: Lakehouse architecture, Delta Lake in particular, brings key new functionality to data lakes built on object storage. Delta Lake works with a large and of applications and compute engines such as Spark, Starburst, Trino, Flink, and Hive, and also includes APIs for Scala, Java, Rust, Ruby and Python. Built for the cloud, Kubernetes-native MinIO enables performant, resilient and secure data lake applications everywhere - at the edge, in the data center and in the public/private cloud. growing list Delta Lake Files A Delta table is a collection of files that are stored together in a directory (for a file system) or bucket (for MinIO and other object storage). To read and write from object storage, Delta Lake uses the scheme of the path to dynamically identify the storage system and use the corresponding LogStore implementation to provide ACID guarantees. For MinIO, you will use S3A, see . It is critical that the underlying storage system used for Delta Lake is capable of concurrent atomic reads/writes, as is MinIO. Storage configuration — Delta Lake Documentation Creating Delta tables is really writing files to a directory or bucket. Delta tables are created (opened) by writing (reading) a Spark DataFrame and specifying the format and path. In Scala, for example: delta // Create a Delta table on MinIO:
spark.range(5).write.format("delta").save("s3a://<your-minio-bucket>/<path-to-delta-table>")

// Read a Delta table on S3:
spark.read.format("delta").load("s3a://<your-mnio-bucket>/<path-to-delta-table>").show() Delta Lake relies on a bucket per table, and buckets are commonly modeled after file system paths. A Delta Lake table is a bucket that contains data, metadata and a transaction log. The table is stored in Parquet format. Tables can be partitioned into multiple files. MinIO supports S3 LIST to efficiently list objects using file-system-style paths. MinIO also supports byte-range requests in order to more efficiently read a subset of a large Parquet file. MinIO makes an excellent home for Delta Lake tables due to industry-leading performance. MinIO’s combination of scalability and high-performance puts every workload, no matter how demanding, within reach. MinIO is capable of tremendous performance - a achieved 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. MinIO more than delivers the performance needed to power the most demanding workloads on Delta Lake. recent benchmark It’s likely that Delta Lake buckets will contain many Parquet and JSON files, which aligns really well with all of the we’ve built into MinIO for use as a data lake. Small objects are saved inline with metadata, reducing the IOPS needed both to read and write small files like Delta Lake transactions. small file optimizations Most enterprises require multi-cloud functionality for Delta Lake. MinIO includes to synchronize data between locations - on-premise, in the public/private cloud and at the edge. Active-active replication enables enterprises to architect for multi-geo resiliency and fast hot-hot failover. Each bucket, or Delta Lake table, can be configured for replication separately for greatest security and availability. active-active replication ACID Transactions with Delta Lake Adding to data lakes is a pretty big deal because now organizations have greater control over, and therefore greater trust, in the mass of data stored in the data lake. Previously, enterprises that relied on Spark to work with data lakes lacked atomic APIs and ACID transactions, but now Delta Lake makes it possible. Data can be updated after it is captured and written, and with support for ACID, data won’t be lost if the application fails during the operation. Delta Lake accomplishes this by acting as an intermediary between Spark and MinIO for reading and writing data. ACID (Atomicity, Consistency, Isolation and Durability) transactions Central to Delta Lake is the , an ordered record of transactions conducted by users and applications. Every operation (like an UPDATE or an INSERT) performed on a Delta Lake table by a user is an atomic commit composed of multiple actions or jobs. When every action completes successfully, then the commit is recorded as an entry in the DeltaLog. If any job fails, then the commit is not recorded in the DeltaLog. Without atomicity, data could be corrupted in the event of hardware or software failure that resulted in data only being partially written. DeltaLog Delta Lake breaks operations into one or more of the following actions: Add file - adds a file Remove file - removes a file Update metadata  - records changes to the table’s name, schema or partitioning Set transaction - records that a streaming job has committed data Commit info - information about the commit including the operation, user and time Change protocol - updates DeltaLog to the newest software protocol It’s not as complicated as it appears. For example, if a user adds a new column to a table and adds data to it, then Delta Lake breaks that down into its component actions - update metadata to add the column and add file for each new file added - and adds them to the DeltaLog when they complete. Delta Lake relies on optimistic concurrency control to allow multiple readers and writers of a given table to work on the table at the same time. Optimistic concurrency control assumes that changes to a table made by different users can complete without conflicting. As the volume of data grows so does the likelihood that users will be working on different tables. Delta Lake serializes commits and follows a rule of mutual exclusion should two or more commits take place at the same time. In doing so, Delta Lake achieves the isolation required for ACID and the table will look the same after multiple concurrent writes as it would if those writes had occurred serially and separately from each other. When a user runs a new query on an open table that has been modified since the last time it was read, Spark consults the DeltaLog to determine if new transactions have posted to the table and updates the user’s table with those new changes. This ensures that a user’s version of a table is synchronized with the master table in Delta Lake to the most recent operation and that users cannot make conflicting updates to a table. DeltaLog, optimistic concurrency control and schema enforcement (combined with the ability to evolve schema) ensure both atomicity and consistency. Digging into DeltaLog When a user creates a Delta Lake table, that table’s transaction log is automatically created in the subdirectory. As the user modifies the table, each commit is written as a JSON file into the subdirectory in ascending order, i.e. , , and on. _delta_log _delta_log 000000.json 000001.json 000002.json Let’s say we add new records to our table from the data files and . That transaction is added to the DeltaLog and saved as the file . Later, we remove those files and add a new file instead. Those actions are recorded as a new file, . 1.parquet 2.parquet 000000.json 3.parquet 000001.json After and were added, they were removed. The transaction log contains both of the operations even though they negate each other. Delta Lake retains all atomic commits to enable complete audit history and time travel features that show users how a table looked at a specific point in time. Furthermore, the files are not quickly removed from storage until a job is run. MinIO provides another layer of assurance against accidental deletion. 1.parquet 2.parquet VACUUM versioning Durability with Delta Lake and MinIO Delta Lake achieves durability by storing tables and transaction logs on persistent media. Files are never overwritten and must be actively removed. All data changes written to storage are available to users atomically as they occur. Partial and corrupt files become a thing of the past. Delta Lake does not hold tables and logs in RAM for very long and writes them directly to MinIO. As long as commit data was recorded in the DeltaLog and the JSON files were written to the bucket, data is durable in the event of a system or job crash. MinIO guarantees durability after a table and its components are written through multiple mechanisms: splits data files into data and parity blocks and encodes it so that the primary data is recoverable even if part of the encoded data is not available. Horizontally scalable distributed storage systems rely on erasure coding to provide data protection by saving encoded data across multiple drives and nodes. If a drive or node fails or data becomes corrupted, the original data can be reconstructed from the blocks saved on other drives and nodes. Erasure Coding captures and heals corrupted objects on the fly to remove this silent threat to data durability Bit Rot Protection protects data saved to MinIO from deletion or modification using a combination of object locking, retention and other governance mechanisms. Objects written to MinIO are never overwritten. Bucket and Object Immutability further protect objects. MinIO maintains a record of every version of every object, even if it is deleted, enabling you to step back in time (much like Delta Lake’s time travel). Versioning is a key component of that allows administrators to move buckets between tiers, for example to use NVMe for performance intensive workloads, and to set an expiration date on versions so they are purged from the system to improve storage efficiency. Bucket and Object Versioning Data Lifecycle Management MinIO secures Delta Lake tables using encryption and regulates access to them using a combination of IAM and policy based access controls. MinIO data in transit with TLS and data on drives with granular object-level encryption using modern, industry-standard encryption algorithms, such as AES-256-GCM, ChaCha20-Poly1305, and AES-CBC. MinIO integrates with external such as ActiveDirectory/LDAP, Okta and Keycloak for IAM. Users and groups are then subject to AWS IAM-compatible PBAC as they attempt to access Delta Lake tables. encrypts identity providers Delta Lake and MinIO Tutorial This section explains how to quickly start reading and writing Delta tables on MinIO using single-cluster mode. Prerequisites Apache Spark. Download and install MinIO. Record the IP address, TCP port, access key and secret key. Download and install MinIO Client. Download and install The following jar files are required. You can copy the jars in any required location on the Spark machine, for example . /home/spark Hadoop - - Delta Lake needs the class from the package, which implements Hadoop’s API for S3. Make sure the version of this package matches the Hadoop version with which Spark was built. hadoop-aws-2.6.5.jar org.apache.hadoop.fs.s3a.S3AFileSystem hadoop-aws FileSystem AWS - aws-java-sdk-1.7.4.jar Set Up Apache Spark with Delta Lake Start the Spark shell (Scala or Python) with Delta Lake and run code snippets interactively. In Scala: bin/spark-shell --packages io.delta:delta-core_2.12:1.2.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" Configure Delta Lake and AWS S3 on Apache Spark Run the following command to launch a Spark shell with Delta Lake and S3 support for MinIO: bin/spark-shell \
 --packages io.delta:delta-core_2.12:1.2.1,org.apache.hadoop:hadoop-aws:3.3.1 \

 
 --conf spark.hadoop.fs.s3a.access.key=<your-MinIO-access-key> \
 --conf spark.hadoop.fs.s3a.secret.key=<your-MinIO-secret-key>

 --conf "spark.hadoop.fs.s3a.endpoint=<your-MinIO-IP:port> \

 --conf "spark.databricks.delta.retentionDurationCheck.enabled=false" \

 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \

 --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" Create a Bucket in MinIO Use the MinIO Client to create a bucket for Delta Lake: mc alias set minio http://<your-MinIO-IP:port> <your-MinIO-access-key>  <your-MinIO-secret-key>

mc mb minio\delta-lake Create a test Delta Lake table on MinIO Try it out and create a simple Delta Lake table using Scala: // Create a Delta table on MinIO:
spark.range(500).write.format("delta").save("s3a://delta-lake/demo1") You will see something output indicating that Spark wrote the table successfully. Open a browser to log into MinIO at with your access key and secret key. You’ll see the Delta Lake table in the bucket: http://<your-MinIO-IP:9001> MinIO and Delta Lake for High-Performance ACID Transactions on Data Lakes The combination of MinIO and Delta Lake enables enterprises to have a multi-cloud data lake that serves as a consolidated single source of truth. The ability to query and update Delta Lake tables provides enterprises with rich insights into their businesses and customers. Various groups access Delta Lake tables for their own analytics or machine learning initiatives, knowing that their work is secure and the data timely. To go deeper, and see for yourself or spin up a marketplace instance on any public cloud. Do you have questions? Ask away on or via hello@min.io. download MinIO Slack Also published . here

Stories, News & Mentions about CHACHA

Chacha’s Stories on HackerNoon

Chacha's News & Mentions