Becoming a financial advisor requires a combination of education, licenses, certifications and work experience. Some licenses are required in order to sell certain products or to buy and sell securities on behalf of clients. Other education and training will help you in working with clients and earning their trust.

How to become a financial advisor

Among the report’s key findings were: Rising underwriting losses caused P&C insurers’ Q2 after-tax net income to sink to just $0.4 billion, the lowest level since 2011. Net income for the first half of 2023 was $8.9 billion, representing a pre-tax return on revenue of 2.3% and after-tax return on statutory surplus of just 1.8% (annualized)

P&C sector facing significant headwinds – report

JEFFERSONVILLE, Ind. (AP) — JEFFERSONVILLE, Ind. (AP) — First Savings Financial Group Inc. (FSFG) on Tuesday reported a loss of $747,000 in its fiscal fourth quarter. The Jeffersonville, Indiana-based company said it had a loss of 11 cents per share. Earnings, adjusted for non-recurring costs, came to 41 cents per share.

First Savings Financial: Fiscal Q4 Earnings Snapshot

Robert Kiyosaki: 7 Money Secrets They Don’t Teach You in School Learn: What To Do If You Owe Back Taxes to the IRS Indeed, new Experian research, released Oct. 25, found that a whopping 68% of American ... of the first steps to overcome financial stress ...

Roughly Three Quarters of Millennials and Gen Z Report Suffering From ‘Financial Trauma’ — 3 Root Causes and Expert Advice

For the first time in years, banks are offering savings accounts ... vacations and large one-time purchases. Financial experts say American households should aim to amass savings to cover at least three months of income. “You want to have a certain ...

Savings accounts now pay serious interest, but most of us aren't claiming it, survey finds

Banks should want to do more work with tribal communities, said Her Many Horses, simply because it’s good for business: Wells Fargo has banking relationships with one out of three federally recognized tribes in the United States,

Native American populations are ‘invisible’ to banks. That needs to change

When my sister got a large inheritance and told me she wanted to travel, I convinced her to open a Bask Bank Mileage Savings account to earn miles instead of interest.

My Sister’s Money Earns Miles Instead of Interest With This Savings Account

Since coming into office in 2021, the Biden-Harris administration has invested billions toward stimulating the Black economy. The Freedman’s Bank […] The post Biden-Harris Treasury pursues ‘unmet promises’ to freed Black Americans appeared first on TheGrio.

Biden-Harris Treasury pursues ‘unmet promises’ to freed Black Americans

The American dream, once painted with the promise of education and prosperity, is being overshadowed by a looming cloud of student loan debt. This financial burden is not only altering the course of individual lives but is also reshaping family dynamics and the pursuit of financial independence.

The Domino Debt Effect: Crippling Student Loans Are Threatening Family Life and Long-Term Financial Freedom

The banking crisis in March put an unwanted spotlight on the sector, and financial stocks across the board have suffered. The looming recession threat also makes investors hesitant about consumer finance stocks.

One of Warren Buffett's Favorite Stocks Just Fell 4%. Buy the Dip?

First American Financial

servicemac

Using Firebase Authentication with the Latest Next.js Features

সর্বশেষ Next.js বৈশিষ্ট্য সহ Firebase প্রমাণীকরণ ব্যবহার করা

Verwenden der Firebase-Authentifizierung mit den neuesten Next.js-Funktionen

Uso de la autenticación de Firebase con las últimas funciones de Next.js

Utilisation de l'authentification Firebase avec les dernières fonctionnalités Next.js

नवीनतम Next.js सुविधाओं के साथ फायरबेस प्रमाणीकरण का उपयोग करना

Usando o Firebase Authentication com os recursos Next.js mais recentes

Использование аутентификации Firebase с новейшими функциями Next.js

Firebase Kimlik Doğrulamasını En Son Next.js Özellikleriyle Kullanma

Sử dụng Xác thực Firebase với các tính năng Next.js mới nhất

将 Firebase 身份验证与最新的 Next.js 功能结合使用

How to Master .NET 8 Web API: From Setup to Security - The Best 50 Tips

কিভাবে .NET 8 ওয়েব এপিআই আয়ত্ত করবেন: সেটআপ থেকে নিরাপত্তা - সেরা 50 টি টিপস

.NET 8 Web API meistern: Von der Einrichtung bis zur Sicherheit – Die 50 besten Tipps

Cómo dominar la API web .NET 8: de la configuración a la seguridad: los 50 mejores consejos

Comment maîtriser l'API Web .NET 8 : de la configuration à la sécurité - Les 50 meilleurs conseils

.NET 8 वेब एपीआई में महारत कैसे हासिल करें: सेटअप से सुरक्षा तक - सर्वोत्तम 50 युक्तियाँ

.NET 8 Web API をマスターする方法: セットアップからセキュリティまで - ベスト 50 のヒント

.NET 8 Web API를 마스터하는 방법: 설정부터 보안까지 - 최고의 50가지 팁

Como dominar a API da Web .NET 8: da configuração à segurança - as 50 melhores dicas

Как освоить веб-API .NET 8: от настройки до безопасности — 50 лучших советов

.NET 8 Web API'sinde Uzmanlaşma: Kurulumdan Güvenliğe - En İyi 50 İpucu

Cách làm chủ API Web .NET 8: Từ thiết lập đến bảo mật - 50 mẹo hay nhất

如何掌握 .NET 8 Web API：从设置到安全 - 50 个最佳技巧

This tutorial shows you how to deploy and automate a Docker container using Pulumi in Google Cloud Platform's Cloud Run with minimum permissions. Here's the GitHub Repository for reference: https://github.com/superjose/deploy-to-cloud-run-go Get the CLIs: . Create a Google Cloud Account Install Google Cloud CLI Install Pulumi CLI - We need it to build the Docker container. Install Docker Bootstrap the project: Create a directory: pulumi mkdir pulumi Run to initialize a pulumi project with Go (It can be your language of choice pulumi new go Navigate to the directory. pulumi cd pulumi Navigate to and create a new project. cloud.google.com Take note of the (Usually the project's name). project-id Generate the necessary permissions using gcloud CLI Login with the Google Auth CLI: gcloud auth login (Optional) Set the project in google cloud CLI (Can be changed anytime). This saves you from passing into every command. --project [PROJECT-ID] gcloud If your machine has multiple GCP projects, skip this step and pass the flag into every command. --project gcloud Create a service account (The account that Pulumi will connect to): gcloud iam service-accounts create pulumi-gcp --description="Pulumi GCP" Download the credentials for the service accounts and store them locally (Remember to replace with your GCP Project Id): [PROJECT-ID] gcloud iam service-accounts keys create ~/keys/gcp/pulumi-service-account-key-file.json --iam-account=pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Set Pulumi's gcp credentials config path:
(This will connect the service account with Pulumi) pulumi config set gcp:credentials ~/keys/gcp/pulumi-service-account-key-file.json Set the GCP Project by doing: pulumi config set gcp:project [PROJECT-ID] Create a file (Inside the dir) and add the required permissions in : roles.gcp.yml pulumi includedPermissions # https://cloud.google.com/iam/docs/creating-custom-roles#creating
# Yaml to define the Pulumi GCP Roles that need to be created with gcloud CLI
title: Pulumi GCP Roles
description: |
  This policy ensures that all GCP roles are created using Pulumi.
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
  - serviceusage.services.list
  - serviceusage.services.enable
  - serviceusage.services.disable
  - serviceusage.services.get
  - serviceusage.services.use
    # Permissions for GCR
  - storage.objects.create
  - storage.objects.delete # Optional: only include if you need to delete images
  - storage.objects.get
  # Permissions for Google Artifact Registry
  - artifactregistry.repositories.create
  - artifactregistry.repositories.delete
  - artifactregistry.repositories.get
  - artifactregistry.repositories.list
  - artifactregistry.repositories.update
  - artifactregistry.repositories.downloadArtifacts
  - artifactregistry.repositories.uploadArtifacts
  - artifactregistry.repositories.deleteArtifacts
  # Permissions
  # Permissions
  - run.services.create
  - run.services.get
  - run.services.list
  - run.services.update
  - run.services.delete
  - run.services.getIamPolicy
  - run.services.setIamPolicy
  - iam.serviceAccounts.actAs
# NOTE: This should be removed the first time you're creating a role.
# This etag is to update the current active role (As GCP lets you manage multiple roles)
# I'm commenting it out so I can always replace the role
# etag: BwYS74Xx5y4= Create the with the file above:
(We assume we're running this code from the directory) pulumi_admin_role pulumi gcloud iam roles create pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' In case you need to make edits, change the file and use: gcloud iam roles update pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' We're also adding the role (I haven't found a better way) (Otherwise we'd get 403 errors when refreshing and updating in Pulumi)<sup>1</sup> serviceAccountAdmin gcloud projects add-iam-policy-binding [PROJECT-ID] --role roles/iam.serviceAccountAdmin   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com (The following will add the new role to the service account) gcloud projects add-iam-policy-binding [PROJECT-ID] --role projects/[PROJECT-ID]/roles/pulumi_admin_role   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Our in the directory:
(Check the code for comments!) main.go pulumi We enable the required services (Artifact Registry, and Cloud Run). Artifact Registry is used to host the docker container image. Cloud Runner will launch the Docker image from Artifact Registry. We build the docker image locally (We specify the platform in case you're using an ARM chip like M1, M2, Snapdragon SQ, X Elite, etc.) We create a chain of "DependsOn" to notify Pulumi:
5.1 Services need to be enabled first
5.2 We create the Artifact Repository
5.3 We build the docker image and push it to Artifact Registry.
5.4 We pull the Docker Image from Artifact Registry and run it.
5.5 We add IAM permissions so it can be accessed from anywhere. package main

import (
	"errors"
	"log"
	"os"

	"github.com/joho/godotenv"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/artifactregistry"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/cloudrun"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/projects"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

// This is the name you created in Google Cloud Platform (GCP).
const gcpProjectId = "deploy-to-cloud-run-go"

// The Docker Image Name
const dockerImageName = "my-app-docker"
const artifactRegistryServiceName = "artifact-registry-api"
const artifactRegistryRepoName = "my-app-artifact-repo"
const artifactRegistryRepoLocation = "us-east1"
const cloudRunAdminServiceName = "cloud-run-admin-service"
const cloudRunServiceName = "cloud-run-service"

// For more info: https://cloud.google.com/run/docs/locations
const cloudRunLocation = "us-east1"

// The tag for the Docker image
const imageTag = "latest"

// This is a url like: us-east1-docker.pkg.dev
// It is used to push the Docker image to Google Container Registry
// For more info: https://cloud.google.com/container-registry/docs/pushing-and-pulling
// The format is: <region>-docker.pkg.dev
var dockerGCPServer = cloudRunLocation + "-docker.pkg.dev"

// The full path to the Docker image
// It is used to deploy the Docker image to Google Cloud Run
// The format is: <region>-docker.pkg.dev/<project-id>/<repo-name>/<image-name>:<tag>
// For more info: https://cloud.google.com/run/docs/deploying
// Example: us-east1-docker.pkg.dev/deploy-to-cloud-run-go/my-app--artifact-repo/my-app-docker:latest
var dockerImageWithPath = dockerGCPServer + "/" + gcpProjectId + "/" + artifactRegistryRepoName + "/" + dockerImageName + ":" + imageTag

func main() {

	// Load the .env file
	err := godotenv.Load()

	if err != nil {
		log.Fatal("Error loading .env file")
	}

	pulumi.Run(func(ctx *pulumi.Context) error {

		enabledServices, serviceResultErr := enableServices(ctx)

		if serviceResultErr != nil {
			return serviceResultErr
		}

		artifactRegistryRepo, createArtifactErr := createArtifactRegistryNewRepository(ctx, &enabledServices)
		if createArtifactErr != nil {
			return createArtifactErr
		}

		dockerImage, buildAndPushErr := buildAndPushToContainerRegistry(ctx, &enabledServices, artifactRegistryRepo)

		if buildAndPushErr != nil {
			return buildAndPushErr
		}

		deployContainerErr := deployContainerToCloudRun(ctx, &enabledServices, dockerImage)

		if deployContainerErr != nil {
			return deployContainerErr
		}

		return nil
	})
}

type EnabledServices struct {
	CloudRunService         *projects.Service `pulumi:"cloudRunService"`
	ArtifactRegistryService *projects.Service `pulumi:"artifactRegistryService"`
}

func enableServices(ctx *pulumi.Context) (EnabledServices, error) {

	cloudRunService, cloudRunAdminErr := projects.NewService(ctx, cloudRunAdminServiceName, &projects.ServiceArgs{
		Service: pulumi.String("run.googleapis.com"),
		Project: pulumi.String(gcpProjectId),
	})

	if cloudRunAdminErr != nil {
		return EnabledServices{}, cloudRunAdminErr
	}

	artifactRegistryService, err := projects.NewService(ctx, artifactRegistryServiceName, &projects.ServiceArgs{
		Service: pulumi.String("artifactregistry.googleapis.com"),
	})

	if err != nil {
		return EnabledServices{}, err
	}
	return EnabledServices{
		CloudRunService:         cloudRunService,
		ArtifactRegistryService: artifactRegistryService,
	}, nil
}

func createArtifactRegistryNewRepository(ctx *pulumi.Context, enabledServices *EnabledServices) (*artifactregistry.Repository, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	dependingResources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
	}

	repo, err := artifactregistry.NewRepository(ctx, artifactRegistryRepoName, &artifactregistry.RepositoryArgs{
		Location:     pulumi.String(artifactRegistryRepoLocation),
		RepositoryId: pulumi.String(artifactRegistryRepoName),
		Format:       pulumi.String("DOCKER"),
		Description:  pulumi.String("The repository that will hold social-log Docker images."),
	}, pulumi.DependsOn(dependingResources))

	if err != nil {
		return nil, err
	}

	return repo, nil
}

func buildAndPushToContainerRegistry(ctx *pulumi.Context, enabledServices *EnabledServices, artifactRegistryRepo *artifactregistry.Repository) (*docker.Image, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	if artifactRegistryRepo == nil {
		return nil, errors.New("artifactRegistryRepo cannot be nil")
	}

	// Lookup GOOGLE_CREDENTIALS environment variable which should hold the path to the JSON key file
	jsonKeyPath, present := os.LookupEnv("GOOGLE_CREDENTIALS_FILE_PATH")
	if !present {
		return nil, errors.New("GOOGLE_CREDENTIALS_FILE_PATH environment variable is not set")
	}

	// Read the JSON key file
	jsonKey, err := os.ReadFile(jsonKeyPath)
	if err != nil {
		return nil, err
	}

	dependingSources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
		artifactRegistryRepo,
	}

	// Build and push Docker image to Google Container Registry using the JSON key
	image, err := docker.NewImage(ctx, dockerImageName, &docker.ImageArgs{
		Build: &docker.DockerBuildArgs{
			Context: pulumi.String("../"), // Adjust the context according to your project structure

			ExtraOptions: pulumi.StringArray{
				// This option is needed for devices running on ARM architecture, such as Apple M1/M2/MX CPUs
				pulumi.String("--platform=linux/amd64"),
			},
		},
		ImageName: pulumi.String(dockerImageWithPath),
		Registry: &docker.ImageRegistryArgs{
			Server:   pulumi.String(dockerGCPServer),
			Username: pulumi.String("_json_key"),     // Special username for GCP
			Password: pulumi.String(string(jsonKey)), // Provide the contents of the key file
		},
	}, pulumi.DependsOn(dependingSources))
	if err != nil {
		return nil, err
	}

	return image, nil
}

func deployContainerToCloudRun(ctx *pulumi.Context, enabledServices *EnabledServices, dockerImage *docker.Image) error {

	if enabledServices == nil || enabledServices.CloudRunService == nil {
		return errors.New("enabledServices cannot be nil")
	}

	if dockerImage == nil {
		return errors.New("dockerImage cannot be nil")
	}

	dependingSources := []pulumi.Resource{
		enabledServices.CloudRunService,
		dockerImage,
	}

	appService, err := cloudrun.NewService(ctx, cloudRunServiceName, &cloudrun.ServiceArgs{
		Location: pulumi.String(cloudRunLocation), // Choose the appropriate region for your service
		Template: &cloudrun.ServiceTemplateArgs{
			Spec: &cloudrun.ServiceTemplateSpecArgs{
				Containers: cloudrun.ServiceTemplateSpecContainerArray{
					&cloudrun.ServiceTemplateSpecContainerArgs{
						Image: dockerImage.ImageName,
						Resources: &cloudrun.ServiceTemplateSpecContainerResourcesArgs{
							Limits: pulumi.StringMap{
								"memory": pulumi.String("256Mi"), // Adjust the memory limit as needed
							},
						},
					},
				},
			},
		},
		Traffics: cloudrun.ServiceTrafficArray{
			&cloudrun.ServiceTrafficArgs{
				Percent:        pulumi.Int(100),
				LatestRevision: pulumi.Bool(true),
			},
		},
	}, pulumi.DependsOn(dependingSources))

	if err != nil {
		return err
	}

	_, iamErr := cloudrun.NewIamMember(ctx, "invoker", &cloudrun.IamMemberArgs{
		Service:  appService.Name,
		Location: appService.Location,
		Role:     pulumi.String("roles/run.invoker"),
		Member:   pulumi.String("allUsers"),
	})

	if iamErr != nil {
		return iamErr
	}

	ctx.Export("containerUrl", appService.Statuses.Index(pulumi.Int(0)).Url().ToOutput(ctx.Context()))
	return nil
} Update the in your Dockerfile:
There's a , and in which you need to export your environment variable to ENV known issue here HOME /root # Set the ENV HOME before your ENTRYPOINT.
ENV HOME=/root

# This is specific to your project. 
ENTRYPOINT ["/whatever-is-your-entrypoint"] Create a .env file in the pulumi directory: Set the * to the one you saved on full path 11) GOOGLE_CREDENTIALS_FILE_PATH="/Users/myusername/keys/gcp/pulumi-service-account-key-file.json" Run pulumi up You should be up and running! Extras The go.mod module social-log-go

go 1.21

toolchain go1.22.0

require (
	github.com/pulumi/pulumi-gcp/sdk/v7 v7.11.2
	github.com/pulumi/pulumi/sdk/v3 v3.108.1
)

require (
	dario.cat/mergo v1.0.0 // indirect
	github.com/Masterminds/semver v1.5.0 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
	github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
	github.com/agext/levenshtein v1.2.3 // indirect
	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
	github.com/atotto/clipboard v0.1.4 // indirect
	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
	github.com/blang/semver v3.5.1+incompatible // indirect
	github.com/charmbracelet/bubbles v0.16.1 // indirect
	github.com/charmbracelet/bubbletea v0.24.2 // indirect
	github.com/charmbracelet/lipgloss v0.7.1 // indirect
	github.com/cheggaaa/pb v1.0.29 // indirect
	github.com/cloudflare/circl v1.3.7 // indirect
	github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
	github.com/djherbis/times v1.5.0 // indirect
	github.com/emirpasic/gods v1.18.1 // indirect
	github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
	github.com/go-git/go-billy/v5 v5.5.0 // indirect
	github.com/go-git/go-git/v5 v5.11.0 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/glog v1.1.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-multierror v1.1.1 // indirect
	github.com/hashicorp/hcl/v2 v2.17.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
	github.com/joho/godotenv v1.5.1 // indirect
	github.com/kevinburke/ssh_config v1.2.0 // indirect
	github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
	github.com/mattn/go-isatty v0.0.19 // indirect
	github.com/mattn/go-localereader v0.0.1 // indirect
	github.com/mattn/go-runewidth v0.0.15 // indirect
	github.com/mitchellh/go-ps v1.0.0 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
	github.com/muesli/cancelreader v0.2.2 // indirect
	github.com/muesli/reflow v0.3.0 // indirect
	github.com/muesli/termenv v0.15.2 // indirect
	github.com/opentracing/basictracer-go v1.1.0 // indirect
	github.com/opentracing/opentracing-go v1.2.0 // indirect
	github.com/pgavlin/fx v0.1.6 // indirect
	github.com/pjbgf/sha1cd v0.3.0 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/pkg/term v1.1.0 // indirect
	github.com/pulumi/appdash v0.0.0-20231130102222-75f619a67231 // indirect
	github.com/pulumi/esc v0.6.2 // indirect
	github.com/pulumi/pulumi-docker/sdk/v3 v3.6.1 // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/rogpeppe/go-internal v1.11.0 // indirect
	github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
	github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/skeema/knownhosts v1.2.1 // indirect
	github.com/spf13/cobra v1.7.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/texttheater/golang-levenshtein v1.0.1 // indirect
	github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
	github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
	github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
	github.com/xanzy/ssh-agent v0.3.3 // indirect
	github.com/zclconf/go-cty v1.13.2 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	golang.org/x/crypto v0.17.0 // indirect
	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
	golang.org/x/mod v0.14.0 // indirect
	golang.org/x/net v0.19.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.15.0 // indirect
	golang.org/x/term v0.15.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	golang.org/x/tools v0.15.0 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
	google.golang.org/grpc v1.57.1 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/warnings.v0 v0.1.2 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	lukechampine.com/frand v1.4.2 // indirect
) If you include this Run , and this will fetch all the packages for you go.mod go tidy Footnotes I fought against permissions for 5 days. The predefined GCP role brought in the additional permissions needed. 1* serviceAccountAdmin

Automate and Deploy a Docker Container to Google Cloud Run from Scratch Using Pulumi and Go

Often, when we build enterprise software solutions meant for independent deployment across different customer environments, we leverage a central image repository. This repository is exposed to the enterprise, who then installs it in their environment. These applications may require variables or files specific to the client's environment, which need to be injected into the image for it to run. As it's impractical to have a separate image for each customer, we use a single image that is customized and injected with required files or variables during deployment. This blog explores how to create a standardized image, inject necessary variables in the form of a file, and deploy it on a cluster. For this we will create a python app, dockerize it, build it and push it to a repository. Once pushed we will be creating standard k8 manifests and Helm charts where we will be injecting our file/variables into it. Kubernetes Application and Image Creation Let's create a simple Python program which retrieves the variables "DB_HOST" and "DB_PASSWORD", then uses them to print both of these variables to the console. import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

db_host = os.getenv("DB_HOST")
db_password = os.getenv("DB_PASSWORD")

print(db_host)
print(db_password) Next, we'll create a Docker image that packages our application using python:3.9-slim and adds all the required packages to the image. The entry point for the image is the above Python file named . Once completed, we'll push our image to an image repository such as ECR or Docker Hub. main.py # Use an official Python runtime as the base image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed dependencies
RUN pip install --no-cache-dir python-dotenv

# Run the Python script when the container launches
CMD ["python", "main.py"] Creating Deployment Files Once we have the image ready in our preferred image repository, we need to deploy these images on a server. Nowadays, most deployments have moved to Kubernetes, so we'll use it as our deployment platform. Deployment in Kubernetes can be achieved by creating Kubernetes manifests directly or creating HELM charts (or Kustomize), which act like package managers to deploy applications. Let's explore both options: Creating standard manifest files Below, we're creating a deployment of one replica, a service of type NodePort to expose the application, and a secret object that takes the content of our .env file and injects it into the pod when we deploy the application. apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: your-container-name
        image: your-image-name:your-image-tag
        ports:
        - containerPort: 80
        volumeMounts:
        - name: env-volume
          mountPath: /app/.env
          subPath: .env
      volumes:
      - name: env-volume
        secret:
          secretName: python-app-env-secret
---

apiVersion: v1
kind: Service
metadata:
  name: python-app-service
spec:
  selector:
    app: python-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30001
  type: NodePort

---

apiVersion: v1
kind: Secret
metadata:
  name: python-app-env-secret
type: Opaque
data:
  .env: |
    DB_HOST=test
    DB_PASSWORD=password Creating HELM chart To create a helm chart, simply run . For our project here, we'll run . This should create a Python-app folder, which contains the standard HELM template that can be modified according to our requirements. The generated folder should have the following folder/file structure: helm create <app-name> helm create python-app my-chart/
├── charts/
├── templates/
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── service.yaml
│   └── serviceaccount.yaml
│   ├── hpa.yaml
│   ├── _helpers.tpl
├── values.yaml
└── Chart.yaml We need to create a configmap.yaml file in the templates folder and add the following content to it: # templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  .env: |
    DB_HOST={{ .Values.env.DB_HOST }}
    DB_PASSWORD={{ .Values.env.DB_PASSWORD }} The deployments.yaml should look something like this: # templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "python-app.name" . }}
  labels:
    app: {{ include "python-app.name" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ include "python-app.name" . }}
  template:
    metadata:
      labels:
        app: {{ include "python-app.name" . }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: cm-volume
          mountPath: /app/.env
          subPath: .env
      volumes:
      - name: cm-volume
        configMap:
          name: {{ include "python-app.name" . }} The service.yaml: # templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "python-app.name" . }}
spec:
  selector:
    app: {{ include "python-app.name" . }}
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30001
  type: NodePort The _helpers.tpl should have the following content: # templates/_helpers.tpl.yaml
{{- define "python-app.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name }}
{{- end }}{{- define "python-app.name" -}}
{{- printf "%s" .Chart.Name }}
{{- end }} In values.yaml, pass your secrets and image repository and tag: # values.yaml
image:
  repository: krishnadutt/my-python-app
  tag: latest
replicaCount: 1
env:
  DB_HOST: test
  DB_PASSWORD: your_database_password For this project, delete the rest of the files in the templates folder. Once done, apply all the manifest files or simply add the helm repository and install it in your k8s. Once deployed, if we execute into the image in Kubernetes, we should see the file and if we search the logs, we should find that .env values are used and those values are printed out by the program. Like my content? Feel free to reach out to my for interesting content and productive discussions. LinkedIn

Enhancing Dockerized Application Deployment: File Passing Strategies with Helm Charts

Overview Helm is the package manager for Kubernetes, like apt, yum, or Homebrew for traditional operating systems. It simplifies the deployment and management of applications on Kubernetes clusters by packaging all necessary components into a single, manageable unit. Why Helm? : Helm packages, known as charts, bundle together all the Kubernetes manifests (like deployments, services, etc.) needed to deploy an application, making the deployment process straightforward and repeatable. Simplifies Kubernetes Deployments : Helm charts can include other charts, allowing complex applications to manage their dependencies effectively. Manages Dependencies : Helm tracks versions of deployments, enabling rollbacks to previous states if needed. Version Control for Deployments What Is It Made of? : A Helm chart is a bundle containing all information necessary to create an instance of a Kubernetes application. It includes resource definitions and configuration files. Chart : The configuration contains customizable parameters that merge with a chart to create a release. These parameters allow the same chart to be used in different environments or configurations. Config : A release is an instance of a chart running in a Kubernetes cluster combined with a specific configuration. Release Helm Architecture Diagram : Helm Client : Local chart development and managing repositories. Chart Development : Manages releases/repos and requests chart installation or release upgrade/uninstallation through Helm Library. Release Management : Helm Library : Handles combining charts and configurations to create releases and installing/upgrading/uninstalling charts. Chart Management : Interfaces with the Kubernetes API server. Interacts with Kubernetes Creating a Chart Creating a Helm chart involves setting up a predefined directory structure with at least two files: for metadata about the chart and for default configuration values. Chart.yaml values.yaml : A basic chart directory will have the following layout: Structure : Contains metadata about the chart such as name, version, and description. Chart.yaml : Specifies default configuration values for the chart. values.yaml : This directory contains template files that generate Kubernetes manifest files based on the values provided. templates/ : Helm uses a templating engine to substitute values in the chart templates, creating Kubernetes manifests tailored to specific deployments. This allows for dynamic adjustment of resources, labels, and configurations without altering the original chart files. Templating Practice : Scenario A startup, "DevOps Solutions" adopts Helm to streamline their Kubernetes deployments. You're a consultant tasked with creating a basic Helm Chart for . It should be customizable for different environments using values. n8n Clone the repo. git clone https://github.com/perplexedyawdie/helm-learn.git Change directory to creating-charts cd helm-learn/creating-charts Spin up the environment using docker-compose. docker compose up -d --build SSH into the container. ubuntu ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes root@localhost -p 2222
# password: test123 Generate a chart directory along with sample files. helm create my-n8n View the structure of the generated directory. tree .

# Output should look similar to this
#`-- my-n8n
#    |-- Chart.yaml
#    |-- charts
#    |-- templates
#    |   |-- NOTES.txt
#    |   |-- _helpers.tpl
#    |   |-- deployment.yaml
#    |   |-- hpa.yaml
#    |   |-- ingress.yaml
#    |   |-- service.yaml
#    |   |-- serviceaccount.yaml
#    |   `-- tests
#    |       `-- test-connection.yaml
#    `-- values.yaml We'll only be using , , and so we can remove the rest since we'll be adding our own manifest files. Chart.yaml values.yaml my-n8n/templates/NOTES.txt my-n8n/templates/_helpers.tpl rm -rf my-n8n/templates/*.yaml my-n8n/templates/tests Update the appVersion in Chart.yaml to "1.27.2" & clear the contents of my-n8n/templates/NOTES.txt file then update with the following. nano my-n8n/Chart.yaml
# appVersion: "1.27.2"

nano my-n8n/templates/NOTES.txt
# Welcome to n8n.
# Wait a few minutes until the status changes to RUNNING.
# After it is ready, access it from: http://localhost:2223 Update the file with the following data. values.yaml # Default values for my-n8n.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: n8nio/n8n
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "1.27.2"

containerPorts:
 - name: http
   port: 5678

service:
  type: NodePort
  port: 30200

livenessProbe:
  httpGet:
    path: /
    port: http
readinessProbe:
  httpGet:
    path: /
    port: http

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

# Additional volumes on the output Deployment definition.
volumes:
- name: n8n-data

persistence:
   claimName: n8n-data

# Additional volumeMounts on the output Deployment definition.
volumeMounts:
 - name: n8n-data
   mountPath: "/home/node/.n8n" Create the , and, files inside the folder. deployment.yaml service.yaml pvc.yaml my-n8n/templates # my-n8n/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: n8n-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi # Adjust the size as per your requirement # my-n8n/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n
  template:
    metadata:
      labels:
        app: n8n
    spec:
      containers:
      - name: n8n
        image: n8nio/n8n
        ports:
        - name: http        
        - containerPort: 5678
        volumeMounts:
        - name: n8n-data
          mountPath: /home/node/.n8n
      volumes:
      - name: n8n-data
        persistentVolumeClaim:
          claimName: n8n-data # my-n8n/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: n8n
spec:
  type: NodePort
  ports:
  - port: 5678
    targetPort: 30200
    protocol: TCP
  selector:
    app: n8n Now, let's parameterize these using the go-template language to convert them to templates. Helm will fetch the appropriate values from the file to generate the manifest that will be deployed on K8S. values.yaml # my-n8n/templates/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Values.persistence.claimName }}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi # Adjust the size as per your requirement # my-n8n/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ printf "%s-%s" .Release.Name .Chart.Name  }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n
  template:
    metadata:
      labels:
        app: n8n
    spec:
      containers:
      - name: n8n
        image: {{ printf "%s:%s" .Values.image.repository .Values.image.tag  }}
        ports:
        {{- range .Values.containerPorts }}        
        - name: {{ .name }}
          containerPort: {{ .port }}
        {{- end }}        
        volumeMounts:
        {{- range .Values.volumeMounts }}
        - name: {{ .name }}
          mountPath: {{ .mountPath }}
        {{- end }}
      volumes:
      {{- range .Values.volumes }}      
      - name: {{ .name }}
        persistentVolumeClaim:
          claimName: {{ $.Values.persistence.claimName }}
      {{- end }} # my-n8n/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ printf "%s-%s" .Release.Name .Chart.Name  }}
spec:
  type: {{ .Values.service.type }}
  ports:
  - port: 5678
    targetPort: 5678
    nodePort: {{ .Values.service.port }}    
    protocol: TCP
  selector:
    app: n8n Ensure that you're in the directory that contains the chart then run the linter. my-n8n helm lint my-n8n Install the chart. helm install my-n8n ./my-n8n Wait until the deployment is complete, you can continually check using the following command. kubectl get all After the status has changed to running, you should be able to access it in your browser from: http://localhost:2223 Recap Great effort! You just learned how to create and configure a Helm chart and then deploy it to a Kubernetes cluster!

Helm 101: How to Create Helm Charts

Efficient Microservices: Harnessing the Power of Data Compression in Direct Service-to-Service.

Greetings Comrades! One of the strangest logging setups I ever had - AWS ALB controller logs. Really, look at the scheme: 1. S3 Bucket for ALB Logs I assume that you already have an ALB Controller in your EKS Cluster. So, we should create an S3 bucket and enable the SQS trigger with Terraform: ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} 2. S3 Trigger to SQS Queue ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} 3.  Installing and Configuring Vector I always use ArgoCD for installing something in the Kubernetes cluster, so here is a manifest: apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb 4. Enabling Our Logs There are three ways you can enable logging: Old-fashioned documentation from AWS - link With Terraform aws_lb resource - link And the easiest way is - Kubernetes Ingress annotation (our way): alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs Now, you should be able to see logs in your OpenSearch console! That`s all! Greetings Comrades! One of the strangest logging setups I ever had - AWS ALB controller logs. Really, look at the scheme: 1. S3 Bucket for ALB Logs I assume that you already have an ALB Controller in your EKS Cluster. So, we should create an S3 bucket and enable the SQS trigger with Terraform: ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} ###
# ALB Controller Logs S3 Bucket
###

resource "aws_s3_bucket" "alb_logs" {
  bucket = "alb-logs"
}

resource "aws_s3_bucket_lifecycle_configuration" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  rule {
    id = "log"

    expiration {
      days = 3
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_policy" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id
  policy = data.aws_iam_policy_document.alb_logs_bucket_policy.json
}

data "aws_iam_policy_document" "alb_logs_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::127311923021:root"] <--- it should be like this!!!
    }
    actions   = ["s3:*"]   <--- I`m so lazy
    resources = ["${aws_s3_bucket.alb_logs.arn}/*", ]
  }
} 2. S3 Trigger to SQS Queue ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} ###
# ALB Logs SQS Queue
###

resource "aws_sqs_queue" "alb_logs" {
  name                    = "alb-logs"
  sqs_managed_sse_enabled = false
}

data "aws_iam_policy_document" "alb_logs_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"   <--- Wow again!!!
      identifiers = ["*"]   <--- And again!!!
    }

    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.alb_logs.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_s3_bucket.alb_logs.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "alb_logs" {
  queue_url = aws_sqs_queue.alb_logs.id
  policy    = data.aws_iam_policy_document.alb_logs_policy.json
}

resource "aws_s3_bucket_notification" "alb_logs" {
  bucket = aws_s3_bucket.alb_logs.id

  queue {
    id        = "alb-logs"
    queue_arn = aws_sqs_queue.alb_logs.arn
    events    = ["s3:ObjectCreated:*"]
  }
} 3.  Installing and Configuring Vector I always use ArgoCD for installing something in the Kubernetes cluster, so here is a manifest: apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: logging
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://helm.vector.dev
    chart: vector
    targetRevision: 0.25.0
    helm:
      values: |
        service:
          enabled: false
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ID:role/eks-alb-ingress-controller
        customConfig:
          transforms:
            alb:
              type: remap
              inputs:
                - s3
              source: >-
                structured =
                  parse_regex!(.message, r'^(?P<Type>[^ ]*) (?P<Time>[^ ]*) (?P<Alb>[^ ]*) (?P<ClientIP>[^ ]*):(?P<ClientPort>[0-9]*) (?P<TargetIP>[^ ]*)[:-](?P<TargetPort>[0-9]*) (?P<RequestProcessingTime>[-.0-9]*) (?P<TargetProcessingTime>[-.0-9]*) (?P<ResponseProcessingTime>[-.0-9]*) (?P<StatusCode>|[-0-9]*) (?P<TargetStatusCode>-|[-0-9]*) (?P<ReceivedBytes>[-0-9]*) (?P<SentBytes>[-0-9]*) \"(?P<RequestVerb>[^ ]*) (?P<RequestUrl>.*) (?P<RequestProto>- |[^ ]*)\" \"(?P<UserAgent>[^\"]*)\" (?P<SslCipher>[A-Z0-9-_]+) (?P<SslProtocol>[A-Za-z0-9.-]*) (?P<TargetGroupArn>[^ ]*) \"(?P<TraceId>[^\"]*)\" \"(?P<DomainName>[^\"]*)\" \"(?P<CertArn>[^\"]*)\" (?P<MatchedRulePriority>[-.0-9]*) (?P<RequestCreationTime>[^ ]*) \"(?P<ActionExecuted>[^\"]*)\" \"(?P<RedirectUrl>[^\"]*)\" \"(?P<LambdaErrorReason>[^ ]*)\" \"(?P<TargetPortList>[^\s]+?)\" \"(?P<TargetStatusCodeList>[^\s]+)\" \"(?P<Classification>[^ ]*)\" \"(?P<Classificationreason>[^ ]*)\"')
                . = merge(., structured)

                .ReceivedBytes = to_int!(.ReceivedBytes)
                .RequestProcessingTime = to_float!(.RequestProcessingTime)
                .ResponseProcessingTime = to_float!(.ResponseProcessingTime)
                .StatusCode = to_int!(.StatusCode)
                .SentBytes = to_int!(.SentBytes)
                .TargetProcessingTime = to_float!(.TargetProcessingTime)

                parsed_url, err = parse_url(.RequestUrl)
                if err == null {
                  .Path = parsed_url.path
                  .RequestDomainName = parsed_url.host
                }
          sources:
            s3:
              type: aws_s3
              compression: auto
              strategy: sqs
              region: "us-east-1"
              sqs:
                delete_message: true
                poll_secs: 5
                visibility_timeout_secs: 300
                queue_url: "https://sqs.us-east-1.amazonaws.com/YOUR_ID/alb-logs"
          sinks:
            opensearch:
              type: elasticsearch
              endpoints:
                - "https://YOUR_OPENSEARCH_URL.us-east-1.es.amazonaws.com"
              inputs:
                - alb 4. Enabling Our Logs There are three ways you can enable logging: Old-fashioned documentation from AWS - link Old-fashioned documentation from AWS - link link With Terraform aws_lb resource - link With Terraform aws_lb resource - link link And the easiest way is - Kubernetes Ingress annotation (our way): And the easiest way is - Kubernetes Ingress annotation (our way): alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-logs Now, you should be able to see logs in your OpenSearch console! That`s all!

ALB Controller Logs With Vector and OpenSearch: A Guide

Prepare Your Environment Deploy Your Own Cluster I highly encourage you to get your personal environment to practice for the exam. Set Up the Master Node The three main things I recommend you to do are: Define aliases (this is already set up in the exam) alias k="kubectl" Define variables export do="--dry-run=client -o yaml"
export do="--force --grace-period 0" Configure vim. Open and include: ~/.vimrc set tabstop=2
set expandtab
set shiftwidth=2 Topics Contexts Get used to moving between different contexts. Although these commands will be given by the CKA questions, it is important you know: How to create set up a user k config set-credentials <name> \
--client-certificate=<path to .crt> \
--client-key=<path to .key> How to set a new context k config set-context <context name> \
--cluster=<cluster name> \
--user=<name> How to change between contexts k config use-context <context name> Useful documentation: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#config JSON Paths Most probably, you will need to get through objects and filter some data. You should be able to be familiar with JSON paths. This does not mean you need to memorize everything. My advice is: to keep an idea of the basic structure of the objects (e.g., when it is a key-value or key-array), and use the following parameter: -o jsonpath="<filter>" Useful documentation: https://kubernetes.io/docs/reference/kubectl/jsonpath/ Important Configuration Files Know where the configuration files for are stored: kubelet /var/lib/kubelet/ Know where the configuration files for are stored: kubeadm /etc/kubernetes/admin.conf Know where the configuration files are stored: CNI /etc/cni/inet.d/ Know where configurations are stored: static pod /etc/kubernetes/manifests/ Declarative Syntax Be comfortable creating and editing YAML files. By practice, get the structure of the key resources: Pod Service PersistentVolume PersistentVolumeClaim Secret ConfigMap Most of the time, we will be using the option to create our template. This is why we created the environment variable at the beginning. dry run do k -n <namespace> run <pod name> --image <image name> $do > pod-example.yaml
k -n <namespace> create deployment <dply name> --image <image name> --replicas <num> $do > dply.yaml The following structure defines the skeleton of any K8s object: apiVersion:
metadata:
kind:
spec: Monitoring How to see nodes and pods, as well as containers, and resource usage: k top nodes
k top pods --containers=true RBAC (Role Based Access Control) Be comfortable creating Roles, RoleBinding, ClusterRole, ClusterRoleBinding, and ServiceAccount objects. For these tasks, I would always try to use imperative commands. It is way easier to create them using imperative commands and this saves us time. Remember that Roles and RoleBindings are namespaced API-resources: The most common steps will be: Create the User or ServiceAccount. Create the Role or ClusterRole object. Create the RoleBinding or ClusterRoleBinding. Test the changes with the command. auth can-i k auth can-i <verb> <resources> --as=<user/sa> --namespace=<namespace> DaemonSet Just a quick tip here. We can create a DaemonSet object using as a template a Deployment YAML file. We just need to change the entry, and remove the , and entries. kind replicas strategy status k -n <ns> create deployment <dply name> --image <img name> $do > daemonset.yaml Container Runtime Get familiar with to start, stop, inspect, and delete containers. crictl Useful documentation: https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/ Schedulers kube-scheduler is a static Pod defined in the manifests we can find under by default. We can stop this pod by simply moving the manifest file from this directory to another one. /etc/kubernetes/manifests The way to schedule a Pod in a node is by using the following directive inside the part: spec nodeName Have in mind that as well as affinity and anti-affinity directives are actually used by the scheduler to decide in which node (if there is one) our Pod will be scheduled. nodeSelector Scheduling Pods We can give directives to the schedulers to choose an appropriate node by using: , it uses node labels. nodeSelector .spec.nodeSelector -> situation in a YAML file

spec:
  nodeSelector:
    <label>: <value> Affinity and anti-affinity constraints. More complex constraints than . nodeSelector .spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions.[]
.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions.[]

spec:
  affinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key:
          operator:
          values:
          -

spec:
  affinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key:
          operator:
          values:
          - Topology spread constraints can be used to control how Pods are spread across your cluster. Taints and tolerations are also a topic which is very easy to manage and important when we talk about scheduling. k taint nodes <node name> <key>=<value>:<effect> In order to remove a taint, we just need to use the same command as we did above but with a minor inclusion: k taint nodes <node name> <key>=<value>:<effect>- We will include a toleration in a Pod spec by using the following directives: .spec.tolerations.[] -> situation in a YAML file

spec:
  tolerations:
  - key:
    operator:
    value:
    effect: Useful documentation: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ Upgrades Suppose a node has never been initialized. Upgrade with Kubeadm will fail because there is nothing to update. We just need to focus on kubectl and kubelet upgrades. After that, it is just a matter of creating the new token to join the node to the cluster with Kubeadm. The steps to upgrade are: Check what versions are being used. Check what is the target version. Check the plan. Upgrade kubeadm. Upgrade kubelet. The process is different if a node is an active node or not. For an active node, we need to drain the node first. Update kubelet and then uncordon the node to make it available. Useful documentation: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ Static Pods Static Pods are not observed by the API server and are managed by Kubelet on a specific node. If you want to create a static Pod, you need to place the YAML manifest for that Pod inside the default manifest path. This path is configured in the Kubelet configuration file ( ) /var/lib/kubelet/config.yaml After you have placed your YAML file there, just restart the kubelet service by using: systemctl restart kubelet etcd Backups It is very common that you will need to backup and restore etcd. You can do it via etcdctl. The following is an example of doing so through the etcd pod running in the cluster: k -n kube-system exec <pod-name> -- /bin/sh -c \ 
"ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /var/lib/etcd/snapshot.db"

1:1 CKA (Certified Kubernetes Administrator): An Essential Guide

How to Structure Your K8s GitOps Repository at Scale [Part 1]

requires no introduction, as it stands as the undisputed king of Continuous Integration. Over the years, it has adapted to all the technological disruptions in the industry, including . Jenkins Kubernetes This blog post delves into an intriguing topic: how to execute on-demand slaves in a remote AWS Fargate cluster from a Jenkins master instance. For those wondering why such a capability is necessary, the following sections will elucidate not only the reasons but also the methods and associated advantages. Everything is remote! Imagine this: you're running cloud-native services on AWS EKS, and as the diligent engineer that you are, you establish two distinct clusters—one for production and another for all non-production purposes. You might be wondering why you would undertake such an approach. Here's a hint: consider the blast radius. If you prioritize the security of your cloud-native services as fervently as we do at NimbleWork, this decision makes sense. The dev cluster operates, among other things, all our Continuous Integration and Delivery tools. Speaking of Continuous Delivery, we run nightly pipelines that test the services for performance, security vulnerabilities, and regression before deploying them to the production cluster. In the production cluster, we adhere to a blue-green deployment model. This entails the Jenkins master, which operates on the dev EKS cluster, to run slaves on the production EKS cluster for various deployment, management, and general housekeeping tasks. Having outlined the reasons for this setup, let's delve into the details of how it is accomplished. Prerequisites This post assumes you have a running AWS , either on Fargate or Worker Nodes. You can refer to this for creating a Fargate cluster or for Worker nodes if you don’t have them handy. The next step is to install Jenkins on Kubernetes, you can refer to this in their official documentation for this. Since we’re configuring Jenkins slaves to run on AWS Fargate, install the Kubernetes Plugin in Jenkins. EKS cluster article this page Serverless Jenkins Slaves Configuring Kubernetes Connection in Jenkins Master Kubernetes cluster can be configured from the Manage Nodes and Clouds option on the Manage Jenkins Page. Navigate to `Manage Jenkins > Clouds > New Cloud` to open the Cloud configuration page Add a name for the cloud, choose Kubernetes in the Type section, and click on "Create” to create the cloud configuration Expand the Kubernetes Cloud details dropdown, this is where we will configure Jenkins master access to the AWS Fargate Cluster. Kubernetes URL Here we add the public API Server URL of the AWS Fargate Cluster. Log in to the AWS Management Console and select Elastic Kubernetes Service, click on the clusters link to list all clusters in your account, and then click on the name of the cluster you want to connect Jenkins with to reach the overview page. Copy the API Server URL from the highlighted section in the image below and paste it to the Kubernetes URL field in the cloud configuration page in Jenkins. Alternatively, you can get the same information by using the command line too. kubectl Point to your EKS cluster: export AWS_ACCESS_KEY_ID="KEY_ID_HERE"
export AWS_SECRET_ACCESS_KEY="ACCESS_KEY_HERE"
export AWS_SESSION_TOKEN="SESSION_TOKEN_HERE" 

aws eks update-kubeconfig --region us-east-1 --name mycluster then run the commands as follows: kubectl kubectl cluster-info
Kubernetes control plane is running at https://XXXXXXXXXXXXX.gr7.us-east-1.eks.amazonaws.com Kubernetes server certificate key We’ll be using a Kubernetes Service Account to authenticate to the API Server. Perform the following steps on the AWS EKS cluster to enable Jenkins access. Create a namespace associated with the Fargate profile jenkins-jobs Create a service account named in the namespace jenkins-service-account Create a secret token named in the namespace associated with the service account jenkins-token Create a role-binding providing the service-account ClusterRole admin in this namespace. jenkins-jobs you can achieve this in multiple ways, via the Management Console or AWS CLI, we like to stick to IAC in NimbleWork so here’s a sample terraform snippet for the same: resource "kubernetes_service_account" "jenkins-service-account" {
  metadata {
    name      = "jenkins"
    namespace = "jenkins-jobs"
    labels = {
      "app.kubernetes.io/name" = "jenkins"
    }
  }
  secret {
    name = "jenkins-token"
  }
  depends_on = [module.fargate-profile]
}

resource "kubernetes_secret" "jenkins-token" {
  metadata {
    name = "jenkins-token"
    namespace = "jenkins-jobs"
    labels = {
      "app.kubernetes.io/name" = "jenkins"
    }
    annotations = {
      "kubernetes.io/service-account.name" = "jenkins"
    }
  }
  type = "kubernetes.io/service-account-token"
}

resource "kubernetes_role_binding" "jenkins-role-binding" {
  metadata {
    name      = "jenkins-role-binding"
    namespace = "jenkins-jobs"
    labels = {
      "name" = "jenkins-role-binding"
    }
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "admin"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "jenkins"
    namespace = "jenkins-jobs"
  }

  depends_on = [kubernetes_service_account.jenkins-service-account]
} Let’s retrieve the certificate key now, run the following command to get the service account certificate key and token % kubectl get secret jenkins-token --namespace=jenkins-jobs -o yaml The output contains and ca.crt token apiVersion: v1
data:
  ca.crt: XXXXXXXXXX
  namespace: XXXXXXXXXXXXXX
  token: XXXXXXXXXX
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: jenkins
    kubernetes.io/service-account.uid: XXXXXXXXXXXXXX
  creationTimestamp: "2023-05-20T17:25:16Z"
  labels:
    app.kubernetes.io/name: jenkins
  name: jenkins-token
  namespace: jenkins-jobs
  resourceVersion: "3388"
  uid: XXXXXXXXXXXXXX
type: kubernetes.io/service-account-token the value is encoded, decode it via the command and paste the resultant value into the Kubernetes server certificate key field. ca.crt base64 base64 -d Kubernetes Namespace Enter the value here jenkins-jobs Credentials Click on Add > Jenkins and choose the Secret text in the Kind dropdown list of the Credentials provider pop-up, then add the base64 decoded value of the from the  output of the command above to create the credentials. token kubectl get secret Click on the Test Connection button and you’ll see the message Connected to Kubernetes v1.28-eks-XXXXXXX When successfully connected! Pod Template and Retention Add the following values to Pod-Template and Retention settings Click Save to finish adding the cloud. Using the Kubernetes Cloud in Pipelines Now that we have the Jenkins configuration in place let’s look at defining builds to run via Fargate Pods as slaves. We’re using the declarative pipeline syntax here. The pipeline job DSL Groovy should mention the  configured cloud name as follows: agent {
    hackernoonkube {
      yamlFile 'builder.yaml'
    }
  } Add the labels defined in the POD template section above to run the job in a Jenkins slave running as a Fargate Pod.

Stories, News & Mentions about SERVICEMAC

Servicemac’s Stories on HackerNoon

Servicemac's News & Mentions