Introducing Paceline — the first-ever fitness rewards platform that adds financial benefit to your physical activity

Paceline

paceline

The Noonification: How to Create a CI/CD Pipeline Using GitHub and AWS EC2 (5/1/2024)

Hey everyone, today I want to do a tutorial on one of my favorite ways to deploy a quick REST API, using NodeJS/Express, to an Amazon EC2 server. This technique essentially enables your EC2 server to respond to changes pushed to your GitHub repository. First Things First, “Hello World” Create a new directory on your machine for a NodeJS/Express app to live. I’m calling mine rekognition Open up a terminal, and run this command to initialize a package.json npm init -y Install express.js npm i express Create a basic express server with a ‘hello world’ at the default route const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => {
    res.send('Hello World!');
});

app.listen(port, () => {
    console.log(`Example app listening at http://localhost:${port}`);
}); Test to make sure you are seeing ‘Hello World!’ at http://localhost:3000/ Good job, step one complete! 💪 Next, Let’s Push Our Code to GitHub Initialize a Git repo in your directory, and make your first commit — git init
git add .
git commit -m "First commit" Create a remote repository on your GitHub account Add the new repo as the origin, and push your branch. git remote add origin <REMOTE_REPOSITORY_URL>
git push -u origin master Replace <REMOTE_REPOSITORY_URL> with the link to the remote repo that you just created. You might need to change your repository settings to use ‘master’ as the default branch instead of ‘main.’ After you do this you can delete the ‘main’ branch. Step 2: complete! We’re cooking with gas ⛽️ Let’s Go to the Cloud — Setup the Ec2 Server Here are the steps to launch your EC2 Instance: Log in to your AWS Management Console. Navigate to the EC2 dashboard, and launch a new instance.



Choose an appropriate AMI (Amazon Machine Image); I am using Amazon Linux 2. Select an instance type (e.g., t2.micro if you are under the free tier). Configure instance details, add storage, and add tags as necessary. Configure a security group to allow inbound traffic on ports you’ll need (at least TCP port 22 for SSH and your application’s port, and then 3000 for web traffic). Review and launch the instance; ensure that you save the key pair used for SSH access. Now, let’s test logging into your server via SSH on a terminal: ssh ec2-user@<YOUR_SERVER_IP> -i <YOUR_PRIVATE_KEY>.pem Replace the variables there with the values you defined for your EC2 instance. If you see this you’ve connected to your server, congrats! 👏 We’re Not Done Just Yet Though — We’re Just Getting Started 😏 Let’s install NodeJS on our server:sudo yum install -y nodejs Install pm2 globally on our server via npm sudo npm i -g pm2 Install git on the server, and configure your credentials: sudo yum install git -y git config --global user.name "Your Name"
git config --global user.email "your.email@example.com" Okay, now we’re going to create a var/www/ folder if it does not already exist: sudo mkdir -p /var/www Then add ownership to our ec2-user: sudo chown -R ec2-user:ec2-user /var/www Navigate to the directory from the root of your server — not the /home folder people 👌 cd ~/var/www Now, from here, we will clone our git repository: git clone https://github.com/USERNAME/REPO_NAME.git YOUR_PROJECT_DIRECTORY Replace [USERNAME] [REPO_NAME] and [YOUR_PROJECT_DIRECTORY] with your own values. Next, we change the directory into our repository, so cd YOUR_PROJECT_DIRECTORYcan create the ecosystem.config.js file for our pm2 setup: nano ecosystem.config.js It looks like this: module.exports = {
  apps : [{
    name: "my-app",  // A name for your application
    script: "app.js",  // The script file to launch the app
    instances: "max",  // This can be set to the number of CPU cores to use, or "max" to use all available
    autorestart: true,  // Automatically restart app if it crashes
    watch: false,  // Enable/disable watching file changes for automatic restart
    max_memory_restart: '1G',  // Optional: Restart app if it reaches a memory limit
    env: {
      NODE_ENV: "development",
    },
    env_production: {
      NODE_ENV: "production",
    }
  }]
}; Cool beans, okay — once that is created and saved, let’s start up the app with pm2: pm2 start ecosystem.config.js Now, we can check that we have a ‘Hello World!’ from our server on port 3000: To make pm2 start automatically on server restarts, let’s run: pm2 save
pm2 startup To check your server is running, you can do pm2 status;  you can read the docs on pm2 here If you’ve made it this far, you are a superstar ⭐️ Integration With GitHub Actions This is the last step in our process. We’re going to create a .yml file for Github actions to update our server anytime a change is pushed to our master branch In your repository, go to the ‘Actions’ tab and create a new custom workflow. I am calling mine nodejs.yml Here is the .yml file for updating the repo and restarting pm2: name: Node.js CI/CD on Amazon Linux

on:
  push:
    branches: [ master ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Install dependencies
      run: npm install

    - name: Deploy to AWS EC2
      uses: appleboy/scp-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        source: "./"  # This now correctly points to the current directory
        target: "/var/www/your_project_directory"

    - name: Restart PM2 application
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd /var/www/your_project_directory
          pm2 reload all  # This command restarts your application Make sure to update ‘your_project_directory’ Lastly, you’ll need to add these environment secrets. REMOTE_HOSTthis is the IP address of your server; REMOTE_USER this is ec2-user for Amazon Linux 2 AMI, and then paste the contents of your .pem file as the SSH_PRIVATE_KEY You can do all of this in the repository settings: That’s it! We’re done yay 😁 You can test this all by changing the ‘Hello World’ message and pushing that change to your master branch; then you should see the update at http://your.ip.address:3000 It is a bit of a process to set up, but when you’re developing, it’s really nice to have an easy CI/CD pipeline automatically deploying changes to your REST API server. That way, you can quickly deploy the changes live and see them in the front end of your web/mobile application. Let me know if this was useful in the comments, and thanks for reading!! Peace out, — Wes Hey everyone, today I want to do a tutorial on one of my favorite ways to deploy a quick REST API, using NodeJS/Express, to an Amazon EC2 server. This technique essentially enables your EC2 server to respond to changes pushed to your GitHub repository. First Things First, “Hello World” Create a new directory on your machine for a NodeJS/Express app to live. I’m calling mine rekognition rekognition Open up a terminal, and run this command to initialize a package.json npm init -y npm init -y Install express.js npm i express npm i express Create a basic express server with a ‘hello world’ at the default route const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => {
    res.send('Hello World!');
});

app.listen(port, () => {
    console.log(`Example app listening at http://localhost:${port}`);
}); const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => {
    res.send('Hello World!');
});

app.listen(port, () => {
    console.log(`Example app listening at http://localhost:${port}`);
}); Test to make sure you are seeing ‘Hello World!’ at http://localhost:3000/ http://localhost:3000/ Good job, step one complete! 💪 Next, Let’s Push Our Code to GitHub Initialize a Git repo in your directory, and make your first commit — git init
git add .
git commit -m "First commit" git init
git add .
git commit -m "First commit" Create a remote repository on your GitHub account Add the new repo as the origin, and push your branch. git remote add origin <REMOTE_REPOSITORY_URL>
git push -u origin master git remote add origin <REMOTE_REPOSITORY_URL>
git push -u origin master Replace <REMOTE_REPOSITORY_URL> with the link to the remote repo that you just created. <REMOTE_REPOSITORY_URL> You might need to change your repository settings to use ‘master’ as the default branch instead of ‘main.’ After you do this you can delete the ‘main’ branch. Step 2: complete! We’re cooking with gas ⛽️ Let’s Go to the Cloud — Setup the Ec2 Server Here are the steps to launch your EC2 Instance: Log in to your AWS Management Console. Log in to your AWS Management Console . AWS Management Console Navigate to the EC2 dashboard, and launch a new instance. Choose an appropriate AMI (Amazon Machine Image); I am using Amazon Linux 2. Navigate to the EC2 dashboard, and launch a new instance. Navigate to the EC2 dashboard, and launch a new instance. Choose an appropriate AMI (Amazon Machine Image); I am using Amazon Linux 2. Choose an appropriate AMI (Amazon Machine Image); I am using Amazon Linux 2. Select an instance type (e.g., t2.micro if you are under the free tier). Select an instance type (e.g., t2.micro if you are under the free tier). Configure instance details, add storage, and add tags as necessary. Configure instance details, add storage, and add tags as necessary. Configure a security group to allow inbound traffic on ports you’ll need (at least TCP port 22 for SSH and your application’s port, and then 3000 for web traffic). Configure a security group to allow inbound traffic on ports you’ll need (at least TCP port 22 for SSH and your application’s port, and then 3000 for web traffic). Review and launch the instance; ensure that you save the key pair used for SSH access. Review and launch the instance; ensure that you save the key pair used for SSH access. Now, let’s test logging into your server via SSH on a terminal: ssh ec2-user@<YOUR_SERVER_IP> -i <YOUR_PRIVATE_KEY>.pem ssh ec2-user@<YOUR_SERVER_IP> -i <YOUR_PRIVATE_KEY>.pem Replace the variables there with the values you defined for your EC2 instance. If you see this you’ve connected to your server, congrats! 👏 We’re Not Done Just Yet Though — We’re Just Getting Started 😏 Let’s install NodeJS on our server: sudo yum install -y nodejs sudo yum install -y nodejs Install pm2 globally on our server via npm pm2 sudo npm i -g pm2 sudo npm i -g pm2 Install git on the server, and configure your credentials: sudo yum install git -y sudo yum install git -y git config --global user.name "Your Name"
git config --global user.email "your.email@example.com" git config --global user.name "Your Name"
git config --global user.email "your.email@example.com" Okay, now we’re going to create a var/www/ folder if it does not already exist: var/www/ sudo mkdir -p /var/www sudo mkdir -p /var/www Then add ownership to our ec2-user: sudo chown -R ec2-user:ec2-user /var/www sudo chown -R ec2-user:ec2-user /var/www Navigate to the directory from the root of your server — not the /home folder people 👌 cd ~/var/www from the root of your server cd ~/var/www Now, from here, we will clone our git repository: git clone https://github.com/USERNAME/REPO_NAME.git YOUR_PROJECT_DIRECTORY git clone https://github.com/USERNAME/REPO_NAME.git YOUR_PROJECT_DIRECTORY Replace [USERNAME] [REPO_NAME] and [YOUR_PROJECT_DIRECTORY] with your own values. Next, we change the directory into our repository, so cd YOUR_PROJECT_DIRECTORY can create the ecosystem.config.js file for our pm2 setup: cd YOUR_PROJECT_DIRECTORY ecosystem.config.js nano ecosystem.config.js nano ecosystem.config.js It looks like this: module.exports = {
  apps : [{
    name: "my-app",  // A name for your application
    script: "app.js",  // The script file to launch the app
    instances: "max",  // This can be set to the number of CPU cores to use, or "max" to use all available
    autorestart: true,  // Automatically restart app if it crashes
    watch: false,  // Enable/disable watching file changes for automatic restart
    max_memory_restart: '1G',  // Optional: Restart app if it reaches a memory limit
    env: {
      NODE_ENV: "development",
    },
    env_production: {
      NODE_ENV: "production",
    }
  }]
}; module.exports = {
  apps : [{
    name: "my-app",  // A name for your application
    script: "app.js",  // The script file to launch the app
    instances: "max",  // This can be set to the number of CPU cores to use, or "max" to use all available
    autorestart: true,  // Automatically restart app if it crashes
    watch: false,  // Enable/disable watching file changes for automatic restart
    max_memory_restart: '1G',  // Optional: Restart app if it reaches a memory limit
    env: {
      NODE_ENV: "development",
    },
    env_production: {
      NODE_ENV: "production",
    }
  }]
}; Cool beans, okay — once that is created and saved, let’s start up the app with pm2: pm2 start ecosystem.config.js pm2 start ecosystem.config.js Now, we can check that we have a ‘Hello World!’ from our server on port 3000: To make pm2 start automatically on server restarts, let’s run: pm2 save
pm2 startup pm2 save
pm2 startup To check your server is running, you can do pm2 status ;  you can read the docs on pm2 here pm2 status pm2 here If you’ve made it this far, you are a superstar ⭐️ Integration With GitHub Actions This is the last step in our process. We’re going to create a .yml file for Github actions to update our server anytime a change is pushed to our master branch .yml In your repository, go to the ‘Actions’ tab and create a new custom workflow. I am calling mine nodejs.yml nodejs.yml Here is the .yml file for updating the repo and restarting pm2: name: Node.js CI/CD on Amazon Linux

on:
  push:
    branches: [ master ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Install dependencies
      run: npm install

    - name: Deploy to AWS EC2
      uses: appleboy/scp-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        source: "./"  # This now correctly points to the current directory
        target: "/var/www/your_project_directory"

    - name: Restart PM2 application
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd /var/www/your_project_directory
          pm2 reload all  # This command restarts your application name: Node.js CI/CD on Amazon Linux

on:
  push:
    branches: [ master ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Install dependencies
      run: npm install

    - name: Deploy to AWS EC2
      uses: appleboy/scp-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        source: "./"  # This now correctly points to the current directory
        target: "/var/www/your_project_directory"

    - name: Restart PM2 application
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.REMOTE_HOST }}
        username: ${{ secrets.REMOTE_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd /var/www/your_project_directory
          pm2 reload all  # This command restarts your application Make sure to update ‘your_project_directory’ Lastly, you’ll need to add these environment secrets. REMOTE_HOST this is the IP address of your server; REMOTE_USER this is ec2-user for Amazon Linux 2 AMI, and then paste the contents of your .pem file as the SSH_PRIVATE_KEY REMOTE_HOST REMOTE_USER ec2-user SSH_PRIVATE_KEY You can do all of this in the repository settings: That’s it! We’re done yay 😁 You can test this all by changing the ‘Hello World’ message and pushing that change to your master branch; then you should see the update at http://your.ip.address:3000 http://your.ip.address:3000 It is a bit of a process to set up, but when you’re developing, it’s really nice to have an easy CI/CD pipeline automatically deploying changes to your REST API server. That way, you can quickly deploy the changes live and see them in the front end of your web/mobile application. Let me know if this was useful in the comments, and thanks for reading!! Peace out, — Wes

How to Create a CI/CD Pipeline Using GitHub and AWS EC2

GitHub এবং AWS EC2 ব্যবহার করে কিভাবে একটি CI/CD পাইপলাইন তৈরি করবেন

So erstellen Sie eine CI/CD-Pipeline mit GitHub und AWS EC2

Cómo crear una canalización de CI/CD utilizando GitHub y AWS EC2

Comment créer un pipeline CI/CD à l'aide de GitHub et AWS EC2

GitHub और AWS EC2 का उपयोग करके CI/CD पाइपलाइन कैसे बनाएं

GitHub と AWS EC2 を使用して CI/CD パイプラインを作成する方法

GitHub 및 AWS EC2를 사용하여 CI/CD 파이프라인을 만드는 방법

Como criar um pipeline de CI/CD usando GitHub e AWS EC2

Как создать конвейер CI/CD с помощью GitHub и AWS EC2

GitHub ve AWS EC2 Kullanılarak CI/CD Ardışık Düzeni Nasıl Oluşturulur

Cách tạo quy trình CI/CD bằng GitHub và AWS EC2

如何使用 GitHub 和 AWS EC2 创建 CI/CD 管道

Building a DevOps Automated Deployment Pipeline using Flask, Aptible, CircleCI, and Prometheus

Data is widely considered the lifeblood of an organization; however, it is worthless—and useless—in its raw form. Help is needed to turn this data into life-giving information. Part of this help is to move this data from its source to destination via data pipelines. But managing the end-to-end data pipeline lifecycle is not easy: ones are challenging to scale on demand, others can result in delays in detecting and addressing issues or be difficult to control. These and the other problems may be solved in the declarative programming paradigm, and in this article we will see it on Terraform. Where Does This Data Come From? The data in these extensive datasets come from multiple disparate sources and are typically loaded into datasets in a data storage like ClickHouse via data pipelines. These pipelines play a critical role in this process by efficiently moving, transforming, and processing the data. For instance, imagine you own a data analytics startup that provides organizations with real-time data analytics. Your client owns an eBike hire startup that wants to integrate weather data into its existing data to inform its marketing messaging strategies. Undoubtedly, your data engineers will set up many different data pipelines, pulling data from many sources. The two we are concerned with are designed to extract data from your client’s CRM and the local weather station. The Layers of the Modern Data Stack A data stack is a collection of different technologies, logically stacked on top of each other, that provides end-to-end data processing capabilities. As described here, a data stack can have many different layers. The modern data stack consists of ingestion (data pipelines), storage (OLAP database), and business intelligence (data analytics) layers. Lastly, there is an orchestration layer—for example, Kubernetes or Docker Compose—that sits on top of these layers, orchestrating everything together so that the stack does what it is supposed to do; ergo, ingest raw data and egest advanced predictive analytics. Data Pipelines are typically structured either as Extract, Transform, Load (ETL) or Extract, Load Transform (ELT) pipelines. However, developing, monitoring, and managing these data pipelines, especially in international enterprises with many branches worldwide, can be highly challenging, especially if the end-to-end data pipeline monitoring process is manual. Developing Data Pipelines, the Traditional Way Data pipelines are broadly categorized into two groups: code-based and deployed in a Direct Acyclic Graph or DAG-like tools such as Airflow and Apache Fink or non-code-based, which are  often developed in a SaaS-based application via a drag-and-drop UI. Both have their own problems. Code-based data pipelines are challenging to scale on demand. The development/test/release cycle is typically too long, especially in critical environments with thousands of pipelines, loading petabytes of data into data stores daily. Additionally, they are typically manually implemented, deployed, and monitored by human data engineers, allowing the risk of errors creeping in and significant downtime. With non-code-based data pipelines engineers don’t have access to the backend code, there can be limited visibility and monitoring, resulting in delays in detecting and addressing issues and failures. Moreover, it is challenging to reproduce errors, as making an exact copy of the pipeline is impossible. Lastly, version control using a tool like Git is complex. It is harder to control the evolution of a pipeline as the code is not stored in a repository such as a GitHub repo. The Solution: Improving Your Data Pipelines with Declarative, Reproducible, Modular Engineering The good news is that there is a sustainable solution to these challenges. It is important to remind ourselves of the difference between imperative and declarative programming. In imperative programming, you control how things happen. In declarative programming, you express the logic without specifying the control flow. The answer to the end-to-end data pipeline development, deployment, monitoring, and maintenance challenges is to utilize the declarative programming paradigm, that is, abstracting the computation logic. In practice, this is best achieved by using Terraform by HashiCorp — an open-source project that is an infrastructure-as-code development tool. Because modern world data-intensive application architecture is very similar to the data stack the applications run on, Terraform provides the framework that will make your data stack declarative, reproducible, and modular. Let’s return to our data analytics startup example described above to elucidate these concepts further. Imagine your data analytics app is a web application that executes inside Terraform. As expected, the most significant part of the application is the ClickHouse database. We must build two data pipelines to ETL data from the CRM system—stored in a PostgreSQL database—into the ClickHouse database, transforming and enriching the data before loading it into ClickHouse. The second pipeline extracts real-time weather data from the local weather service—through an API and transforms it—ensuring there are no errors and enriching it—before loading it into ClickHouse. Using Terraform to Code the Data Pipelines Classic infrastructure for modern cloud-based applications looks like: This is an oversimplification, but a more or less solid way of building applications. You have a public subnet with a user facing API / UI, which is connected to a private subnet where data is stored. To simplify things, let’s look at how to use Terraform to build the first data pipeline—to ETL data from your RDBMS (in this case PostgreSQL) system into the ClickHouse database. This diagram visualizes the data pipeline that ELTs the data from the RDBMS database into the ClickHouse database, and then exposed as a connection into Visualization service. This workflow must be reproducible as your company’s information technology architecture has at least three environments: dev, QA, and prod. Enter Terraform. Let’s look at how to implement this data pipeline in Terraform: It is best practice to develop modular code, distilled down to the smallest unit, which can be reused many times. The first step is to start with a , containing nothing more than the provider’s definition. main.tf file provider "doublecloud" {
  endpoint       = "api.double.cloud:443"
  authorized_key = file(var.dc-token)
}
provider "aws" {
  profile = var.profile
} The second step is to create a BYOA netwok: data "aws_caller_identity" "self" {}

data "aws_region" "self" {}

# Prepare BYOC VPC and IAM Role
module "doublecloud_byoc" {
  source  = "doublecloud/doublecloud-byoc/aws"
  version = "1.0.2"
  providers = {
    aws = aws
  }
  ipv4_cidr = var.vpc_cidr_block
}

# Create VPC to peer with
resource "aws_vpc" "peered" {
  cidr_block                       = var.dwh_ipv4_cidr
  provider                         = aws
}

# Get account ID to peer with
data "aws_caller_identity" "peered" {
  provider = aws
}

# Create DoubleCloud BYOC Network
resource "doublecloud_network" "aws" {
  project_id = var.dc_project_id
  name       = "alpha-network"
  region_id  = module.doublecloud_byoc.region_id
  cloud_type = "aws"
  aws = {
    vpc_id          = module.doublecloud_byoc.vpc_id
    account_id      = module.doublecloud_byoc.account_id
    iam_role_arn    = module.doublecloud_byoc.iam_role_arn
    private_subnets = true
  }
}

# Create VPC Peering from DoubleCloud Network to AWS VPC
resource "doublecloud_network_connection" "example" {
  network_id = doublecloud_network.aws.id
  aws = {
    peering = {
      vpc_id          = aws_vpc.peered.id
      account_id      = data.aws_caller_identity.peered.account_id
      region_id       = var.aws_region
      ipv4_cidr_block = aws_vpc.peered.cidr_block
      ipv6_cidr_block = aws_vpc.peered.ipv6_cidr_block
    }
  }
}

# Accept Peering Request on AWS side
resource "aws_vpc_peering_connection_accepter" "own" {
  provider                  = aws
  vpc_peering_connection_id = time_sleep.avoid_aws_race.triggers["peering_connection_id"]
  auto_accept               = true
}

# Confirm Peering creation
resource "doublecloud_network_connection_accepter" "accept" {
  id = doublecloud_network_connection.example.id

  depends_on = [
    aws_vpc_peering_connection_accepter.own,
  ]
}

# Create ipv4 routes to DoubleCloud Network
resource "aws_route" "ipv4" {
  provider                  = aws
  route_table_id            = aws_vpc.peered.main_route_table_id
  destination_cidr_block    = doublecloud_network_connection.example.aws.peering.managed_ipv4_cidr_block
  vpc_peering_connection_id = time_sleep.avoid_aws_race.triggers["peering_connection_id"]
}

# Sleep to avoid AWS InvalidVpcPeeringConnectionID.NotFound error
resource "time_sleep" "avoid_aws_race" {
  create_duration = "30s"

  triggers = {
    peering_connection_id = doublecloud_network_connection.example.aws.peering.peering_connection_id
  }
} This is preparing our Stage for later clusters. Architecture after terraform apply looks follows: Once we prepare a stage, we can create a cluster in this private-subnet: resource "doublecloud_clickhouse_cluster" "alpha-clickhouse" {
  project_id = var.dc_project_id
  name       = "alpha-clickhouse"
  region_id  = var.aws_region
  cloud_type = "aws"
  network_id = doublecloud_network.aws.id

  resources {
    clickhouse {
      resource_preset_id = "s1-c2-m4"
      disk_size          = 34359738368
      replica_count      = 1
    }
  }

  config {
    log_level       = "LOG_LEVEL_TRACE"
    max_connections = 120
  }

  access {
    ipv4_cidr_blocks = [
      {
        value       = doublecloud_network.aws.ipv4_cidr_block
        description = "DC Network interconnection"
      },
      {
        value       = aws_vpc.tutorial_vpc.cidr_block
        description = "Peered VPC"
      },
      {
        value       = "${var.my_ip}/32"
        description = "My IP"
      }
    ]
    ipv6_cidr_blocks = [
      {
        value       = "${var.my_ipv6}/128"
        description = "My IPv6"
      }
    ]
  }
} This is added to our Stage for simple Clickhouse Cluster. But this cluster still empty, so we must enable transfer between PostgreSQL and ClickHouse: resource "doublecloud_transfer_endpoint" "pg-source" {
  name       = "chinook-pg-source"
  project_id = var.dc_project_id
  settings {
    postgres_source {
      connection {
        on_premise {
          tls_mode {
            ca_certificate = file("global-bundle.pem")
          }
          hosts = [
            aws_db_instance.tutorial_database.address
          ]
          port = 5432
        }
      }
      database = aws_db_instance.tutorial_database.db_name
      user     = aws_db_instance.tutorial_database.username
      password = var.db_password
    }
  }
}

data "doublecloud_clickhouse" "dwh" {
  name       = doublecloud_clickhouse_cluster.alpha-clickhouse.name
  project_id = var.dc_project_id
}

resource "doublecloud_transfer_endpoint" "dwh-target" {
  name       = "alpha-clickhouse-target"
  project_id = var.dc_project_id
  settings {
    clickhouse_target {
      connection {
        address {
          cluster_id = doublecloud_clickhouse_cluster.alpha-clickhouse.id
        }
        database = "default"
        user     = data.doublecloud_clickhouse.dwh.connection_info.user
        password = data.doublecloud_clickhouse.dwh.connection_info.password
      }
    }
  }
}

resource "doublecloud_transfer" "pg2ch" {
  name       = "postgres-to-clickhouse-snapshot"
  project_id = var.dc_project_id
  source     = doublecloud_transfer_endpoint.pg-source.id
  target     = doublecloud_transfer_endpoint.dwh-target.id
  type       = "SNAPSHOT_ONLY"
  activated  = false
} This is adding a ELT snapshot process between PostgreSQL and ClickHouse. As a last piece we can add a connection to this newly created cluster to visualization service: resource "doublecloud_workbook" "dwh-viewer" {
  project_id = var.dc_project_id
  title      = "DWH Viewer"

  config = jsonencode({
    "datasets" : [],
    "charts" : [],
    "dashboards" : []
  })

  connect {
    name = "main"
    config = jsonencode({
      kind          = "clickhouse"
      cache_ttl_sec = 600
      host          = data.doublecloud_clickhouse.dwh.connection_info.host
      port          = 8443
      username      = data.doublecloud_clickhouse.dwh.connection_info.user
      secure        = true
      raw_sql_level = "off"
    })
    secret = data.doublecloud_clickhouse.dwh.connection_info.password
  }
} As you can see, most of the code consists of variables. Therefore, executing this code to set up the different environments is easy by adding a file and running with it: stage_name.tfvars terraform apply See the for more information on .tvars files. Terraform documentation // This variable is to set the
// AWS region that everything will be
// created in
variable "aws_region" {
  default = "eu-west-2" // london
}

// This variable is to set the
// CIDR block for the VPC
variable "vpc_cidr_block" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

// This variable holds the
// number of public and private subnets
variable "subnet_count" {
  description = "Number of subnets"
  type        = map(number)
  default = {
    public  = 1,
    private = 2
  }
}

// This variable contains the configuration
// settings for the EC2 and RDS instances
variable "settings" {
  description = "Configuration settings"
  type        = map(any)
  default = {
    "database" = {
      allocated_storage   = 10            // storage in gigabytes
      engine              = "postgres"    // engine type
      engine_version      = "15.4"        // engine version
      instance_class      = "db.t3.micro" // rds instance type
      db_name             = "chinook"     // database name
      identifier           = "chinook"     // database identifier
      skip_final_snapshot = true
    },
    "web_app" = {
      count         = 1          // the number of EC2 instances
      instance_type = "t3.micro" // the EC2 instance
    }
  }
}

// This variable contains the CIDR blocks for
// the public subnet. I have only included 4
// for this tutorial, but if you need more you
// would add them here
variable "public_subnet_cidr_blocks" {
  description = "Available CIDR blocks for public subnets"
  type        = list(string)
  default = [
    "10.0.1.0/24",
    "10.0.2.0/24",
    "10.0.3.0/24",
    "10.0.4.0/24"
  ]
}

// This variable contains the CIDR blocks for
// the public subnet. I have only included 4
// for this tutorial, but if you need more you
// would add them here
variable "private_subnet_cidr_blocks" {
  description = "Available CIDR blocks for private subnets"
  type        = list(string)
  default = [
    "10.0.101.0/24",
    "10.0.102.0/24",
    "10.0.103.0/24",
    "10.0.104.0/24",
  ]
}

// This variable contains your IP address. This
// is used when setting up the SSH rule on the
// web security group
variable "my_ip" {
  description = "Your IP address"
  type        = string
  sensitive   = true
}
// This variable contains your IP address. This
// is used when setting up the SSH rule on the
// web security group
variable "my_ipv6" {
  description = "Your IPv6 address"
  type        = string
  sensitive   = true
}

// This variable contains the database master user
// We will be storing this in a secrets file
variable "db_username" {
  description = "Database master user"
  type        = string
  sensitive   = true
}

// This variable contains the database master password
// We will be storing this in a secrets file
variable "db_password" {
  description = "Database master user password"
  type        = string
  sensitive   = true
}

// Stage 2 Variables

variable "dwh_ipv4_cidr" {
  type        = string
  description = "CIDR of a used vpc"
  default     = "172.16.0.0/16"
}
variable "dc_project_id" {
  type        = string
  description = "ID of the DoubleCloud project in which to create resources"
} In Conclusion… There you have it. A small modern data stack that ETLs data from a PostgreSQL database into a ClickHouse database. The best news is that you can set up multiple data pipelines—as many as you need—using the code in this example. There is also a complete example that you can find , with additional features like adding a peer connection to your existing VPC and creating a sample ClickHouse database with a replication transfer between the two databases. here Have fun playing with your small modern data stack! This article was also . published on DZone

Declarative Engineering: Using Terraform to Code Your Data Pipelines

Introduction Databricks is emerging as one of the main players in the MLOps and DevOps world. In the last month, I experienced this as part of my project at Areto and decided to write a series of hands-on blog posts on how to implement a MLOps Pipeline with Databricks. This is an ongoing learning process for me as well. I write each blog post after I experiment with different features and tools. So, I like to see this blog series as collective learning. Any comments and suggestions are welcome. In this blog post, we'll explore how to build a pipeline using the Databricks DBX tool and GitLab. This guide is designed for data engineers and scientists who are looking to streamline their data processing workflows. We'll cover key concepts, practical applications, and step-by-step instructions to help you integrate these tools effectively into your data projects. Continuous Integration (CI) CI/CD (Continuous Integration/Continuous Delivery): Continuous Integration and Continuous Delivery/Deployment (CI/CD) have become fundamental in modern software development practices. At its core, CI/CD is a method that emphasizes short, frequent updates to software through automated pipelines. While traditionally associated with software development, its principles are increasingly being adopted in data engineering and data science. The essence of CI/CD lies in its ability to automate the stages of software development, particularly in building, testing, and deploying code.  Continuous Integration begins with regular code commits to a shared repository branch, ensuring collaborative development without version conflicts. Each commit undergoes an automated build and test process, validating changes to enhance the quality and reliability of the final product. Databricks: Databricks has emerged as a premier cloud-based platform, uniting the realms of data engineering, machine learning, and analytics. Databricks excels in handling large-scale data processing and complex analytical tasks, but to leverage its full potential, teams need to navigate its multifaceted environment effectively. Integrating CI/CD processes within Databricks environments can streamline workflows, ensuring consistent, reproducible, and scalable data operations. This integration is crucial for teams aiming to maintain agility and efficiency in their data projects, enabling them to deliver reliable, high-quality data solutions consistently. Databricks Workflows: Databricks workflows are particularly significant in the context of CI/CD, as they offer a robust platform for automating and orchestrating data pipelines crucial to continuous integration and deployment. These workflows allow for the seamless scheduling and execution of tasks, which is essential for maintaining the frequent, automated update cycles characteristic of CI/CD. In a CI/CD pipeline, Databricks workflows can be used to automatically process and test large datasets, ensuring that data transformations and analyses are consistently accurate and up-to-date. The diversity of tasks that can be executed within Databricks workflows, such as Notebooks for exploratory data analysis, Python scripts for structured data processing, and Python wheels for custom package deployment, aligns well with the varied needs of CI/CD pipelines. By integrating these tasks into CI/CD workflows, data teams can ensure that every aspect of their data processing and analysis is continuously tested and integrated into the larger data strategy. This integration is key for developing resilient, scalable, and efficient data operations, enabling teams to deliver high-quality, reliable data products rapidly. Databricks CLI Extension (DBX): The Databricks CLI Extension, or DBX, is a pivotal tool in integrating Databricks with CI/CD pipelines, enhancing the automation and management of data workflows. The ability to programmatically control and manipulate data processes is crucial to implementing a CI/CD workflow, and DBX fills this role effectively. It provides a command-line interface for interacting with various Databricks components, such as workspaces, workflows, and clusters, facilitating seamless integration into automated pipelines. Hands-on In our hands-on example, we will create a (very) minimal project that uses dbx to deploy and run a Databricks workflow to manipulate, analyze, and test some data. This practical exercise showcases how Databricks can be used for real-world data analysis tasks. Our CI pipeline will play a crucial role in this process, as it will automate the deployment of our code as a Databricks workflow and validate its output. This validation step is vital to ensure the accuracy and reliability of our analysis. By the end of this exercise, you'll have a clear understanding of how to set up and run a Databricks workflow within a CI pipeline and how such a setup can be beneficial in analyzing and deriving insights from large datasets. This example will not only demonstrate the technical application of Databricks and CI principles but also offer a glimpse into the practical benefits of automated data analysis in a business context. Development pattern using Databricks GUI There are various development patterns to implement our CI pipeline. The common development workflows with dbx are to develop, test, and debug your code on your local environments and then use dbx to batch-run the local code on a target cluster. By integrating a remote repository with Databricks, we use a CI/CD platform such as , , or to automate running our remote repo’s code on our clusters. GitHub Actions Azure DevOps GitLab In this tutorial, for the sake of simplicity, we use Databricks GUI to develop and test our code. We follow these steps: Create a remote repository and clone it into our Databricks workspace. We use here. Gitlab Develop the program logic and test it inside the Databricks GUI. This includes Python scripts to build a Python Wheel package, scripts to test the data quality using Pytest, and a notebook to run the Pytest. Push the code to GitLab. The git push will trigger a Gitlab Runner to build, deploy and launch our workflows on Databricks using dbx Git integration As the first step, we . Next, we create a remote repository and . To allow our Gitlab runner to communicate with Databricks API through dbx, we should add two environment variables, and to our CI/CD pipeline configurations. configure Git credentials & connect a remote repo to Databricks clone it to our Databricks repo DATABRICKS_HOST DATABRICKS_TOKEN to generate a Databricks token, in your Databricks, go to User Settings →  Developer → Access tokens → manage → Generate new token. The Databricks host is the URL when you login into your Databricks workspace. It would be something like . The last part of the URL is your workspace id, and you should ignore that. https://dbc-dc87ke16-h5h2.cloud.databricks.com/ Finally we add the token and host to our Gitlab CI/CD setting. For that open your repo in Gitlab, go to Settings→ CI/CD → Variables → Add variables The project skeleton the project is structured into several folders and key files, each serving a specific purpose: .dbx folder: : defining the configuration of your DBX project. It contains environment settings, dependencies, and other project-specific parameters. project.json conf folder: : outlining the deployment configurations and environment settings. It defines workflows and their respective parameters and cluster configurations. deployment.yml : our wheel package. This folder includes: my_package folder A subfolder containing the main ETL task script . The ETL task load our dataset and create two new tables. tasks sample_etl_job.py The file. it includes common utilities that provide access to components such as common.py SparkSession Contains two Jupyter notebooks: notebooks folder: : plot the distribution of different features in our dataset explorative_analysis : used to execute for unit testing run_unit_test pytest This folder is dedicated to testing: tests folder: : includes pytest fixtures conftest.py : contains unit tests to validate the data structure in our tables. test_data.py In addition to these folders, there are two important files in the root directory: : the configuration file for GitLab's Continuous Integration (CI) service, defining the instructions and commands the CI pipeline should execute. .gitlab-ci.yml : for building my Python wheel package. It defines the package's metadata, dependencies, and build instructions. setup.py dbx-tutorial/
├─ .dbx/
│  ├─ project.json
├─ conf/
│  ├─ deployment.yml
├─ my_package/
│  ├─ tasks/
│  │  ├─ __init__.py
│  │  ├─ sample_etl_job.py
│  ├─ __init__.py
│  ├─ common.py
├─ tests/
│  ├─ conftest.py
│  ├─ test_data.py
├─ notebooks/
│  ├─ explorative_analysis
│  ├─ run_unit_test
├─ .gitlab-ci.yml
├─ setup.py Deployment configuration file is crucial for defining your DBX project's configuration. This file could be generated automatically when you run . Check the dbx website for more details about the . project.json dbx --init Profile file reference The “ ” option is used for local development. If you run the command inside the CI tool, you need to specify the and environment variables, and they will the profile variable. Since we don’t want to use the local environment and develop and run our code on Databricks UI, we don’t need to specify this option here. Instead, the dbx will pick them up from the CI/CD setting that we set earlier. profile dbx databricks host databricks token overwrite Your files that dbx will automatically upload to the system will be stored in They will be stored in .  You can read more about this in ml experiment. artifact_location dbx documentation {
    "environments": { 
        "default": {
            "profile": "dbx-tutorial",
            "storage_type": "mlflow",
            "properties": {
                "workspace_directory": "/Shared/dbx/dbx-tutorial", 
                "artifact_location": "dbfs:/Shared/dbx/projects/dbx-tutorial"
            }
        }
    },
    "inplace_jinja_support": false,
    "failsafe_cluster_reuse_with_assets": false,
    "context_based_upload_for_execute": false
} outlines the deployment configurations and environment settings. Here, we use and format to define the workflows. Each workflow defines an object inside the Databricks workspace: or deployment.yml 2.1 API wheel_task job delta live tables Pipeline. custom:
  existing_cluster_id: &existing_cluster_id
    existing_cluster_id: "1064-xxxxxx-xxxxxxxxx"

environments:
  default: #this should be same envoriment name as you defined in the project.json
    workflows:
      - name: "etl_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            python_wheel_task:
              package_name: "my_package"
              entry_point: "etl_job" # take a look at the setup.py entry_points section for details on how to define an entrypoint
          - task_key: "eda"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/explorative_analysis"
              source: WORKSPACE
            depends_on:
              - task_key: "main"    
      - name: "test_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/run_unit_test"
              source: WORKSPACE
            libraries:
            - pypi:
                package: pytest Here, we define two workflows: : consists of two tasks/jobs. "etl_job" The first task ( ) is of type . It used to run ETL task using the entry point that we define in the file. main python_wheel_task setup.py The second task ( ) is of type . It is used to run the notebook after the main tasks run successfully. Notice the use of property. etl notebook_task explorative_analysis depends_on : consists of only one job of type . It is used to run the notebook that is "test_job" notebook_task responsible for running the pytest. : in the above example, we run the workflow on an existing cluster. You can also create and run a new cluster for every deployment and launch of your work. For that, you should change the code as follows: Note custom:
  basic-cluster-props: &basic-cluster-props
    spark_version: "11.3.x-cpu-ml-scala2.12"

  basic-static-cluster: &basic-static-cluster
    new_cluster:
      <<: *basic-cluster-props
      num_workers: 1
      node_type_id: "Standard_E8_v3"

environments:
  default: 
    workflows:
      - name: "et_job"
        tasks:
          - task_key: "main"
            <<: *basic-static-cluster
						.... CI pipeline configuration Your GitLab CI pipeline is structured to automate the testing and deployment processes of your Databricks project. It consists of two main stages: and . In the stage, the runs the unit tests and deploys a separate workflow for testing. The stage, activated upon successful completion of the test stage, handles the deployment of your main ETL workflow. In general, the pipeline follows these steps: test deployment test unit-test-job deploy Build the project Push the build artifacts to the Databricks workspace Install the wheel package on your cluster Create the jobs on Databricks Workflows Run the jobs image: python:3.9

stages:          # List of stages for jobs, and their order of execution
  - test
  - deploy

unit-test-job:   # This job runs in the test stage.
  stage: test    # It only starts when the job in the build stage completes successfully.
  script:
    - echo "Running unit tests... This will take about 60 seconds."
    - echo "Code coverage is 90%"
    - pip install -e ".[local]"
    - dbx deploy --deployment-file conf/deployment.yml test_job --assets-only
    - dbx launch test_job --from-assets --trace

deploy-job:      # This job runs in the deploy stage.
  stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.
  script:
    - echo "Deploying application..."
    - echo "Install dependencies"
    - pip install -e ".[local]"
    - echo "Deploying Job"
    - dbx deploy --deployment-file conf/deployment.yml etl_job
    - dbx launch etl_job --trace
    - echo "Application successfully deployed."
    - echo "remove all workflows." For workflow deployment and lunch, we use two commands: : to deploy the workflow definitions to the given environment. dbx deploy : to launch the given workload by its name in a given environment. dbx lunch In the we use the flag to avoid creating a job definition on our Databricks workflows during deployment. For the lunch command, we should use the flag. If this flag is provided, the launch command will search for the latest deployment that was assets-only. Lunching this, we can not see anything in the Workflows UI, but it will be a non-titled job  running by submit API. You can see this on the Job Runs page **** of your Databricks workspace. Check out the dbx documentation to read about and unit-test-job --assets-only --from-assets assest-based workflows other commands If you check the “Experiments” page of your Databricks workspace, you will find the build artifacts of each run of your CI pipeline. Databricks Asset Bundles currently, Databricks recommends using Asset Bundles for CI/CD and offers a migration guide from dbx to bundles.  If you understand the concepts in this post, it would be easy to switch to bundles. In the next post, I am going to explain how we can convert this project to an asset bundle project. Resources for this tutorial. Make sure you update the file with the correct and Repository conf/deployment.yml cluster_id DB username dbx documentation Databricks Best Practices: Development loop and CI/CD on Databricks with dbx Databricks Jobs API CI/CD techniques with Git and Databricks Repos What is CI/CD on Azure Databricks?

Building a CI Pipeline with Databricks dbx Tool and GitLab

The pipeline design pattern in is a valuable tool for software engineers looking to optimize data processing. By breaking down a complex process into multiple stages, and then executing those stages in parallel, engineers can dramatically reduce the processing time required. This also simplifies complex operations and enables engineers to build scalable data processing pipelines. C# design pattern In this article, I’ll provide a detailed overview of how to implement the pipeline design pattern in C#. I’ll share the fundamental concepts behind the pattern, provide example code to illustrate implementation and provide tips for optimizing its performance. We’ll also highlight some common pitfalls and share guides on how to avoid them. Finally, we’ll discuss real-world scenarios where this pattern can be applied and provide specific use cases to illustrate. Understanding the Pipeline Design Pattern The pipeline design pattern is commonly used in software engineering for efficient data processing. This design pattern utilizes a series of stages to process data, with each stage passing its output to the next stage as input. The pipeline structure is made up of three components: The source: Where the data enters the pipeline The stages: Each stage is responsible for processing the data in a particular way The sink: Where the final output goes Implementing the pipeline design pattern offers several benefits, with one of the most significant benefits the efficiency of processing large amounts of data. By breaking down the data processing into smaller stages, the pipeline can handle larger datasets. The pattern also allows for easy scalability, making it easy to add additional stages as needed. The pipeline design pattern offers us a flexible and efficient way to process large datasets. With its straightforward structure made up of three components, you can easily create pipelines that meet your specific needs and scale as their data processing requirements grow. Follow along with this video on the pipeline design pattern in C#: https://www.youtube.com/watch?v=Z4qUWxTH0Mw&embedable=true Implementing the Pipeline Design Pattern in C# To implement the pipeline design pattern in C#, there are specific steps that you’ll need to follow. First, you must define each stage of the pipeline. After creating the stages, you’ll need to chain them together in the correct order, connecting the output of each stage to the input of the next. Finally, you’re going to need to define a sink component to receive the output after the final stage has processed the data. Creating Pipeline Stages To create each stage of the pipeline, you can utilize the C# delegate method. Of course, we can get more specific with creating dedicated APIs through interfaces for the pipeline stages… But using a delegate is quick and easy. First, you’ll define the delegate’s input and output types. Next, you’ll need to code the stage to handle the input data and process it as required. The stage’s output data type must match the next step’s input data type in the pipeline. Here is an example of how to define a delegate for pipeline stages: delegate OutputType MyPipelineStage(InputType input); Chaining Pipeline Stages To execute the pipeline stages in sequence, you’re going to need to chain each stage to the next. To do this, we define the input delegate for each stage to receive the output of the previous stage. Here is an example of how to chain two pipeline stages together: MyPipelineStage firstStage = (InputType input) =>
{
   // process input and return OutputType
};

MyPipelineStage secondStage = (InputType input) =>
{
   var outputFromFirst = firstStage(input);
   // process outputFromFirst and return OutputType
}; We can create a pipeline with multiple stages by repeating the process of defining each stage and chaining them together. The final step is to send the output of the last stage to the sink component, which will look the same. The difference is we don’t continue passing data beyond that point. Example of the Pipeline Design Pattern in C# We’re going to look at an example of the Pipeline Design Pattern in C# that tackles text analysis! I find it’s often best to understand concepts when we can apply them to a practical situation. In this case, we’ll require several stages in a pipeline that can work together: Sanitize the text Some type of frequency analysis Summarization of the results With these roughly as the stages of the pipeline, let’s see how we can get started! Example C# Pipeline Overview Let’s start by defining delegates for each stage of the pipeline. We could declare a specific interface that the pipeline needs to implement, but we’re going to simplify this example by keeping things lightweight and flexible: public delegate string TextCleaner(string input);
public delegate Dictionary<string, int> WordCounter(string input);
public delegate string TextSummarizer(Dictionary<string, int> wordFrequency); Next, we’d have code for each stage. I’ll go into more detail on this in the next section, but for now, we can mark these as follows: TextCleaner cleaner = text =>
{
    /* normalization logic */
    return cleanedText;
};

WordCounter counter = cleanedText =>
{
    /* validation logic */
    return wordFrequency; 
};

TextSummarizer summarizer = wordFrequency =>
{
    /* transformation logic */
    return summary;
}; Next, we need to chain the stages from one to the next. Again, given that this is a simple example, we’ll manually set these stages up to be configured how we need them. Consider though that you could write code that automatically wires these up! Here’s the manual approach: var inputText = "Your input text here";
var cleanedText = cleaner(inputText);
var wordFrequency = counter(cleanedText);
var summary = summarizer(wordFrequency); Each stage is a specific task in text processing. The text cleaner removes unnecessary characters, the word counter creates a frequency map of words, and the summarizer generates a summary based on the most frequent words. The pipeline processes the text through each stage in sequence, demonstrating how different tasks can be modularly connected in a pipeline. Implementations of Pipeline Stages The following are just for demonstration purposes, but here are some implementations that you could consider for the pipeline stages that I listed above: TextCleaner cleaner = text =>
{
    // Example: Remove punctuation and convert to lower case
    var cleanedText = new string(text.Where(c => !char.IsPunctuation(c)).ToArray());
    return cleanedText.ToLower();
}; This stage modifies the text to our liking and returns it as the result of the stage. Next, we’ll look at the counting stage: WordCounter counter = cleanedText =>
{
    var wordFrequency = new Dictionary<string, int>();
    var words = cleanedText.Split(' ');

    foreach (var word in words)
    {
        if (string.IsNullOrWhiteSpace(word))
        {
            continue;
        }

        if (wordFrequency.ContainsKey(word))
        {
            wordFrequency[word]++;
        }
        else
        {
            wordFrequency[word] = 1;
        }
    }

    return wordFrequency;
}; This stage keeps a count of the different words that we have in the text. TextSummarizer summarizer = wordFrequency =>
{
    // Example: Summarize by picking top 3 frequent words
    var topWords = wordFrequency
        .OrderByDescending(kvp => kvp.Value)
        .Take(3)
        .Select(kvp => kvp.Key);
    return $"Top words: {string.Join(", ", topWords)}";
}; Finally, the summary step gets a string together with information about the input data based on the pipeline stages. With these implementations, we now have something that can process some text input with a pipeline! Optimizing the Pipeline Design Pattern in C# To make the pipeline design pattern as efficient as possible, you can make some optimizations in your approach. This section will cover techniques for optimizing the pipeline design pattern in C#. Optimizing Pipeline Performance One way to optimize pipeline performance is through parallelism. Developers can execute multiple stages of the pipeline concurrently. They can implement this technique using the . TPL provides a simple API for executing on multiple processors. Task Parallel Library (TPL) in C# tasks concurrently Another way to optimize pipeline performance is through batch processing. The idea is to batch the input data and apply the pipeline stages to the batches. This technique ensures that the pipeline uses all available resources, which increases efficiency. Developers can control the batch size to find the optimal balance between resource usage and performance. If you’re considering running things in parallel, check out this video for some interesting results on Task.WhenAll and Parallel.ForEachAsync: https://www.youtube.com/watch?v=GE-phyI6lEM&embedable=true Avoiding Common Pitfalls While implementing the pipeline design pattern, you may come across some common pitfalls that hurt pipeline performance. One pitfall is not designing stages that can execute concurrently. It is important to design each stage to be independent from other stages to ensure parallel execution. Another pitfall is not considering the effect of a stage on other stages’ performance. For example, if a stage introduces a lot of processing overhead, it could slow down the pipeline’s overall throughput. To avoid this, you must create stages that are as efficient as possible. Even if you have concurrent steps, this one may become the bottleneck! Optimizing the pipeline design pattern involves leveraging parallelism, batch processing, and designing stages that balance performance with efficiency. By avoiding common pitfalls, you can improve the overall effectiveness of the pipeline. You can use tools like BenchmarkDotNet to measure the performance of your code too! Check out this video to see BenchmarkDotNet in action: https://www.youtube.com/watch?v=Wa3sdKGp3wE&embedable=true Real-world scenarios and Use Cases Pipeline design pattern has numerous real-world applications where it can significantly improve the efficiency of data processing. Here are some examples of scenarios where the pipeline design pattern can be applied: : In this scenario, data scientists and analysts can use a pipeline to process large datasets efficiently. The pipeline can help preprocess data, filter outliers, scale datasets, standardize feature representations, train models, or make predictions. Data Analytics Pipeline : Image processing involves multiple steps such as resizing, cropping, filtering, and enhancing. The pipeline can be tailored to the specific image processing workflow and optimized it for running on large datasets. Image Processing Pipeline : Automated software testing pipelines provide early bug detection and reduce the risk of regressions during software development. The pipeline can be divided into multiple stages, each responsible for running a specific type of test, such as unit or integration tests. Automated Testing Pipeline : Real-time data streaming is becoming increasingly popular in various industries, including finance, healthcare, and telecommunications. The pipeline can help to preprocess and filter incoming data in real-time, enabling quick decision-making. Data Streaming Pipeline : Video processing involves multiple tasks such as decoding, encoding, resizing, and filtering. The pipeline can assist in optimizing these processes, making it easier to handle large amounts of video files. Video Processing Pipeline Wrapping Up Implementing the pipeline design pattern in C# can greatly improve the efficiency of data processing in software development. By breaking down complex tasks into smaller stages, pipelines can reduce latency and improve overall performance. Some of the key takeaways from this article include understanding the structure of a pipeline, creating pipeline stages, and optimizing pipeline performance. It is important to avoid common pitfalls when implementing the pipeline design pattern and monitor and resolve bottlenecks as they arise — And make sure you benchmark your code before prematurely optimizing! By implementing the pipeline design pattern, software engineers and developers can achieve faster and more efficient data processing. If you’re interested in more learning opportunities, and ! subscribe to my free weekly newsletter check out my YouTube channel Also published here.

The Pipeline Design Pattern - Examples in C#

Seamless Implementation of Safety Practices in the DevOps Pipeline

In the modern fast-paced digital landscape, there is a high demand for streamlined and efficient data management tools and services. With the exponential growth of data-driven development and decision-making in organizations, there is a need for robust solutions to optimize data pipelines. Jupyter Notebooks have emerged as one of the most popular choices for organizations to use in data exploration and analysis due to their interactive and user-friendly interface. However, as the scale and complexity of data operations grow, the requirement to easily migrate Jupyter Notebooks into production environments becomes more essential. Here comes the (VDK), a framework that simplifies data ingestion and data processing. A toolset enabling you to run data jobs & comprehensive solutions for the productionization of Jupyter Notebooks. With its powerful features and capabilities, VDK serves as a game-changing tool that enables organizations to easily integrate Jupyter Notebooks into complex data pipelines while ensuring scalability, reproducibility, and enhanced workflow efficiency. Versatile Data Kit In this blog, we see all about , exploring the potential of this open-source toolkit to revolutionize the way organizations manage and process their data. Productionizing Jupyter Notebooks with VDK Understanding the Challenges with Jupyter Notebooks Despite their popularity in data analysis and exploration, Jupyter Notebooks have several inherent limitations when used in production environments. Sometimes, these limitations of the Jupyter Notebook pose significant hurdles during seamless integration into complex data pipelines. Understanding these challenges is important for companies aiming to use Jupyter Notebooks within their production workflows. Some of the biggest challenges include: Jupyter Notebooks sometimes struggle to handle large-scale data processing efficiently. When dealing with large datasets or sophisticated computations, the lack of optimised memory management and the sequential nature of execution might cause performance bottlenecks, impeding the seamless expansion of data pipelines. Scalability and Performance: Version control for Jupyter Notebooks in a collaborative environment can be difficult. Merging changes, managing revisions, and ensuring consistency across various versions of a notebook may become time-consuming, potentially resulting in version conflicts and data inconsistencies, especially when numerous team members are working at the same time. JSON file with lots of irrelevant information as you can see in the image below. Version control and Collaboration: During the deployment of Jupyter Notebooks in production, there is a significant challenge to ensure the reproducibility of results. Variations in the runtime environment, dependencies, and external libraries can all have an impact on the consistency of results, making it difficult to accurately replicate analyses or experiments, especially when switching across computing environments. Reproducibility and Environment Management: If proper security measures and access controls are not implemented in a production environment, Jupyter Notebooks can present security risks. Allowing unrestricted access to notebooks or exposing sensitive data might jeopardize data integrity and confidentiality, potentially leading to security breaches and unauthorized data tampering. Security and Access Control: Few more challenges exist, like irrelevant code, modularization and no proper method for testing due to lack of libraries and tools. Solving Problems with VDK VDK plays an important role in facilitating the seamless integration of Jupyter Notebooks into production pipelines. It is a robust and comprehensive framework designed to simplify the complex process of data ingestion, transformation, and deployment within production pipelines using Python and SQL. Its capabilities for version control, environment management, and scalable data processing enable organizations to overcome the inherent challenges associated with deploying Jupyter Notebooks in production environments. . Check the VDK GitHub repo to learn more We have discussed some of the challenges with Jupyter Notebook above; it’s time to see how we can solve those challenges with VDK. Non-Linear Execution and Hidden State Risks Notebooks support non-linear code execution, which might result in hidden dependencies when cells run out of order. It increases risks when moving to the production environment. Follow this image for understanding. To know about problems, try to focus on the first two objectives — Retrieve Data and Data Cleaning. To retrieve the data, We will first and then load it. After the execution, we can check the data. Now we will clean the data by removing some . import pandas testuser As you can see, while retrieving and cleaning the data, we are unaware of the exact operations that were executed; they are hidden. In the production environment, we aim to ensure clarity about the executed processes, thereby eliminating the need for hidden dependencies or states. Let's explore the VDK solution; we will employ the VDK cell tag (located at the top right in the image). Essentially, it will assign numbering to the tagged cell, indicating the order of execution as 1, 2, 3 when deployed. This approach provides assurance that there are no hidden dependencies or states, and we can only execute what is visible. Irrelevant Code Excessive irrelevant code, like unused statements or unrelated snippets, can be found in notebooks. During the experimental stages of development, a few snippets and useful algorithms that help in interactive changes can be problematic in production. Let’s see the VDK solution; we will do the data classification by assigning into predefined categories for clarity, as you can see in the image below. After executing code from cell 9, we get a new column that contains the types of users. scores To check if something unknown is added to this data, we have defined methods in file. After the visualization, we are sure that only Detractors and Promotors are present and classification is right. Now we have to ingest the organized data using vdk . helper.py job_input # sending data for ingestion
 job_input.send_tabular_data_for_ingestion(
 df.itertuples(index=False),
 destination_table="nps_data",
 column_names=df.columns.tolist()
 ) If we see our whole code, we can know for sure that and will not be executed in production. Some codes are relevant for development and irrelevant to deployment, so we can't just remove these always. VDK helps here to maintain performance without removing irrelevant code. df visualize_data(df) Testing When working with Jupyter Notebooks, there is a lack of proper tools and methods for testing notebooks. It becomes challenging to verify the code you've written, and resorting to alternative solutions is not practical, especially in larger teams. VDK addresses this issue by offering an end-to-end testing solution through the command, as illustrated in the image below. run After running this command, it will execute vdk cells of the above Jupyter Notebook like a production environment. If something fails, it will give the error message and the details of where it originated from. It will be easier for us to fix things. We can also use the testing feature from the create deployment method by checking the box. It will also behave the same, and in case of any failures, it will stop the deployment and show us the error and the exact trace. Run data job before deployment Version Control Version control with notebooks can often become a complex process due to their JSON-based format, which can contain excessive noise. VDK offers a solution to this predicament through two key features. See the version control after using VDK in the image below. Firstly, it implements Noise Reduction, a mechanism that cleanses the notebook’s JSON by eliminating non-essential elements, including execution counts and outputs, known to contribute to the “noise” in the data. This helps streamline the version control process by focusing solely on the relevant content. Secondly, VDK ensures Seamless Integration with Git, facilitating the seamless committing of code to Git during deployment in a cleaner state, free from unnecessary metadata. By simplifying version control and minimizing clutter, VDK enables a more efficient and streamlined workflow for managing notebook-based projects. If you want to understand what we are doing in a step-wise manner with Jupyter Notebook UI, check this . guide Conclusion To sum up, the application of VDK in the productionization of Jupyter Notebooks has numerous benefits. By using the Noise Reduction feature, VDK significantly simplifies the version control process by eliminating unnecessary components that could hinder effective data management. Furthermore, VDK’s seamless integration with Git ensures a tidier and more organized environment for code deployment and collaboration, reducing clutter and simplifying the workflow. Adopting VDK can greatly enhance data pipeline management, enabling users to streamline their operations and improve overall workflow efficiency. It is a valuable resource for those aiming to fully utilize their data-driven projects. Additional Resources 💡Check Versatile Data Kit GitHub Repo 💡Check Youtube Video Tutorial 💡Check the of VDK to learn more Getting Started guide 💡Check VDK in Jupyter Notebook UI guide 💡Go through VDK user guides Also published . here

How to Use Versatile Data Kit to Turn Your Jupyter Notebooks Into Scalable & Reliable Data Pipelines

Telecom's AI Revolution: How Predictive Analytics Optimizes CI/CD Pipelines

How are you, hacker?
🪐 has got you covered with fresh content from our trending stories of the day! Set email preference . Want to know what's trending right now?: The Techbeat by HackerNoon here A Guide to Building an AWS Serverless API and CICD Pipeline By [ 11 Min read ]
This article demonstrates the development of a Serverless API utilizing AWS services @pchandel09 Read More. Everything You Need to Know to Deploy MinIO in Virtualized Environments By [ 4 Min read ]
When deploying MinIO in virtualized environments, it’s important to make sure that the proper conditions are in place. @minio Read More. Enhancing DevOps Efficiency With ChatOps By [ 6 Min read ]
In this article, we will explore how ChatOps tools can be harnessed to supercharge your DevOps practices @vladislavbilay Read More. Bitcoin's Moment in the Sun is Coming By [ 4 Min read ]
Just 2 days ago, Bitcoin’s whitepaper turned 15. Much has happened since those very first days, however, Bitcoin’s promise of decentralisation still stands. @ulriklykke Read More. No Coding Required: 5 Mind-Blowing Uses of GPT-4 By [ 2 Min read ]
What can actually be done using GPT-4? @whatsai Read More. I Trained an AI to Act Like Me: Here's What Happened By [ 3 Min read ]
I trained my official AI by re-claiming my data from social platforms. It was my data in the first place! Now I have an AI version of me that can speak for me. @tomwom Read More. Solving the Hard Problems: A Conversation with Matan Almakis, the Head of Project at DOP By [ 6 Min read ]
A discussion with Matan Almakis, the Head of Project at Data Ownership Protocol. @danstein Read More. The Tech Company News Brief: It's a Bad Time to be Intel By [ 4 Min read ]
As if the underwhelming launch of the "next generation" processors wasn't bad enough, Intel now has to contend with competitors eating at its market share. @sheharyarkhan Read More. Synchronising Asynchronous Remote Work 🌍 By [ 7 Min read ]
Hack smoother remote work with these tested strategies. @walo Read More. Django SaaS Architecture: Single-Tenant vs Multi-Tenant - Which Is Right for You? By [ 9 Min read ]
Single Tenant vs Multi-Tenant SaaS application example using Django @pbityukov Read More. Top 15 Trends of the Upcoming Cryptocurrency Bull Run in 2024-2025 By [ 15 Min read ]
In this article, I want to give my overview of the nascent crypto industry trends, which I believe is coming soon. @serkhitrov Read More. The Pocket Guide to Understanding Docker Containers By [ 2 Min read ]
Docker containers have revolutionized software development and deployment by offering a flexible, efficient, and consistent way to package and run applications. @orioninsist Read More. Open the Doors to the Metaverse With Lado Okhotnikov's Meta Force and Royalty NFT By [ 2 Min read ]
Meta Force and Lado Okhotnikov are scaling the Metaverse and giving away Royalty NFT bonuses. @btcwire Read More. The Ten Commandments of ChatGPT By [ 5 Min read ]
ChatGPT has changed the game for many professions. None more so, perhaps, that in the consulting world. @adrien-book Read More. Making a Good Engineer a Great One: Training Your Skills and Training Your Mindset By [ 7 Min read ]
Navigating challenges in engineering growth: from coding struggles to leadership dilemmas, a narrative of perseverance and self-discovery. @jurajmalenica Read More. Embracing the Future: The Unexplored Synergy of AI in Web Development By [ 8 Min read ]
Explore the dynamic role of AI in reshaping the web development landscape — from streamlining operations to enhancing user experience. Dive into real-world appl @hackerclo3f6rop00003b6vpescpy4b Read More. The Tech Company News Brief: How Much Money Did Tech Companies Make Last Quarter? By [ 4 Min read ]
It's been a busy few weeks, with each of the tech behemoths reporting their earnings for the third quarter of the year. @sheharyarkhan Read More. The State of Webhooks in 2023 By [ 4 Min read ]
In this report on the State of Webhooks in 2023, we looked at over 100 of the top API providers and our own internal data to analyze webhook implementations. @tomhacohen Read More. Newsletter Businesses Are About to 10x Due to AI By [ 7 Min read ]
Are we in a newsletter bubble? I think it’s about to go 10x due to AI and platforms like beehiiv that make it easy to start, scale and monetize. @richard Read More. The Battle Between Proprietary and Open Source AI By [ 7 Min read ]
OpenAI started the AI race with ChatGPT and GPT-4. But after it, many other players have joined the AI race, both in proprietary and open-source AI. 🧑‍💻 What happened in your world this week? It's been said that , , . Feeling stuck? We got you covered ⬇️⬇️⬇️ We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.
See you on Planet Internet! With love,
The Hacker Noon Team ✌️ @juanfrank77 Read More. writing can help consolidate technical knowledge establish credibility and contribute to emerging community standards ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME

The TechBeat: A Guide to Building an AWS Serverless API and CICD Pipeline (11/5/2023)

Overview This article demonstrates the development of a Serverless API utilizing AWS services and establishes a Continuous Integration/Continuous Deployment (CICD) pipeline within the AWS environment. Explores the creation of a Lambda function to handle requests from the API Gateway and persist the data in DynamoDB using the Serverless Application Model. Part 1: Details the necessary steps to configure a CodeCommit repository in AWS and set up a CICD pipeline that automatically initiates a build process upon the submission of new changes to the repository. Part 2: Prerequisites For this project, you'll need an AWS account (Free Tier is sufficient). The following AWS components will be utilized: AWS API Gateway AWS Lambda AWS DynamoDB AWS CodeCommit AWS CodeBuild AWS CodePipeline AWS S3 Miscellaneous - CloudWatch, IAM, etc Ensure you have your local development environment set up as follows: : Follow the guide to install the AWS Command Line Interface. Install AWS CLI here : Install SAM CLI by following the instructions . Install AWS SAM (Serverless Application Model) here : Use IntelliJ or a similar IDE for development. I prefer IntelliJ Choose an IDE : Make sure you have Maven installed for packaging your application. Maven for Packaging (if you need to test Lambda functions locally): Install Docker if you plan to test Lambda functions locally. Optional: Docker These tools and components will form the foundation of the project. Part 1: Development In this section, we will see the process of creating a starter project using AWS SAM, including completing the handler class, building, deploying to AWS, and conducting testing using Postman Environment Setup : AWS Setup Go to the AWS console at , Log in using your admin user credentials. https://aws.amazon.com/console/ : Create a User in IAM In IAM, create a user dedicated for local development and CLI/SAM usage. : Create a DynamoDB Table Name: "users", Primary Key: "id" (Type: String) : Configure AWS CLI on Local Machine Open a terminal and run $ aws configure Provide the access keys for the IAM user created earlier and other default values : Initialize a Project using AWS Serverless Application Model (SAM) Open your terminal and run $ sam init Choose the AWS quick start template. Opt for a "Hello World" example. Select Java 11 or 17, package type as zip, and use Maven as the dependency manager. Enable logging and monitoring with CloudWatch and XRay. : Rename the project to your preferred name. Rename the Project : Launch IntelliJ, and open the project. Open the Project in IntelliJ : Add Dependencies to pom.xml Add the necessary dependencies to the file. You only need to add DynamoDB, as other dependencies will be automatically included by SAM. pom.xml <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.2</version>
        </dependency>
        <dependency>
          <groupId>com.amazonaws</groupId>
          <artifactId>aws-lambda-java-events</artifactId>
          <version>3.11.0</version>
        </dependency>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.13.2</version>
          <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-dynamodb</artifactId>
            <version>1.12.573</version>
        </dependency>
    </dependencies> Write the Handler Class For the lambda function, edit the handler class generated automatically by sam, and add the following code; this is a simple code, and for real projects, you may want to use more modular code. public class UserRequestHandler implements RequestHandler<Map<String,String>, Map<String,String>> {

    private AmazonDynamoDB amazonDynamoDB;
    private String DYNAMODB_TABLE_NAME = "users";
    private Regions REGION = Regions.US_EAST_1;

    @Override
    public Map<String,String> handleRequest(Map<String,String> input, Context context) {
        this.initDynamoDbClient();
        LambdaLogger logger = context.getLogger();
        logger.log("Input payload:" + input.toString());
        String userId = UUID.randomUUID().toString();

        String firstName= input.get("firstName");
        String lastName= input.get("lastName");

        Map<String, AttributeValue> attributesMap = new HashMap<>();

        attributesMap.put("id", new AttributeValue(userId));
        attributesMap.put("firstName", new AttributeValue(firstName));
        attributesMap.put("lastName", new AttributeValue(lastName));

        logger.log(attributesMap.toString());

        amazonDynamoDB.putItem(DYNAMODB_TABLE_NAME, attributesMap);


        Map<String, String> response = new HashMap<>();
        response.put("id", userId);
        response.put("firstName", firstName);
        response.put("lastName", lastName);

        return response;

    }

    private void initDynamoDbClient() {
        this.amazonDynamoDB = AmazonDynamoDBClientBuilder.standard()
                .withRegion(REGION)
                .build();
    }
} Update SAM Template file The SAM template file plays a pivotal role in building and deploying changes to AWS. Update the file for the project. The key elements to focus on within this file are the Lambda function names and the API Gateway endpoints. They are central to the functionality of your serverless application. AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  pc-aws-user-api

  Sample SAM Template for pc-aws-user-api

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 20
    MemorySize: 128

    Tracing: Active
  Api:
    TracingEnabled: true
Resources:
  UserRequestHandlerLambdaFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: PcAwsUsersApi
      Handler: com.pc.aws.users.UserRequestHandler::handleRequest
      Runtime: java11
      Architectures:
      - x86_64
      MemorySize: 512
      Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
        Variables:
          PARAM1: VALUE
          JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1 # More info about tiered compilation https://aws.amazon.com/blogs/compute/optimizing-aws-lambda-function-performance-for-java/
      Events:
        PcUsers:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /users
            Method: post

  ApplicationResourceGroup:
    Type: AWS::ResourceGroups::Group
    Properties:
      Name:
        Fn::Sub: ApplicationInsights-SAM-${AWS::StackName}
      ResourceQuery:
        Type: CLOUDFORMATION_STACK_1_0
  ApplicationInsightsMonitoring:
    Type: AWS::ApplicationInsights::Application
    Properties:
      ResourceGroupName:
        Ref: ApplicationResourceGroup
      AutoConfigurationEnabled: 'true'
Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  PcAwsUsersApi:
    Description: API Gateway endpoint URL for Prod stage
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/users/"
  UserRequestHandlerLambdaFunction:
    Description: Lambda Function ARN
    Value: !GetAtt UserRequestHandlerLambdaFunction.Arn
  UserRequestHandlerLambdaFunctionIamRole:
    Description: Implicit IAM Role created
    Value: !GetAtt UserRequestHandlerLambdaFunctionRole.Arn Build and Deploy Code using SAM In IntelliJ, open the terminal, and execute the following commands: $ sam build $ sam deploy –guided When prompted, provide the Stack name as "PcAwsUsersApi", and choose the default options. The output will showcase the CloudFormation stack that has been created and will also provide the API Gateway endpoint. Test the API Before testing the API, grant DynamoDB access to the Lambda role created by SAM. Open the Lambda function in the AWS Console. Navigate to "Permissions", and locate the role name. Add the "DynamoDBFullAccess" permission to this role. This step ensures that the Lambda function has the necessary permissions to interact with DynamoDB. Navigate to API Gateway, then follow these steps to obtain the URL for your service: Go to API Gateway in the AWS Console. Select your API. In the left sidebar, click on "Stages." Under "Stages", select the "Prod" stage. In the Stage Editor section, you'll find the "Invoke URL". Copy this URL. Launch the Postman application. Create a POST Request: Set up a new request with the following details: Method: POST URL: Paste the API Gateway endpoint URL obtained earlier. Set Headers: Add any necessary headers (e.g., ). Content-Type: application/json Add Request Body: Create a JSON object representing the user data you want to send in the request. This would be the data you expect to be stored in DynamoDB. Part II Implement CICD Pipeline In this section, we'll demonstrate how to build a CICD pipeline using AWS CodeCommit, CodeBuild, and CodePipeline. The pipeline will initiate a build process that fetches code from the repository, builds it using the build file, and triggers CloudFormation to create the stack. Create CodeCommit Repository Login to AWS, and search for CodeCommit service. Create a new repository in AWS CodeCommit to store your project's source code. Commit Code to Repository In IntelliJ, open the terminal and enter the following commands to commit the code created in Step I $ git init → This initialize local git 
$ git add . → This will stage files
$ git commit -m "commit to CodeCommit" Push Changes to the remote Repo Copy the CodeCommit repo URL from aws console. $ git remote add origin <repo URL>

$ git push --set-upstream origin master --> This will prompt for user/password Create AWS CodeBuild project Set up a CodeBuild project that specifies how to build your application. This includes defining build environments, build commands, and dependencies. Create a S3 bucket to store the build files. Search for AWS Code Build, create a new Project, and provide CodeCommit as a source for code and S3 for artifacts storage To set up CodeBuild, you'll need to create a build specification file named in your project directory. This file will contain the build commands and instructions for CodeBuild. buildspec.yml You can refer to the official documentation for detailed instructions on creating a file. here buildspec.yml version: 0.2

phases:
  install:
    runtime-versions:
      java: corretto11
  pre_build:
    commands:
      - echo Nothing to do in the pre_build phase...
  build:
    commands:
      - echo Build started on `date`
      - sam build
      - sam package --output-template-file pcoutputtemplate.yaml  --s3-bucket com-appsdev-pc-001
  post_build:
    commands:
      - echo Build completed on `date`
artifacts:
  files:
    - pcoutputtemplate.yaml Push the new file to the repository from local. After this, you can build the project to see if the Code is Pulled from Repo and artifacts are built. Create a CodePipeline Create a new pipeline in AWS CodePipeline. Connect the pipeline to your CodeCommit repository as the source. Configure the pipeline to use CodeBuild as the build stage. Add a deploy stage to trigger CloudFormation using the template you created. The pipeline will automatically build and deploy code based on the git commit. Test the Pipeline Open your project in IntelliJ. Make a small change in one of your files, for example, add another logger line. Commit the change to your local Git repository. Push the commit to the CodeCommit repository. $ git branch CR01
$ git checkout CR01
$ git add .
$ git commit -m “CR01”
$ git push --set-upstream origin CR01

You cand also create a pull request in aws code commit, just for simplicity I am merging from local 

$ git checkout master
$ git merge --ff-only CR01
$ git push Go to the AWS CodePipeline console. You should see the pipeline start automatically. It will pull the latest code from the repository, build the project using CodeBuild, and deploy it using CloudFormation. Monitor the progress in the CodePipeline dashboard. This will simulate the process of making changes to your code, pushing them to the repository, and having the CICD pipeline automatically trigger and deploy the updated code. By harnessing AWS API Gateway, Lambda, DynamoDB, CodeCommit, CodeBuild, and CodePipeline, the article demonstrates how to architect a resilient and automated deployment process. Thank you for reading. May your serverless endeavors be met with success and innovation!

A Guide to Building an AWS Serverless API and CICD Pipeline

একটি AWS সার্ভারলেস API এবং CICD পাইপলাইন তৈরির জন্য একটি নির্দেশিকা৷

Ein Leitfaden zum Aufbau einer serverlosen AWS-API und CICD-Pipeline

Una guía para crear una API sin servidor de AWS y una canalización CICD

Un guide pour créer une API sans serveur AWS et un pipeline CICD

AWS सर्वर रहित एपीआई और CICD पाइपलाइन बनाने के लिए एक गाइड

AWS サーバーレス API と CICD パイプラインを構築するためのガイド

AWS 서버리스 API 및 CICD 파이프라인 구축 가이드

Um guia para construir uma API sem servidor AWS e um pipeline CICD

Руководство по созданию бессерверного API AWS и конвейера CICD

AWS Sunucusuz API ve CICD İşlem Hattı Oluşturma Kılavuzu

Hướng dẫn xây dựng quy trình CICD và API AWS Serverless

构建 AWS Serverless API 和 CICD 管道的指南

The networking field is constantly advancing and adapting to new technologies. We have moved beyond the command-line interface and have embraced APIs. With the implementation of principles, we have now entered the era of NetdevOps. The networking industry can be divided into two groups: those who welcome change and those who cling to traditional methods. If you happen to be in the latter group, don't worry; you're not alone! 😀 DevOps Have you ever encountered incorrect configurations or network issues caused by human errors? Do network teams tend to pass the blame around when something goes wrong or when the network becomes unstable? It's high time to move away from traditional networking management methods and adopt more efficient and reliable approaches. The challenge The constantly evolving technology sector is heavily influenced by user trends in today's digital world. As a result, networks need to be able to scale quickly to meet the demands of users. Unfortunately, the Command Line Interface (CLI) is no longer sufficient for modern networking and can create collaboration challenges for network teams. Human errors can cause complicated problems that are difficult to control without automation in networks. Did you know that a study conducted by Cisco revealed that of network changes in the industry are done manually? Unfortunately, this can often lead to misconfiguration in the network. That's why many experts believe that automation is the answer to this problem. By automating network processes, we can not only reduce the risk of errors but also improve efficiency and productivity. By the way, are you still working as a network engineer on the OSI model's third layer? 95% NetdevOps is a new highway for modern networking In the world of traditional networking, there are a multitude of issues that can contribute to a fear-based culture. From erroneous network configurations to time-consuming debugging, network teams often find themselves struggling to keep up with the constant demand for rapid keystrokes and reliable connectivity. However, with the emergence of netdevOps and other innovative technologies, we can now streamline our processes and eliminate many of these challenges. DevOps has revolutionized the IT industry, and its principles are now being applied to networking, giving rise to the concept of " ." By incorporating automation, NetdevOps has simplified complicated processes, enabling us to save time. With the help of DevOps approaches like CI/CD and IaC, network changes can be made incrementally, automated, and performed reliably and efficiently. This has resulted in significant improvements in the speed, accuracy, and consistency of network operations. NetdevOps What is a netdevOps pipeline? NetdevOps is a concept that emphasizes the application of DevOps principles in networking. It is a culture change and a shift in mindset for modern IT teams. The goal is to create a collaborative environment where developers and operations work together to streamline the process of creating and deploying configurations in a production environment. This is achieved through the use of automated tools and techniques, which are collectively referred to as a netdevOps pipeline. By leveraging this pipeline, teams can improve efficiency, reduce errors, and enhance the overall quality of their network infrastructure. Core components of a netdevOps pipeline There are four components similar to a DevOps pipeline : Configuration Build configuration Test configuration Release configuration What’s the value proposition? The implementation of a netdevOps pipeline comes with a host of benefits, including the promotion of daily collaboration amongst network engineers and the eradication of fear-based culture. Automation plays a crucial role in reducing human errors, while agility is of the essence for ensuring customer satisfaction in a rapidly changing digital society. Moreover, the pipeline facilitates continuous testing, thus streamlining the entire process and ensuring excellent network stability. What’s next? Each tech firm has a unique vision and set of goals that it aims to achieve. The success of a company's product pipeline depends on what it wants to offer, and a great plan is the key element to making a difference. A clear strategy serves as the blueprint upon which the pipeline will be created, ensuring that each step is aligned with the overall vision and goals of the company. Are your objectives and goals S.M.A.R.T? As you embark on the journey of building your pipeline, it's essential to start with a clear understanding of your goals and objectives. This will help you answer critical questions along the way and achieve the desired results. For example, ask yourself whether you want to increase your agility, optimize your configurations to deliver faster, or reduce operating expenses. Answering these questions will give you a holistic view of your pipeline and help you make informed decisions at each step of the way. The zero-blame game culture To create an effective pipeline, it's essential to understand your stakeholders. In today's IT organizations, there are various roles and departments, such as QA teams, management, developers, and operations, that play a critical role in the success of the project. Accountability is key to identifying errors and understanding who is responsible for them. With the help of tracking systems and automated technologies, you can easily track the participation of various stakeholders in the pipeline activity. This approach eliminates the blame game and encourages a culture of responsibility and ownership. Effective communication between teams is also vital for optimal achievement. By fostering an open and collaborative environment, you can encourage teamwork, reduce silos, and accelerate the delivery of high-quality software. The ocean of tools Selecting the right tools for your pipeline can be overwhelming with the plethora of options available. It's like 😀 - there are so many options that it's hard to know where to start. selecting your first Linux distro To make the best choice, you must identify your objectives and goals first. Trying to use every tool available may not be the most effective approach, so it's important to choose wisely. Some of the popular tools used in modern software development pipelines include Travis CI, Jenkins, Slack, and Ansible. Let’s get down to business. After careful planning, you now have a comprehensive view of your pipeline. The next step is to bring everything together based on your specific requirements for flawless integration, automated testing, and reliable deployment. This will ensure that your workflows perform as intended, resulting in a smooth and efficient process. There are various requirements for a reliable pipeline, and the following are : Design your pipeline and understand the workflows Implement a version control system and code repository strategies Define your environment management and configuration management A pipeline is a continuous integration and continuous delivery (CI/CD) pipeline that is specifically designed for managing network infrastructure. It automates the process of building, testing, and deploying network configurations. NetDevOps Continuous Integration/Continuous Delivery (CI/CD) is a vital process in modern software development and infrastructure teams. It allows for the automation of code or configuration testing and deployment to production, depending on the components present in the pipeline. By implementing CI/CD processes, teams can streamline their workflows, reduce errors, and deploy software or infrastructure changes with greater speed and efficiency. Let’s discover the best practices : an excellent approach to smoothly push changes to production while avoiding manual errors that could lead to hours of troubleshooting. CI/CD servers to automate the build, test, and deployment of network modifications : an excellent approach to track network configuration and promote collaboration and responsibility amongst teams, revert to prior configurations at any moment. Use a version control system (VCS) : an excellent approach to deploy changes quickly and reliably Automate network configuration deployment using a configuration management tool : enables the speedy construction and destruction of network resources to be automated. Define your network infrastructure as code using an IaC tool : This is a fantastic approach to avoid unexpected network outcomes due to bugs. Test your network configurations : it's the compass for guiding the team through various modifications; therefore, always document. Document your configurations What’s in a netdevOps toolbox Tool When to use Jenkins automate, build, test, and deploy network changes Git a distributed VCS to track network changes and enable collaboration Ansible to automate the deployment of network configurations Terraform to define and provision network infrastructure PyATS to test network configurations, features, and performance Ansible tower to automatically get network documentation generated from devices Conclusion In recent years, network teams have shifted from a culture of fear and blame to one of collaboration. While the command line interface (CLI) is still relevant, it can no longer keep up with the rapidly evolving technology trends. Fortunately, there are a plethora of tools available that can make a significant impact in the industry. However, it's important to note that not all tools can be used in a pipeline, and each company has its own unique needs. Today, there's a zero-blame culture in network teams that encourages collaboration and lacks a fear-based mentality. This shift has enabled teams to learn, fail, and innovate without fear of repercussions. As a result, teams are better equipped to tackle complex challenges and drive meaningful change in the industry. I am the Pablo Picasso of technical writing. In every challenge I face, I see a top-notch article to deliver. Don’t forget to subscribe to my Contributor page for more DevOps content, read my previous articles: : NetdevOps https://hackernoon.com/from-devops-to-netdevops-where-are-we-now-in-networking : Shift left testing https://hackernoon.com/shift-left-testing-why-security-should-start-on-the-left : Starting a devOps journey https://hackernoon.com/moving-to-devops-discover-5-python-modules-to-shine-in-your-new-journey : The blame game culture https://hackernoon.com/devops-from-blame-game-to-collaboration Adopt NetDevOps today!

Networking: Best Practices for a NetDevOps Pipeline

is an open-source platform to programmatically author, schedule, and monitor workflows. It was originally developed by the engineering team at Airbnb but was given to the Apache Software Foundation where it is licensed under Apache 2.0. Apache Airflow Airflow is commonly used in data engineering and data science pipelines to automate the execution of tasks, such as data transformation, loading and analysis. It is also used in other industries, such as finance, healthcare and e-commerce, to automate business processes. Airflow is very flexible with regard to what it can connect to. This includes data lakes, data warehouses, databases, APIs and, of course, object stores. It excels in those use cases that benefit from data-pipelines-as-code, such as: Automation driven data flows Machine learning training models and retraining Backups and snapshots Airflow is written in Python and uses a to represent the workflow. Each node in the DAG represents a task, and the edges between the nodes represent dependencies between the tasks. A DAG does not care about the tasks themselves, just the order, number of retries etc. A complex DAG can become brittle and difficult to troubleshoot, particularly if there are dozens of tasks that must be managed by the architect. directed acyclic graph (DAG) Airflow provides a web interface to manage and monitor the workflows, as well as an API to create, update, and delete workflows. It also has a rich set of features, including support for scheduling, alerting, testing, and version control. MinIO is the perfect companion for Airflow because of its industry-leading performance and scalability, which puts every data-intensive workload within reach. By storing petabytes of data in MinIO buckets, you can create data pipelines in Airflow to process vast amounts of data which is essential for DAGs to run as quickly as possible. Once the processing is done, you can even store the end result back in a MinIO bucket for other tools to consume. MinIO is capable of tremendous performance - a recent benchmark achieved 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. In addition to this, Apache Airflow can also store its logs in a MinIO bucket. This can be helpful in cloud or container orchestration environments, where the local filesystem is ephemeral and the logs can be lost if the machine is terminated or the container is stopped. To minimize the risk of data loss in these environments, it is recommended to use a more durable and cloud-native storage solution like MinIO to store petabytes of data and logs. This ensures that the data is persisted even if the machine or container is terminated. Further, they can be accessed from any location, networking and security permitting. There are several reasons to use MinIO with Apache Airflow: MinIO makes Airflow DAG runs quick with industry-leading that makes it process faster than any other data store at a fraction of the cost. By having all the data required for your DAGs in MinIO, you can realize considerable savings on data storage costs, while at the same time get the best possible performance to cost ratio. performance MinIO relies on to provide high availability and durability for your data and also has to ensure that your data is not lost even in the event of server failures or other disasters. erasure coding configurable data protection policies MinIO can store data in multiple regions and , allowing you to store your data in a location that is close to your users or in a region that is compliant with your data regulations. This can reduce latency and improve the performance of your workflow. replicate it between them MinIO integrates seamlessly with Apache Airflow, allowing you to use the S3 API to store and retrieve your data and other logs. This makes it easy to set up and use MinIO with Airflow, without the need for any additional configuration. MinIO’s mean that it works smoothly with the most widely-implemented software and frameworks. cloud-native integrations Apache Airflow and MinIO Tutorial In this tutorial we’ll show you multiple use cases of Airflow with MinIO. First we’ll show you how to send logs to a MinIO bucket from MinIO DAG runs. Next we’ll create a custom DAG to send objects from an API to a MinIO bucket after post processing. Install Airflow Install Airflow using Pip. You have to install if its not installed and if is not available you also need to symlink it. pip python apt-get install python3-pip

ln -s /usr/bin/python /usr/bin/python3

AIRFLOW_VERSION=2.5.0
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)
pip install "apache-airflow==${AIRFLOW_VERSION}" Install the Amazon provider for connecting to MinIO pip install apache-airflow-providers-amazon Start airflow in standalone mode airflow standalone Configuring Logs Create a bucket in MinIO using mc make bucket mc admin alias minio http://<IP>:9000 minioadmin minioadmin
mc mb minio/airflow-logs Open and add the following settings under /root/airflow/airflow.cfg [logging] [logging]
remote_logging = True
remote_base_log_folder = s3://airflow-logs
remote_log_conn_id = my_s3_conn
encrypt_s3_logs = False For use the bucket name you created in MinIO in the previous step remote_base_log_folder The should match the name of the connection ID we’ll create in the next step. remote_log_conn_id Within the Airflow UI, go to Admin -> Connections Create a new connection with the name . my_s3_conn Enter for the Access Key and Secret Key. minioadmin In Extras, let's set the URL to our local MinIO deployment with the following syntax { "endpoint_url": "http://<ip>:9000" } Now, to test and confirm this is working, Go to DAGs -> example_sensor_decorator and enable this DAG. On the right hand side using the “Play” button, trigger the DAG. After a few seconds, once the DAG is finished running, run the following command to see the logs. For each DAG run, a separate log folder is created. mc ls minio/airflow-logs Using a MinIO bucket to store logs for Airflow DAG runs is just one of the use cases we’re exploring. In the next phase we’ll create a custom DAG to demonstrate more use cases. Creating a custom DAG In this example we’re going to create a custom DAG. What will this DAG do? We’ll connect to the Ghost Blog API Fetch blogs based on certain parameters Back them up to a bucket in MinIO Let’s set up the Framework for the DAG. Import the required python libraries for the DAG framework from airflow.decorators import dag, task Create a schedule for how often to run the DAG @dag(
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
) Next let's create a Task to pull the blogs from the Ghost API and put them in a MinIO bucket. Let’s import a couple of python packages to connect to the Ghost API and MinIO bucket import json
import requests

from minio import Minio
import urllib3 Using the module lets connect to the Ghost API and fetch some blogs requests api_token = "<token>"

page = 1
total_pages = 1

while page <= total_pages:
    api_url = ("https://minio.ghost.io/ghost/api/content/posts/?limit=1&page=%s&key=%s" % (page, api_token))
    response_str = requests.get(api_url)
    response_json = requests.get(api_url).json()

    print(response_json["meta"])
    print(response_json["posts"][0]["url"])

    total_pages = response_json["meta"]["pagination"]["pages"]

    page = page + 1 Putting blogs in MinIO bucket config = {
  "dest_bucket":    "processed", # This will be auto created
  "minio_endpoint": "http://<ip>:9000",
  "minio_username": "minioadmin",
  "minio_password": "minioadmin",
}

# Since we are using self-signed certs we need to disable TLS verification
http_client = urllib3.PoolManager(cert_reqs='CERT_NONE')
urllib3.disable_warnings()

# Initialize MinIO client
minio_client = Minio(config["minio_endpoint"],
               secure=True,
               access_key=config["minio_username"],
               secret_key=config["minio_password"],
               http_client = http_client
               )

# Create destination bucket if it does not exist
if not minio_client.bucket_exists(config["dest_bucket"]):
  minio_client.make_bucket(config["dest_bucket"])
  print("Destination Bucket '%s' has been created" % (config["dest_bucket"]))

      minio_client.fget_object(bucket_name, object_path, object_path)
      print("- Doing some pseudo image resizing or ML processing on %s" % object_path)
      minio_client.fput_object(config["dest_bucket"], object_path, object_path)
      print("- Uploaded processed object '%s' to Destination Bucket '%s'" % (object_path, config["dest_bucket"]))
      minio_client.fput_object(config["dest_bucket"], object_path, object_path)
      print("- Uploaded processed object '%s' to Destination Bucket '%s'" % (object_path, config["dest_bucket"])) Please note the above code is not meant to work “out of the box” but rather give you the idea and show you the path to create your own DAG using your preferred input source; the destination will always be MinIO. Cloud-Native Data Pipelines with Airflow and MinIO The possibilities are endless when you have cloud-native high-performance storage such as MinIO integrated with cloud-native tools such as Airflow. In this example, we’ve shown you some of the basics, such as saving DAG logs in a MinIO bucket and also writing a custom DAG that can talk to any API and perform operations on it, in this case to back up an entire blog to a MinIO bucket. But this is just the beginning, when you go cloud-native you tap into myriad integrated frameworks. With Airflow, you can create any number of multi-cloud pipelines. For instance, you can ETL thousands of terabytes of unstructured data into structured data placed in MinIO that other processes can then read and analyze. You could even use this pipeline as an Image Resizer (similar to ) by taking images of various sizes, resizing them to the size required by your business, and then putting them into another MinIO bucket where they are available for web serving. Orchestrate Complex Workflows Using Apache Kafka and MinIO Give Airflow and MinIO a try for yourself, if you have any questions be sure to reach out to us on ! Slack

Creating Data Pipelines With Apache Airflow and MinIO

The Noonification: The Best Practices For DevOps Pipelines  (10/1/2023)

Hey there, DevOps enthusiast! If you've clicked on this article, it's a safe bet that you're no stranger to the intricate world of Continuous Integration and Continuous Deployment (CI/CD). You understand the importance of streamlining your development process to ensure swift, reliable, and efficient code delivery. If you are not familiar with Continuous Integration (CI) and Continuous Deployment (CD), I advise that you check out two of my articles: & The Best Practices For DevOps Pipelines Confessions of a DevOps Guru: How I Learned to Stop Worrying and Embrace Continuous Integration Continuous Integration and Continuous Deployment (CI/CD) are regarded as pillars for any successful DevOps. Yet, even the most seasoned DevOps engineers can glean valuable insights for further optimizing their CI/CD pipelines. In this in-depth tutorial, I will show you key steps to optimizing your CI/CD pipeline for ultimate efficiency. Choosing the Right CI/CD Tools Would you set off on a road trip without the right vehicle? The same applies to your CI/CD pipeline. The choice of CI/CD tools significantly influences your pipeline's efficiency. Make your tool selection a thoughtful process. Choosing the right CI/CD tools is fundamental to your pipeline success. There is a hoard of options, such as , , , and , and these options offer varying features and integrations that can suit your variable needs. GitLab CI Jenkins Travis CI CircleCI Consider factors like: : What features are important to you? Do you need a tool with support for multiple languages and frameworks? Do you need a tool with advanced features like security scanning and code coverage analysis? Features How easy is the tool to set up and use? Do you have the resources to maintain and troubleshoot the tool? Ease of use: : Will the tool be able to scale to meet your future needs? Scalability : How much does the tool cost? Are there any open-source or free alternatives? Cost You might also want to explore Infrastructure as Code (IaC) tools like Terraform and Ansible for managing infrastructure alongside code changes. Streamlining Your Workflow Once you have selected the right CI/CD tools, you need to simplify your pipeline by breaking it down into manageable phases. A streamlined workflow dismantles bottlenecks and accelerates the delivery process. Each phase should have transparent criteria to meet. Automation should be the cornerstone of your workflow, automating everything from code commits to deployment. To further refine your workflow, consider adopting version control strategies like or . GitFlow GitHub Flow Here are some additional best practices to help streamline your workflow: Store your CI/CD pipeline configuration as code: It is standard practice that your pipeline configuration is stored in a text file, such as a YAML file. This is like having a recipe for your pipeline; it makes things like change tracking and management easy for new contributors. : The whole essence of CI/CD is to automate. This means that you should automate everything that can be automated, especially tasks like building your code, running tests, and deploying it. Automate everything : Many CI/CD tools support parallel execution, which means that multiple steps in your pipeline can be executed at the same time. Like having multiple chefs prepare different dishes, this can significantly speed up your build and deployment times. Use parallel execution : A continuous monitoring tool like a CI/CD dashboard, can help you to monitor your application's performance and health in production. This can help you to identify and resolve problems before they impact users. Use a CI/CD dashboard Additionally, you can embrace containerization technologies such as Docker and Kubernetes to facilitate deployment and scalability. Automated Testing and QA Automation should be used for your testing strategy. Automated testing can be very instrumental for any team that wants to streamline the Continuous Integration and Continuous Deployment process. Automated testing can help you identify bugs early, ensure that your software meets the team's standards, and release new features. Why rely on manual testing when automation delivers speed and precision? -- Random DevOps Guy When choosing an automated testing tool, make sure to consider the following factors: : What features are important to you? Do you need a tool that supports multiple languages and frameworks? Do you need a tool with advanced features like code coverage analysis and security scanning? Features : How easy is the tool to set up and use? Do you have the resources to maintain and troubleshoot the tool? Ease of use : Will the tool be able to scale to meet your future needs? Scalability : How much does the tool cost? Are there any open-source or free alternatives? Cost Here are some popular testing tools to integrate into your pipeline: , , , TestNG, and Cypress. JUnit Appium Selenium Advocate for Continuous Testing, a practice that subjects every code change to rigorous scrutiny, eliminating regressions and enhancing both speed and reliability. Here are a few tips for using automated testing and QA in your CI/CD pipeline: Use a test pyramid: A test pyramid is a way to organize your automated tests into different layers. The base of the pyramid is unit testing, followed by integration testing, and then end-to-end testing. This structure helps to ensure that you have a comprehensive test suite that covers all aspects of your software. Use continuous testing: This is a technique, where you run automated tests continuously throughout the development process. This helps to ensure that your software is always in a tested state. Parallelizing Builds Parallelizing builds is a great way to improve the efficiency of your CI/CD pipeline. By running multiple builds at the same time, you can improve scalability, reduce build times, and increase reliability. If you are looking to improve the efficiency of your CI/CD pipeline, then parallelizing builds is a great place to start. – Random DevOps enthusiast Keep in mind that there are two ways to parallelize builds: : Option 1 Parallel Builds on a Single Machine This involves running multiple builds on the same C/CD hardware and the same computer. This can be done by writing scripts or using special tools like GNU Parallel. Option 2: Parallel Builds Across Multiple Machines This option involves running multiple builds using different nodes or agents. Several cloud-based CI/CD CircleCI or GitHub Actions provide features that allow for parallelization of builds. Alternatively, organizations usually establish and manage their dedicated CI/CD infrastructure, orchestrating builds across a network of machines. When executing parallel builds, consider the following factors: : The first step to parallelizing builds is to identify the tasks in your build pipeline that can be run independently. This typically includes tasks such as unit testing, integration testing, and code deployment. Identify independent tasks : Once you have identified the independent tasks in your build pipeline, you can use a parallel execution tool to run them simultaneously. There are many different parallel execution tools available, such as GNU Parallel, Make, and Maven. Use a parallel execution tool : If you have multiple CI/CD agents, or if your builds are resource-intensive, then you may want to use a cloud-based CI/CD service to parallelize builds. Cloud-based CI/CD services typically have a large pool of CI/CD agents that you can use to scale your builds up or down as needed. Use a cloud-based CI/CD service Monitoring and Feedback Loops To ensure a smoothly run CI/CD pipeline, employ monitoring and feedback loops. This allows you to identify problems and also helps you track your builds and deployment progress. These checks act as gatekeepers that halt deployments when any critical issues are detected and stop bad code from getting to the main production. Monitoring paves the way for early issue detection, upholding pipeline efficiency. With a lot of monitoring tools available at your disposal, consider these factors when choosing the right one: : What features are important to you? Do you need a tool that can monitor builds, deployments, and infrastructure? Do you need a tool that can provide real-time monitoring and alerts? Features : How easy is the tool to set up and use? Do you have the resources to maintain and troubleshoot the tool? Ease of use : Will the tool be able to scale to meet your future needs? Scalability : How much does the tool cost? Are there any open-source or free alternatives? Cost Some of the popular monitoring tools available are , , Datadog, and New Relic. Prometheus Grafana Once you have chosen a monitoring tool, you need to integrate it into your CI/CD pipeline. This involves configuring your CI/CD tool to collect metrics and send them to you and your team. Security Checks The repercussions of security breaches can be devastating. Security must be an integral cog in your CI/CD machine. Security checks can help you find threats and fix security vulnerabilities early in development, helping you prevent security vulnerabilities from being exploited by attackers. This is why checks are an essential part of any CI/CD pipeline. Some key factors to consider when choosing a security checking tool: : What features are important to you? Do you need a tool that can scan for common security vulnerabilities? Do you need a tool that can scan for more advanced security vulnerabilities? Features : How easy is the tool to set up and use? Do you have the resources to maintain and troubleshoot the tool? Ease of use : Will the tool be able to scale to meet your future needs? Scalability : How much does the tool cost? Are there any open-source or free alternatives? Cost Some of the popular security checking tools are , , and Synk SonarQube OWASP ZAP You can also use security checks to create feedback loops. For example, you can configure your security checking tool to send alerts to your team members when security vulnerabilities are found. This will help to ensure that security vulnerabilities are fixed quickly. It is good practice to implement automated security scans and vulnerability assessments to be sure of the integrity of your code and all its dependencies. Continuous Improvement Change is the only constant==.== Going forward, I want you to acknowledge that your CI/CD pipeline is a living entity. Continually improve your CI/CD pipeline, periodically evaluate, and refine your processes. This can be achieved by making small, incremental changes or even larger, disruptive changes. The goal is " "-- to make your CI/CD pipeline as efficient as possible. continuous improvement Listen to feedback from your team and stakeholders. Cultivate a culture of continuous improvement to remain at the forefront of the changing world of DevOps. Use the (Plan-Do-Check-Act) as your compass for iterative pipeline perfection. Remember, it's not about reaching an endpoint; it's about embarking on a perpetual quest for excellence. Deming Cycle Conclusion Congratulations, DevOps Guru! If you follow these tips and guidelines, you can successfully create a CI/CD pipeline that is efficient, reliable, and secure. By continually optimizing your CI/CD pipeline, you're paving the way for swifter, more dependable software delivery. Tip: Keep in mind that it's not about achieving perfection; it's about the relentless pursuit of perfection. The effort invested in refining your CI/CD pipeline leads to expedited, reliable software delivery. Keep optimizing and revel in the DevOps ethos! Go forth, and may your pipelines flow ever more efficiently! : Read Next Continuous Integration (CI) is the secret sauce that brings sanity to the chaos of DevOps. CI systems like Jenkins, Travis CI, and CircleCI watch your code repository like a hawk. It's like having a personal assistant who diligently checks your code for errors. CI doesn't just find problems; it also fixes them.... Read More

How to Optimize Your CI/CD Pipeline for Maximum Efficiency 

The Noonification: The Best Practices For DevOps Pipelines  (9/23/2023)

DevOps pipelines are the core of modern software development. DevOps pipelines are automated processes carefully structured to enable seamless integration, testing, and deployment of code changes. The significance of a smoothly run DevOps pipeline cannot be overstated; they enable rapid, reliable software releases, enabling agility and competitiveness in today's tech landscape, something that has never been seen before. In this article, I will give you insights, best practices, and suggestions for DevOps pipelines that help many organizations harness their full potential. Understanding DevOps Pipelines DevOps pipelines are the lifelines of software development, symbolizing the principles of collaboration, automation, and continuous improvement. The concept of pipelines refers to an intricate set of processes and tools that work together to ensure an efficient flow of code from the development phase to the deployment phase. Key Components Of DevOps Pipeline: These are four of the major pillars of DevOps Pipelines: Continuous Integration (CI) Continuous Delivery (CD) Automation And Orchestration Deployment Strategies Benefits Of Implementing DevOps Pipelines Using DevOps pipelines, also known as software delivery pipelines, brings a lot of benefits, including accelerated development cycles, the elimination of human errors, improved collaboration, streamlined testing, and improved software quality. It is also important to note that, it enables the practice of continuous improvement, thereby giving organizations a competitive edge in this modern, fast-paced world of software development. By automating repetitive tasks and optimizing processes, DevOps pipelines empower teams to focus on innovation and delivering value to end-users. Planning Your DevOps Pipeline The success of any DevOps pipeline is directly proportional to the meticulous planning made in alignment with your organization's plans, requirements, and goals. This is a very important stage because it serves as the foundation upon which the entire pipeline will be constructed. Here are a few tips you can follow or check to ensure a well-planned DevOps pipeline: Accessing Your Organization's Needs Before you really plan anything, start by conducting a comprehensive assessment of your organization's current software development and deployment processes. Identify manual processes and bottlenecks where automation can speed up and bring improvements. Defining Pipeline Objectives and Goals To define your organization's pipeline objectives and goals, ask questions like, "What does your organization want to achieve with your DevOps pipeline?" " Do you want to deliver faster, improve code quality, or lower costs?" " Are you looking to accelerate time-to-market?" " Do you want to reduce operations costs?" Getting the right answers to these questions can help provide a clear direction for your pipeline implementation. ldentifying Key Stakeholders Identify who will be involved in the pipeline, including developers, operations personnel, QA teams, and management. Because effective communication and efforts between all parties involved are also crucial for success. Choosing the Right Tools and Technologies Selecting the right DevOps tools and technologies that suit your organization's needs and requirements is paramount. Consider factors such as integration capabilities, scalability, and community support. Popular choices include Jenkins, GitLab CI/CD, Kubernetes for container orchestration, and infrastructure-as-code tools like Terraform. Designing An Effective DevOps Pipeline Now that you have completed the planning of your pipeline's structure, it is time to design an effective DevOps pipeline for the needs of your organization. Implementing your pipeline design based on your plan sets the groundwork for seamless integration, automated testing, and reliable deployment, ensuring that your software development process operates like a well-oiled machine. Focus on these key elements to ensure a stable foundation: Creating a High-Level Pipeline Architecture Define the overarching structure of your pipelines. Strictly outline how code will flow from development through various phases up until deployment. Consider the integration points, dependencies, and potential parallelization of tasks. Establishing Version Control and Code Repository Strategies Version control is a fundamental aspect of DevOps. Select a version control system (e.g., the popular Git) and define branching and merging strategies. Ensure that code repositories are organized, and access controls are in place to maintain code integrity and security. Defining Environment Management and Configuration Implement infrastructure-as-code (laC) principles using tools like Terraform or Ansible to define and provision environments. Efficiently managing the environment will prove pivotal for continuous testing and deployment. Continuous Integration (CI) After designing a tailored overarching structure for the DevOps pipeline, the next thing to do is to implement Continuous Integration. This is how to go about this: Setting Up a Cl Server Begin by configuring a Cl server such as Jenkins, Travis CI, or CircleCI.  This server will monitor your code repository for changes and trigger automated builds and tests. Automating Build and Test Processes Automate the process to compile code, package artifacts, and prepare them for deployment. Automate testing by running unit tests, integration tests, and other relevant checks. This automation ensures consistency and reliability in the build and test phases. Handling Code Branches and Merges Establish clear rules that simplify code merging and branches. Make new branches for new development, and ensure they are merged back into main production as quickly as possible. By employing Cl, you ensure that code changes are continuously verified and integrated efficiently, allowing you and your team to deliver software with speed and reliability. Continuous Deployment Continuous deployment, often regarded as an extension of Continuous Integration (CI) extends CI by automatically deploying code changes into main production after they passed the testing stage. Here are some essential practices for effective CD implementation: Implementing Automated Deployment Ensure that the deployment process is automated so that releases are error-free and consistent. Use deployment scripts, containerization (e.g., Docker), and orchestration tools (e.g., Kubernetes) to streamline the deployment of applications and services. Managing Configuration Across Environments Make sure that environment-specifics are kept separated from the main codebase. Use environment variables, configuration management systems (e.g., Ansible), and container orchestrators to manage configurations for different environments (e.g., development, testing, and production). Blue-Green and Canary Deployment Strategies Blue-green deployments involve using two identical environments (blue and green) and switching traffic between them to minimize downtime. Canary deployments gradually roll out changes to a subset of users. Both strategies reduce risk and allow for quick rollbacks in case of issues. Testing And Quality Assurance Testing helps ensure that new code meets quality standards. Here are key practices for effective testing and quality assurance in DevOps: Types of Testing in DevOps Pipelines: DevOps Pipelines Incorporate Various Types of Testing, Including: Unit Tests Integration Testing Regression Testing Performance Testing Security Testing User Acceptance Testing(UAT) Automated Testing Tools and Frameworks Take advantage of the automated testing tools available for your application stack. For example, Selenium for web testing, JUnit for Java applications, and pytest for Python. Automating tests ensures consistency and quick feedback. Monitoring and Reporting Test Results Employ monitoring and tracking tools to track test results and performance metrics. Tools like Prometheus and Grafana can provide real-time visibility into the health of your applications. Establish clear reporting practices to quickly identify and address issues. Automatation and Orchestration Automation and orchestration enable you to achieve consistent, rapid, and error-free delivery of software and infrastructure changes. Here are some practices for successful automation and orchestration: Creating Automated Workflows Design end-to-end automated workflows that cover everything from code integration and testing to deployment and monitoring. Define, schedule, and execute these workflows with tools like Apache Airflow and GitHub Actions. Managing Infrastructure as Code (laC) Implement laC principles by defining infrastructure configurations using code. Tools like Terraform and Ansible enable you to version, provision, and manage infrastructure reliably. This approach ensures that infrastructure changes are automated, repeatable, and documented. Orchestrating Deployment Pipelines Orchestration involves coordinating multiple tasks and processes in a DevOps pipeline. Container orchestration platforms like Kubernetes and Docker Swarm are invaluable for managing containerized applications, ensuring they are deployed, scaled, and maintained efficiently. Conclusion As you already know from this quick lesson, the DevOps pipeline is essential to modern software development. It enables developers to rapidly deliver high-value software efficiently while also reducing human error. In this article, I have tried to show you essential DevOps pipeline best practices, from planning and design to testing, automation, and orchestration. These practices lay the very foundation for success in the fast-paced world of software development. You must also remember that as DevOps continues to evolve, organizations that invest in optimizing their pipelines will be well-positioned to deliver innovative, high-quality software that meets the demands of a dynamic digital landscape.

The Best Practices For DevOps Pipelines 

DevOps পাইপলাইনগুলির জন্য সর্বোত্তম অনুশীলন৷

Die Best Practices für DevOps-Pipelines

Las mejores prácticas para canalizaciones de DevOps

Les meilleures pratiques pour les pipelines DevOps

DevOps पाइपलाइनों के लिए सर्वोत्तम अभ्यास

DevOps パイプラインのベスト プラクティス

DevOps 파이프라인 모범 사례

As melhores práticas para pipelines DevOps

Лучшие практики для конвейеров DevOps

DevOps İşlem Hatları İçin En İyi Uygulamalar

Các phương pháp thực hành tốt nhất cho quy trình DevOps

DevOps 管道的最佳实践

Hi, HackerNoon! My name is Sofia, and I am a DevOps/Cloud engineer. I decided to participate in the contest by HackerNoon and Aptible. In this article, I will talk about the features of the DevSecOps pipeline and the concept of Shift Left. You will learn about the main stages of the DevSecOps pipeline, automated security checks in software development, and free and open-source tools. You'll also find tips to help you detect vulnerabilities earlier and improve application security. This article will help you assess the maturity of your DevSecOps pipeline, develop a roadmap for its development, choose the right tools for each task, and better understand how to manage projects following the DevSecOps philosophy. Shift Left Concept The main goal of the DevSecOps methodology is to introduce automated security checks into the DevOps pipelines, i.e., into the software development process itself. Not so long ago, information security specialists conducted tests after completing the development process. This approach is inefficient - if errors are discovered during security testing, the entire development cycle has to be restarted. This is time-consuming and expensive. Take a look at the image below. You can see that the cost of error correction increases with each step. It costs almost nothing to fix security issues discovered during development and functional testing. All necessary changes can be made to the source code immediately and sent for rechecking. Fixing problems at the artifact build stage is at least twice as expensive. You will have to make changes to the build process, for example, edit the Dockerfile, which means you will need the help of DevOps engineers. If an error is detected during integration testing, it will be 10 times more expensive to fix it. In this case, you must restart the development cycle and involve developers, DevOps engineers, and testers. Security problems detected at the deployment stage can lead to user data leaks. The company can receive millions of dollars in fines and damage to its reputation, which means the cost of a mistake increases hundreds of times. Hence, the main conclusion is that safety checks should be performed as early as possible. If you visualize it, move them to the left side of the Pipeline. This is the concept of Shift Left, which security experts love to talk about so much. DevSecOps Pipeline Structure DevSecOps pipelines are an automated chain of processes and tools that build, test, and deliver applications in a production environment and secure them at every stage. The image above shows the DevSecOps Pipeline structure with all the main phases of security checks. Here are a few words about each of them: Pre-commit. It implies checking the security of source code before the developer commits the code to the version control system. This check allows you to detect unencrypted confidential information such as passwords or API keys. Pre-build. It implies checking the security of source code after it has entered the version control system. The tools perform static analysis of source code and its dependencies to detect common vulnerabilities, such as many of the list. OWASP Top Ten Post-build. It is the security check of an artifact built from source code, such as a Docker file, RPM package, or JAR archive. The tools analyze the application environment and dependencies, finding outdated versions of packages and libraries that already have patches in newer versions. Test-time. It implies testing the security of an application running from a collected artifact. Tools in this phase try to "break" the application by simulating the attackers' actions. Post-deployment. It implies checking the security of an application after it has been deployed in a production environment. The tools monitor open registries of known vulnerabilities in real time and notify when a threat to the application is detected. Next, let's take a closer look at what these checks are and what tools are used to perform them. Pre-commit Checks Pre-commit checks allow you to analyze the source code on the developer's machine before committing changes to the local copy of the repository. If any of the statements returns an error, the commit is not performed. In this case, the developer must fix the mistake and commit again. This check avoids the situation when unchecked code, for example, with unencrypted secrets, gets into the repository. For performing pre-commit source code checks, it is necessary to install tools on developers' local machines. This approach has not only advantages but also disadvantages. For example, environmental differences: different versions of the instruments, various operating systems, and even processor architectures. If developers use different versions of pre-commit tools, the results of checking will differ, and this may make it challenging to work together. Consider this when implementing pre-commit checks within a team or across the company. Tools for Pre-Commit Checks Many open-source tools can be used to set up pre-commit checks: Gitleaks git-secrets Trivy Secret Scanning Whispers git-all-secrets detect-secrets gitty leaks These are great tools for pre-commit checks. Pre-Build Checks In the pre-build phase, White Box Testing is performed. These checks are used to analyze the source code, like in the previous step. In this case, the application is a "white box" because we know and understand its architecture and internal structure. There are several types of pre-build checks: Secret Detection Static Application Security Testing (SAST) Software Composition Analysis (SCA) Now, let’s discuss them in detail. Pre-build Secret Detection Check Secret Detection is an automated check that detects unencrypted sensitive information: unencrypted secrets in source code that have entered a version control system. Pre-build checks are performed after the source code has entered the repository of the version control system. Therefore, all unencrypted sensitive data in the storage can potentially be accessed by attackers, for example, if the repository's contents are leaked. In addition, the process of implementing secret-detection checks may occur not from scratch but in an evolving project. In this case, unencrypted secrets may be found in old commits and different repository branches. To solve these problems, we need to do many different things. For example, we must clean repositories of unencrypted secrets or implement various secret management tools like . Hashicorp Vault Tools for Secret Detection Secret Detection can be configured using , , or . However, it is best to use services provided by well-known CI/CD platforms, for example, . Gitleaks git-secrets detect-secrets GitLab Secret Detection Pre-Build SAST Check Static Application Security Testing (SAST) is a test when analyzers do not just check syntactic correctness but also measure code quality with the help of unique metrics. The main task of SAST scanners is security testing. In particular, SAST-analysers contain source code for common vulnerabilities, for example, some of the lists. It is essential to say that SAST scanners find not only the vulnerability itself but also the code fragment that caused it. OWASP Top Ten SAST analysis is called White Box Testing because the analyzer can access the application's internal structure. It is important to note that analyzers check the source code without running it, so they may generate false positives and fail to detect some vulnerabilities. For this reason, you should not limit yourself to SAST analysis only. It is better to approach the issue comprehensively and use different types of analysis: SCA, DAST, IAST, and OAST. SAST Tools Proprietary solutions: e SonarQub Checkmarx SAST Solar appScreener The free solution is GitLab SAST. Pre-Build Source SCA Check Software Composition Analysis (SCA) is a methodology that secures applications by analyzing their open-source components. SCA scanners look for outdated dependencies that contain known vulnerabilities and exploits. In addition, some SCA tools can determine the license compatibility of components and the risks of license infringement. Source SCA analyses the source code and, more specifically, the application dependencies defined in the source code. This analysis is often referred to as Dependency Scanning. SCA allows you to detect an issue where a vulnerability in one component can lead to security problems in all applications. A good example is . This vulnerability was discovered in late 2021 in Log4j, a popular Java logging framework. It became one of the most feared vulnerabilities because it allowed attackers to execute arbitrary Java code on hundreds of millions of devices. Log4Shell SCA tools is an open-source solution. Trivy Commercial solution with free pricing plans: FOSSA Mend SCA As part of GitLab (available in Ultimate-version only), you can use . GitLab Dependency Scanning Post-Build Checks: Binary SCA The post-build phase comes after artifacts have been built from source code in the CI Pipeline: a Docker image, an RPM package, or a JAR archive. With post-build checks, you can analyze the dependencies in these binary artifacts. Binary SCA analysis involves inspecting binary artifacts such as Docker images, RPM packages, and JAR/WAR/EAR archives. It is also often referred to as Container Scanning. It makes sense to run it when the binary artifacts are built, i.e., not before the post-build stage. There are some exciting features when working with Docker images. Binary SCA analyzers check layers of Docker images and look for them as hash sums in public vulnerability databases such as or . National Vulnerability Database (NVD) Snyk Vulnerability DB Some of the analyzers can not only find vulnerable components but also suggest a fix, for example, by specifying the minimum required version of a component with an already fixed vulnerability. Examples of Popular Binary SCA Analyzers The free solution is . GitLab Container Scanning Open Source solutions: Trivia Clair Grape Quay JFrog Xray Docker Hub Docker Registry with built-in analyzers - . Harbour Test-Time Checks: DAST, IAST, and OAST At this stage, dynamic Gray Box Testing and Black Box Testing are performed. When the application's internal structure is partially or wholly unknown, these security checks are performed by emulating the actions of an attacker who finds vulnerabilities and tries to "break" the application by exploiting them. Let`s briefly discuss each of them: DAST, IAST, and OAST. Test-Time DAST Check DAST scanners automatically test applications by simulating external attacks through various vulnerabilities. The application is a black box for the analyzer; nothing is known about it. For DAST checks, it is necessary to have a running instance of the application available for the scanner. Moreover, the closer the parameters of the test instance of the application are to the production installation, the fewer false positives there will be. For example, suppose you have deployed a test application instance accessible only via HTTP protocol, but in production, the application is accessible only via HTTP protocol. In that case, the DAST scanner will generate some errors related to the lack of traffic encryption between the client (analyzer) and the server (application instance). Examples of DAST Tools Tools that are worth your attention: — only available in the Ultimate version GitLab DAST — an open-source solution, which is also used in GitLab DAST OWASP Zed Attack Proxy (ZAP) Acunetix Fortify WebInspect HCL Security AppScan Synopsys Managed DAST (Web App Scanning) Tenable.io Veracode Dynamic Analysis Great, move on. Test-Time IAST Check IAST (Interactive Application Security Testing) is a Gray Box Testing that combines the advantages and strengths of White Box Testing SAST and Black Box Testing DAST. To collect all the advantages and eliminate the disadvantages of SAST and DAST methods, developers invented IAST - a hybrid mechanism combining these methods. It increases the accuracy of dynamic testing because it gives more information about the application operation through code analysis. It is worth remembering that besides its advantages, IAST has limitations. The most basic one is the dependence on the language in which the application under test is written. IAST solutions work at the application level and require access to the executable code to analyze its behavior. This means that IAST solutions must be able to understand the programming language in which the application is written. It should be noted that support for a particular IAST solution may be better implemented for some languages than others. IAST analysis also takes a long time. It depends on many factors, such as the size and complexity of the application, the number of external dependencies, and the performance of the IAST tool used. Besides, the analysis process does not scan the code itself but checks only those fragments that are directly executed. Therefore, IAST analysis is best combined with functional testing when you can test as many application scenarios as possible. Examples of IAST Tools Here are great tools for you: Synopsys Seeker Veracode Interactive Analysis Microfocus Fortify WebInspect Checkmarx CxIAST Contrast Assess Hdiv Detection (IAST) Great, move on. Test-Time OAST Check OAST (Out-of-band Application Security Testing) is a technique developed by and is an extension of DAST that finds vulnerabilities that DAST does not see, such as log4shell. PortSwigger Examples of OAST tools I recommend you: Burp Suite OWASP ZAP with OAST plugin OAST services , , and BOAST TukTuk interact Move on. Post-Deploy Checks This is an essential stage in ensuring application security and operability. Unlike pre-commit checks, which are performed at the development stage, post-deploy checks allow you to identify possible problems already during application operation in the natural environment. Monitoring the Safety of Artifacts For detecting new vulnerabilities in time, it is necessary to perform regular artifact checks, including after an application has been deployed. This can be done using special repositories or tools, such as , Trivy, GitLab Container Scanning, or developing your integration module. Snyk Container Application Security Monitoring Another way to ensure security is to monitor the application itself. There are several technologies available for this purpose: Web Application Firewall (WAF) is a traffic filtering tool. It works at the application layer and protects web applications by analyzing HTTP/HTTPS traffic. RASP (Runtime Application Self-Protection) is a technology that detects and blocks attacks on an application in real time. It adds a defense function to the runtime environment, allowing the application to self-protect automatically. The main advantage of RASP over WAF is that it understands how the application works and detects attacks and illegitimate traffic. RASP can see not only typical attacks using signature matching but also attempts to exploit zero-day vulnerabilities by reacting to anomalies. Unlike WAF, RASP works within and with the application. WAF can be fooled. If an attacker changes the application code, WAF cannot detect it because it does not understand the execution context. RASP has access to diagnostic information about the application, can analyze it, detect anomalies, and block attacks. RASP technology has a specialty of focusing on preventing attacks by default. The system does not require access to the source code. RASP Fools I recommend you: Fortify Sqreen is an open-source solution developed by Baidu OpenRASP Signal Sciences Jscrambler Hdiv Imperva Great, move on. Visualization of DevSecOps P ipelines Results and Vulnerability Management At different stages of the Pipeline, I use many Application Security Testing/DevSecOps analyzers. Each of them generates a report in its own format, and some of them generate so-called False Positives. Because of this, it isn't easy to analyze the security of an application as a whole. To effectively analyze vulnerabilities and manage the remediation process, specialized Vulnerability Management tools, often referred to simply as , are used. Security Dashboards Security Dashboards solve the problem by delivering the results of DevSecOps analyses in a unified and easy-to-analyse form. In this way, security engineers can see what issues exist, which ones are most critical, and which ones need to be addressed first. DevSecOps Tools A wide range of tools are usually used as Security Dashboards: for example, classic SOAR/SIEM systems and the specialized DevSecOps orchestrator AppSec.Hub from Swordfish Security or the open-source vulnerability management toolkit DefectDojo. DefectDojo is a widespread tool. It is widely used even in enterprise companies. However, despite its popularity and solid age, this tool occasionally has some initial-level defects when contributors break the basic functionality in the commit. However, what is nice is that DefectDojo developers and maintainers help to sort out the complexities. DefectDojo developers are very responsive people - we recommend contacting them directly via Issues on GitHub. Shift Left Concept: Let's Summarize Once again, here is a quick recap of what was in the article. I explained what the Shift Left concept is and how it helps reduce the cost of bug fixes and development time: the earlier in the development process you start security checks, the better. Then, I deconstructed the DevSecOps Pipeline structure and looked at what security checks are performed at each stage of the Pipeline: Pre-commit. It implies checking the safety of source code before it enters the version control system. Pre-build. It is a check of source code security in the version control system (Secret Detection, SAST, SCA). Post-build. It is a security check of an artifact built from source code (Source SCA, Binary SCA). Test-time. It implies testing the security of the application that is running from the collected artifact (DAST, IAST, and OAST). Post-deploy. It implies checking application security after deployment in the production environment (WAF, RASP). Now, you understand how the DevSecOps Pipeline works. Now, you can assess the maturity of your DevSecOps pipelines, develop a roadmap for its development, and choose the right tools for each task.

How to Turn A DevOps Pipeline Into a DevSecOps Pipeline: A Shift Left Concept Overview

কীভাবে একটি DevOps পাইপলাইনকে একটি DevSecOps পাইপলাইনে পরিণত করবেন: একটি শিফট বাম ধারণা ওভারভিউ

So verwandeln Sie eine DevOps-Pipeline in eine DevSecOps-Pipeline: Eine Übersicht über das Shift-Left-Konzept

Cómo convertir una canalización de DevOps en una canalización de DevSecOps: una descripción general del concepto de cambio a la izquierda

Comment transformer un pipeline DevOps en pipeline DevSecOps : un aperçu du concept Shift Left

एक DevOps पाइपलाइन को DevSecOps पाइपलाइन में कैसे बदलें: एक शिफ्ट लेफ्ट कॉन्सेप्ट अवलोकन

DevOps パイプラインを DevSecOps パイプラインに変える方法: シフトレフトの概念の概要

DevOps 파이프라인을 DevSecOps 파이프라인으로 전환하는 방법: Shift Left 개념 개요

Como transformar um pipeline DevOps em um pipeline DevSecOps: uma visão geral do conceito Shift Left

Как превратить конвейер DevOps в конвейер DevSecOps: обзор концепции сдвига влево

DevOps Boru Hattını DevSecOps Boru Hattına Dönüştürme: Sola Kaydırma Konseptine Genel Bakış

Cách biến quy trình DevOps thành quy trình DevSecOps: Tổng quan về khái niệm chuyển hướng sang trái

如何将 DevOps 管道转变为 DevSecOps 管道：左移概念概述

Stories, News & Mentions about PACELINE

Paceline’s Stories on HackerNoon