Building a CI/CD Pipeline with AWS, K8S, Docker, Ansible, Git, Github, Apache Maven, and Jenkins

In my previous article, I delved into the fascinating world of microservices - . This was the beginning of my comprehensive article series on microservices and their patterns. Microservice Architecture Patterns Part 1: Decomposition Patterns While the logical step would be to continue with Part 2 of this series, I have decided that the next thing that I would like to explore and tell you about is the vital process of delivering these microservices to the end user seamlessly. Content Overview Motivation [Module 1]: AWS EC2 Virtual Server [Module 2]: Jenkins Server [Module 3]: Git and Github [Module 4]: Apache Maven [Module 5]: Docker [Module 6]: Ansible [Module 7]: Kubernetes Conclusion About the Author Motivation CI/CD is a technique for delivering apps to customers, achieved by adding automation to different stages of app development. I believe that grasping CI/CD (Continuous Integration and Continuous Deployment) can empower developers to gain a better understanding of how backend project artifacts exist beyond the boundaries of the project repository. This comprehension can also create a fundamental shift in a developer's perspective. Instead of merely viewing their work as lines of code, they can start to embrace the broader context of their project as a valuable product. In this article, we aim to demystify the CI/CD process through practical application. We'll take you through a step-by-step tutorial, breaking it down module by module, where you'll build a CI/CD pipeline manually. To do this, we'll harness the power of contemporary DevOps tools like and . So, let's begin this journey! AWS, Docker, Kubernetes, Ansible, Git, Apache Maven, Jenkins [Module 1]: AWS EC2 Virtual Server This module is dedicated to the creation of an AWS EC2 Virtual Server instance. As part of this article, you will be setting up three EC2 instances for Jenkins, Ansible, and Kubernetes. For now, you can proceed with the next modules and revisit this module in "[module 2]: Jenkins", “[module 6]: Ansible" and "[module 7]: Kubernetes" sections. Step 1: Create an AWS Account Go to https://aws.amazon.com. Click the button . Create an AWS Account Follow the instructions on the create account web page. Step 2: Sign In to your AWS Account Go to https://console.aws.amazon.com/console/home. Click the button. Sign In Enter all necessary credentials on this web page. Step 3: Find EC2 Virtual Server Find EC2 in the search box. Choose EC2 Virtual Server by clicking . EC2 Service Click the button . Launch Instance Step 4: Configure “Name and tags“ section Go to the section. “Name and tags” Provide a name for a new AWS EC2 Virtual Server instance in the section. “Name” You can also add additional tags for your virtual server by clicking . ”Add additional tags” Step 5: Configure “Application and OS Images (Amazon Machine Image)“ section Go to the section. "Application and OS Images (Amazon Machine Image)" To play with the virtual server for : FREE Select the operating system for your virtual server - . Amazon Linux In the section, select a machine with the . Amazon Machine Image (AMI) Free tier eligible tag Step 6: Configure “Instance type“ section Go to the section. ”Instance type” To play with the virtual server for : FREE Select a type with the in the section. Free tier eligible tag Instance type For me it is . t2.micro (Family: t2 1cCPU 1 GiB Memory Current generation:true) Step 7: Configure “Configure storage“ section Go to the section. ”Configure storage” To play with the virtual server for : FREE Do not change default settings. Free tier eligible customers can get . 30 GB of EBS General Purpose (SSD) or Magnetic storage Step 8: Configure “Network settings“ section Go to the section. “Network settings“ You need to set up the security of your virtual server. To do this, Click on the button. “Create security group” Add the name of your new security group in the section. “Security group name” Add a description of your new security group in the section. “Description” By default, your virtual server is accessible via ( ). If you need additional connection types, add them by adding additional inbound security group rules. Type - SSH, Protocol - TCP, Port - 22 Step 9: Configure “Key pair (login)“ section Go to the section. ”Key pair (Login)” Create a new key-pair if you haven't created it yet. If you haven't created yet: “key-pair” Click the button. “Create new key pair” Give your new key-pair a name in the section. “Key pair name” Select key-pair type or . I choose the type. RSA ED25519 RSA Select Private key file format. Choice of and . I choose the format. .pem .ppk .pem Click on the button. “Create key pair” You will get a pop-up window that will prompt you to download the Private key file. Agree and download the file to your computer. Step 10: Launch the EC2 Virtual Server Instance Launch the EC2 Virtual Server instance by clicking the button . “Launch instance” After the completion of the EC2 Virtual Server instance creation process, you will see the following. Then you should go to the section by clicking button. “Instances“ “View all instances” Now you can see that your AWS EC2 Virtual Server instance is running. [Module 2]: Jenkins Server Now, let’s configure JenkinsServer on the EC2 Virtual Server instance. Step 1: Create an AWS EC2 Virtual Server instance You need a virtual server to run Jenkins. Follow instructions from section of this tutorial to finish this step and create an EC2 virtual server instance with the name JenkinsServer. [Module 1]: AWS EC2 Virtual Server Do not forget to add a security group setup. It allows and to work on port and respectively. Jenkins SSH 8080 22 Use the name to distinguish your EC2 Virtual Server instance. “JenkinsServer” Create security group and for a new AWS EC2 instance. You can reuse them further in the article. “CI_CD_Pipeline” “CI_CD_Pipeline_Key_Pair“ “JenkinsServer” Step 2: Connect to an AWS EC2 Virtual Server instance Go to → → AWS Console home page EC2 Management Console Dashboard Instances. Then you should choose and then click the button. JenkinsServer “Connect” Then you will see this web page. You should again click the button. “Connect” Now you can see EC2 virtual server instance online terminal. Step 3: Download the Jenkins repository Now you need to download Jenkins on your EC2 virtual server instance. Follow these instructions: Go to Jenkins download . webpage You can see Stable (LTS) and Regular releases (Weekly) options. Choose LTS option. Red Hat/Fedora/Alma/Rocky/CentOS You will see this web page. Copy command and execute it to download Jenkins files from the Jenkins repository on the Internet and save them to the specified location on your EC2 virtual server instance. “sudo get..” sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo Now Jenkins is downloaded. Step 4: Import Jenkins key To finish the Jenkins installation, we need to import the Jenkins key. To import the Jenkins key we need to copy the command and execute it. “sudo rpm..” sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key This way package manager can verify that the Jenkins packages you install are exactly the ones published by the Jenkins project, and that they haven't been tampered with or corrupted. “rpm” Step 5: Install Java To run Jenkins, we need to install on our EC2 virtual server instance. Java To install , use this command. Java sudo amazon-linux-extras install java-openjdk11 -y Verify whether was installed correctly using this command: Java java -version You will see something like that. Step 6: Install fontconfig To run Jenkins, you need to install on our EC2 virtual server instance. fontconfig Use this command. sudo yum install fontconfig java-11-openjdk -y Fontconfig is a library designed to provide system-wide font configuration, customization and application access. It's required by Jenkins because Jenkins has features that render fonts. Step 7: Install Jenkins In earlier steps, you configured your EC2 virtual server instance to use a specific Jenkins repository and then you imported the GPG key associated with this repository. Now, you need to run the command that will search all the repositories it knows about, including the Jenkins one you added, to find the Jenkins package. Once it finds the Jenkins package in the Jenkins repository, it will download and install it. Let’s run this command. sudo yum install jenkins -y Step 8: Start Jenkins You can start Jenkins using this command. sudo systemctl start jenkins To check that Jenkins is running use this command. sudo systemctl status jenkins You will see the output as it is on the screenshot below: Jenkins should now be up and running. Step 9: Access Jenkins To access the Jenkins application, open any web browser and enter your EC2 instance’s public IP address or domain name followed by port 8080. http:// :8080 The first time you access Jenkins, it will be locked with an autogenerated password. You need to display this password using the following command. sudo cat /var/lib/jenkins/secrets/initialAdminPassword Copy this password, return to your browser, paste it into the Administrator password field, and click "Continue". Then you will be able to see this web page. Now, you can use your Jenkins Server. Step 10: Create new Jenkins pipeline Now, as Jenkins is working fine, you can start creating the Jenkins pipeline. To create Jenkins pipeline you need to create a new “Freestyle project”. To create a new “Freestyle project” you need to go to the Jenkins dashboard and click the button. “New Item” Enter the name of the Github “Freestyle project” (“pipeline” name is going to be used further) and then click the button . “OK” Then provide the of the pipeline. Description Then click the button “Apply” and “Save”. After that, it means you created the fundament of the pipeline which is going to be built in this tutorial. [Module 3]: Git and Github Now as Jenkins is running on AWS EC2 Virtual Server instance, you can configure Git with the pipeline. Git is a distributed version control system (VCS) designed to help software teams keep track of every modification to the code in a special kind of database. If a mistake is made, developers can turn back the clock and compare earlier versions of the code to help fix the mistake while minimizing disruption to all team members. VCS is especially useful for teams since they help them to reduce development time and increase successful deployments [1]. free and open source DevOps Git as the most popular version control system enables us to pull the latest code from your project Github repository to your EC2 virtual server instance where your Jenkins is installed. Step 1: Install Git Use this command to install Git. sudo yum install git -y Now verify Git is working, using this command. git --version Now Git is working fine on EC2 Virtual Server instance. Step 2: Open Jenkins dashboard As Git is working fine on EC2 Virtual Server instance, we can integrate Jenkins with Git now. To start this integration let’s install Jenkins Github plugin. Go to Jenkins dashboard section. Step 3: Open Jenkins Plugin Manager Click the button and then click the button . “Manage Jenkins” “Manage Plugins” Step 4: Find Github Jenkins plugin Click the button . “Available plugins” Find the plugin Search box. Github Select plugin. Github Step 5: Install Github Jenkins plugin Select plugin. And then click the button . Github “Install without restart” Wait for the end of the Github plugin downloading. Yes! The Jenkins Github plugin is installed. Step 6: Configure Github Jenkins Plugin Now as the GitHub Jenkins plugin is installed, you can configure this plugin to integrate Jenkins with Git finally. To do that you need to return to the main page by clicking the button “Go back to the top page”. Then on the main page, you need to click the button and then click the button . “Manage Jenkins” “Global tool configuration” Then on the Global Tool Configuration web page you should go to the Git section. In the Git section, you need to configure Git by providing the name and path to Git on the computer. Then click the and buttons**.** “Apply” “Save” Here, you have finished configuring the Jenkins Github plugin. Step 7: Integrate Git into the pipeline Now, as the Jenkins Github plugin is installed and configured, you're now able to utilize this plugin within your pipeline. This will allow your pipeline which you created in module 2 to pull your project code from the specified GitHub repository. Well, to integrate this plugin into your pipeline you need to go to the Source Code Management section and choose Git in your pipeline. Then you need to provide your project repository URL. If your project repository is public on Github, you do not need to provide credentials. If the project repository is private on Github, you need to provide credentials. You can use my project with the next Repositiry URL: . https://github.com/Sunagatov/Hello.git Just copy and paste it to the “ input. Then click the and buttons to finish the integration Git with the pipeline. Repository URL” “Apply” “Save” Step 8: Test Git integrated into the pipeline Now you can use your updated pipeline to pull a project from Github. To do that you need to click the **“Build Now”**button. As a result, you will see a successful build in the build history. Open the first build from the build history. Now you can see the successful job result of the first build. If you open your AWS EC2 terminal. You can check that the pipeline works well. Just use this command. cd /var/lib/jenkins/workspace/{your pipeline name} This way you can see that your project from Github was pulled to your AWS EC2 virtual server instance. [Module 4]: Apache Maven is a widely used build automation and project management tool in software development. It streamlines the process of compiling, testing, and packaging code by managing project dependencies and providing a consistent build lifecycle. Maven employs XML-based configuration files (POM files) to define project structure, dependencies, and tasks, enabling developers to efficiently manage and deploy complex software projects. Apache Maven Now that you have integrated Git into the pipeline, you can enhance the pipeline further by incorporating Apache Maven which enables you to build, test, and package your project. To do that you need to install Apache Maven on your AWS EC2 Virtual Server instance where Jenkins and Git were installed. Step 1: Download Apache Maven To download Apache Maven go to the directory. “/opt” cd /opt And then use this command. sudo wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz This command will download the latest official Apache Maven (check the latest version on the official Apache Maven website). To find the latest official Apache Maven release, use the link . https://maven.apache.org/download.cgi Step 2: Extract Apache Maven from the archive Use this command, to extract Apache Maven from the downloaded archive: sudo tar -xvzf apache-maven-*.tar.gz Step 3: Add JAVA_HOME and M2_HOME Move to the root folder using this command. cd ~ Edit file using this command. .bash_profile vi .bash_profile Add and M2_HOME variables. JAVA_HOME Assign the path to JDK11 for and path to the maven directory for variable. JAVA_HOME M2_HOME To find JDK path, use this command. sudo find / -name java How to use VIM To the file press the keyboard button to insert data. edit “ I “ To the file press the keyboard button “ esc “ and enter “:w“. save To from the file press the keyboard button and enter . exit “ esc “ “:q” Save the changes. Then, execute this command to refresh system variables. source .bash_profile To verify , use this command. $PATH echo $PATH To verify , use this command. Apache Maven mvn -v If you have done everything correctly, you will be able to view the version of Apache Maven. Step 4: Install Apache Maven Jenkins plugin Since Apache Maven can be used on an EC2 instance, you can install the Apache Maven plugin to integrate it with the pipeline. To achieve this, follow these steps: Navigate to “Dashboard“ → “Manage Jenkins“ → “Manage Plugins“ → “Available”. In the search box, enter . “Maven” Choose plugin. “Maven Integration” Wait for the end of the downloading process. And then click the button . “Go back to the top page” Step 5: Configure Apache Maven Jenkins plugin With the successful installation of the Apache Maven Jenkins plugin, you are now able to utilize this plugin within the pipeline which you created and updated in modules 2 and 3. To do so, follow these steps: Go to “Dashboard“ → “Manage Jenkins“ → “Global Tool Coonfiguration“ → “JDK” Click the button “Add JDK”. Uncheck “Install automatically”. Then go to section. Click the button . Uncheck “Maven” “Add Maven” “Install automatically”. Then add and path. name MAVEN_HOME Click the and buttons. “Apply” “Save” Here, you have finished configuring the Apache Maven Jenkins plugin. Step 6: Integrate Apache Maven into the pipeline Now as the Apache Maven GitHub plugin is installed and configured, you're now able to utilize Apache Maven within your pipeline. This will allow your pipeline which you created in the “[module 2]: Jenkins Server” to build your project code to create a jar artifact. To integrate Apache Maven into the pipeline you need to follow these steps: Navigate to “Dashboard“ → “CI_CD_Pipeline“ → “Configure“ → “Build Steps”. Click button. “Add build step” Choose option. “Invoke top-level Maven targets” Choose as “Apache-Maven” “Maven Version”. Add command to input. “clean package” “Goals” Click button. “Advanced“ Add “pom.xml” to input. “POM” Finally, you should click and buttons to finish the integration of Apache Maven with the pipeline. “Apply” “Save” Step 7: Test Apache Maven integrated into the pipeline Now you can use your updated pipeline to build your Github project. To do that you need to click the **“Build Now”**button. As a result, you will see a successful job result in the build history. If you open your AWS EC2 terminal. You can check that the pipeline works well. Just use this command. cd /var/lib/jenkins/workspace/{your pipeline name}/target This way you can see the JAR artifact, indicating the successful build of your project from GitHub. [Module 5]: Docker Now let’s create a new EC2 instance with the name "Ansible Server" where you are going to install Docker and Ansible. Step 1: Launch an AWS EC2 Virtual Server instance Use instructions from “ section of this tutorial to finish this step. Do not forget to add a security group setup. It allows Docker and SSH to work on ports 8080 and 22 respectively. Launch an AWS EC2 Virtual Server instance” Step 2: Connect to an AWS EC2 Virtual Server instance Click the “Connect” button. Now you can see the EC2 Virtual Server instance online terminal. sudo chown ansible-admin:ansible-admin /opt/docker Step 3: Install Docker on the Ansible EC2 virtual server. Now you need to install docker on your Ansible EC2 instance. To do that you need to create a new docker folder. sudo mkdir /opt/docker Then, install docker by executing the next command. sudo yum install docker -y Step 4: Add current user to Docker group on Ansible EC2 virtual server You need to add the current user to the Docker group on the EC2 virtual server to give Docker admin privileges. “ansible-admin” “AnsibleServer” sudo usermod -a -G docker ansible-admin You will need to logout and log back in for these changes to take effect. Then you can execute the next command id ansible-admin to see that the new docker user exists. Step 5: Launch Docker Now as Docker is installed on the Ansible EC2 instance you can launch it by executing the next command. sudo systemctl start docker As Docker is started you can execute the next command sudo systemctl status docker to see that docker is active and running now. Step 6: Create Dockerfile In the final version of the pipeline, the process will involve creating a new Docker image from your GitHub project and pushing it to Docker Hub. To accomplish this, your GitHub project should contain a Dockerfile. If you used the project which was offered in “ module, then you don’t need to create a new Dockerfile as this project repository has already contained Dockerfile. “Hello” [Module 3]: Git and Github” FROM eclipse-temurin:17-jre-jammy ENV HOME=/opt/app WORKDIR $HOME ADD hello-0.0.1-SNAPSHOT.jar $HOME ENTRYPOINT ["java", "-jar", "/opt/app/hello-0.0.1-SNAPSHOT.jar" ] If you used your own project repository and it didn't contain a Dockerfile, you will need to create a new Dockerfile. To create a new Dockerfile, execute the following command, which will create the new file. sudo touch Dockerfile Then you can fill this file with a sequence of commands that describe how to build a containerized application environment. These commands include actions like copying files into the image, installing software, setting environment variables, and configuring the container. To fill the Dockerfile with these commands, execute the following command. vim Dockerfile The Dockerfile is ready to use. Step 7: Login Docker Now that your Dockerfile is prepared for use, proceed by copying your project's JAR artifact from the **"JenkinsServer"**EC2 instance and pasting it onto the EC2 instance. It is important to note that this transfer will be automated through the pipeline further. "AnsibleServer" By completing this step, you'll be ready to test your Dockerfile along with the Docker environment you've set up. Before you begin testing, make sure to authenticate yourself on Dockerhub. Execute the following command. docker login This command will prompt you to provide your Dockerhub login credentials, including your username and password. With this, you have completed the process of logging into and are now ready to proceed with testing. Docker Testing the Docker environment and Dockerfile Step 8: With your successful login to Dockerhub completed, you are now ready to begin testing the Dockerfile you have prepared. Execute this command to create a docker image. docker build -t hello:latest . Next, execute the subsequent command to establish a tag that will facilitate the image's upload to Dockerhub: docker tag hello:latest zufarexplainedit/hello:latest Finally, proceed to push the Docker image to Dockerhub through the execution of this command. docker push zufarexplainedit/hello:latest Following these steps, navigate to your Dockerhub account to verify whether you can see a new image or not. You should now observe that the image has been effectively added. This outcome confirms the successful installation of the Docker environment and that your Dockerfile is correct. [Module 6]: Ansible Now let us configure the Ansible Server on the EC2 Virtual Server instance. Step 1: Create an AWS EC2 Virtual Server instance You need a virtual server to run Ansible. Follow instructions from section of this tutorial to finish this step and create an EC2 virtual server instance for Ansible. [Module 1]: AWS EC2 Virtual Server Do not forget to add a security group setup. It allows and to work on port and respectively. Ansible SSH 8080 22 Use the name to distinguish your EC2 Virtual Server instance. “AnsibleServer” You can reuse security group and for a new EC2 instance. “CI_CD_Pipeline” “CI_CD_Pipeline_Key_Pair“ “AnsibleServer” Step 2: Connect to an AWS EC2 Virtual Server instance Go to AWS Console home page → EC2 Management Console Dashboard → Instances → AnsibleServer. Then click the button. “Connect” Then you will see this web page. You should again click the button. “Connect” Now you can see the EC2 virtual server instance online terminal. Now let’s configure Ansible Server on EC2 virtual server instance. Step 3: Change the hostname of “AnsibleServer” AWS EC2 Virtual Server instance The first thing you need to do when you want to configure AnsibleServer on EC2 virtual server instance is changing its hostname. Let’s do it. Execute this command to open hostname file: sudo vi /etc/hostname You should see something like this: Replace this hostname with Then, reboot it. “ansible-server”. sudo init 6 Step 4: Add and configure a new user in “AnsibleServer” AWS EC2 Virtual Server instance Now let’s add a new user to the AWS EC2 Virtual Server instance. ansible-admin To do that use this command: sudo useradd ansible-admin Then, set the password for user. ansible-admin sudo passwd ansible-admin Also, you need to configure user privileges by editing the file. sudoers sudo visudo Add to this file. “ansible-admin ALL=(ALL) ALL” sudoers Also, you need to edit file to enable PasswordAuthentication. /etc/ssh/sshd_config sudo vi /etc/ssh/sshd_config Then you need to reload the service to confirm these changes. sudo service sshd reload As a result of executing this command, you will see: Now you can use this command to avoid adding sudo for all further commands. sudo su - ansible-admin Step 5: Generate key in an AWS EC2 Virtual Server instance You are planning to manage remote servers such as K8s EC2 virtual server instances further in this article. That is why you need to set up SSH keys. ssh-keygen As a result of executing this command, you will see: Now SSH keys are generated and ready to use. Step 6: Install Ansible Now you can install Ansible on your EC2 virtual server instance. “AnsibleServer” Let’s do it. Execute this command to install Ansible. sudo amazon-linux-extras install ansible2 To verify Ansible, use this command: ansible --version As a result of executing this command, you will see: Step 7: Install Jenkins “Publish over SSH“ plugin on JenkinsServer EC2 instance As Ansible is installed on your EC2 virtual server instance, you can configure Jenkins to integrate it with Ansible. You need to install the plugin to integrate Jenkins with the EC2 Virtual Server instance where Ansible is installed and with other EC2 Virtual Server instances where is installed. “AnsibleServer” “Publish over SSH” Kubernetes Go to → “ → → . “Dashboard” Manage Jenkins” “Configure System” “Available plugins” Then enter in the search box. “Publish over SSH“ Click the button . Wait for the end of the downloading process. “Install without restart” Now the “Publish over SSH” plugin is installed on the Jenkins EC2 Virtual Server instance. Step 8: Configure Jenkins “Publish over SSH“ plugin With the successful installation of the Apache Maven Jenkins plugin, you are now able to utilize this plugin within the pipeline which you created and updated in modules 2 and 3. To do so, follow these steps: Go to → → → . “Dashboard“ “Manage Jenkins” “Configure System” “Publish over SSH” Enter all required data as it is on the screenshot, including hostname, username, and private key (or password if applicable). Then click the and buttons. “Apply” “Save” Here you have finished configuring the Jenkins plugin. “Publish over SSH“ Next, click “Test Configuration” to validate that the plugin is working correctly. On the left side you can see that the test plugin configuration status is “Success”. It means the plugin configuration is correct. Step 9: Create new Docker directory in AnsibleServer You need to make a new folder on the AnsibleServer EC2 instance where the project JAR will be stored. This jar will be used for creating a Docker image later. Let’s start. Go to folder in AnsibleServer EC2 instance. “/opt” cd /opt Create a new folder there. “docker” sudo mkdir docker Give privileges to this folder. “docker” sudo chown ansible-admin:ansible-admin docker Now, check the folder privileges by executing this command. “docker” ll You can see that the folder is accessible with the user. “docker” “ansible-admin” Github plugin into Step 10: Integrate “Publish over SSH“ the pipeline Now as Github plugin is installed and configured, you're now able to integrate it into the pipeline which you created in the “[module 2]: Jenkins Server” to transfer a project jar artifact from “ to . “Publish over SSH“ JenkinsServer” “AnsibleServer” Well, to integrate Github plugin into the pipeline you need to follow these steps: “Publish over SSH“ Navigate to “Dashboard“ → “CI_CD_Pipeline“ → “Configure“ → “Post-build Actions”. Click button. “Add post-build action” Choose option. “Send build artifacts over SSH” Add into input in section**.** “AnsibleServer” “Name” “SSH Server” Add into input in section**.** “target/*.jar” “Source files” “Transfer Set” Add into input in section**.** “target/” “Remove prefix” “Transfer Set” Add into input in section**.** “//opt/docker/” “Remote directory” “Transfer Set” For now, simply put an empty string in the 'Exec command' input under the 'Transfer Set' section. Finally, you should click and buttons to finish the integration plugin with the pipeline. “Apply” “Save” “Publish over SSH“ Github plugin Step 11: Test “Publish over SSH“ integrated into the pipeline Now you can use your updated pipeline to transfer a project jar artifact from “ to . To do that you need to click the button. As a result you will see a successful job result in the build history. JenkinsServer” “AnsibleServer” “Build Now” If you open your AWS EC2 terminal. You can check that the pipeline works well. “AnsibleServer” Just use this command. cd /opt/docker This way you can see the JAR artifact, indicating the successful build of your project from GitHub. Step 12: Configure Ansible hosts When you run an Ansible playbook, you specify the hosts it should run on. You can do this in two ways: In your playbook, you can set the parameter to a list of IP addresses or hostnames. Specify hosts directly in the playbook: hosts Ansible allows you to define a list of hosts in an inventory file and reference this file when running your playbook. The default inventory file is . Use an inventory file: /etc/ansible/hosts By editing , you can easily manage groups of hosts without having to write out their IP addresses each time you run a playbook. /etc/ansible/hosts Let’s find out the AnsibleServer EC2 instance host by executing the following command. sudo ifconfig After you find out AnsibleServer EC2 instance host, you can add it to the Ansible hosts file by executing the following command. sudo vi /etc/ansible/hosts You can also add “[ansible]” as a reference If you're managing a cluster of servers and you want to apply a playbook to all of them, instead of specifying each server's IP address in the playbook, you can simply add all the servers to a group in the inventory file and then specify the group in the playbook. Passwordless SSH authentication to localhost for Ansible Step 13: Configure Ansible is designed for automating tasks on remote servers. Passwordless SSH authentication empowers Ansible to connect to those servers without the need for manual password entry. Execute this command to create a secure connection from your computer to another computer (like the one at IP address 172.31.34.41), using the ansible-admin user's SSH key. sudo ssh-copy-id -i /home/{your user name}/.ssh/id_rsa.pub {your user name}@{your host address} In my case, it looks like this. sudo ssh-copy-id -i /home/ansible-admin/.ssh/id_rsa.pub ansible-admin@172.31.34.41 Now you can see . It means that the passwordless SSH authentication installation was successfully completed. “Number of key(s) added: 1” Step 14: Create a new Ansible playbook for Docker tasks Now that Ansible is all set up and good to go, you can make a new Ansible playbook for your pipeline. This playbook will let Ansible create and send a new Docker image to Dockerhub. Here's how you can do it: Start by creating a new Ansible playbook file. Just use this command. touch hello-app.yml Then, edit the newly made file. Open it up for editing with this command. hello-app.yml vi hello-app.yml Inside the file, type in the following stuff. --- - hosts: ansible user: root tasks: - name: create docker image command: docker build -t hello:latest . args: chdir: /opt/docker - name: create tag to push image onto dockerhub command: docker tag hello:latest zufarexplainedit/hello:latest - name: push docker image onto dockerhub command: docker push zufarexplainedit/hello:latest The Ansible playbook for Docker tasks is ready to use. Step 13: Test the Ansible playbook with Docker tasks With Ansible, Ansible playbook, AnsibleServer, and JenkinsServer all properly configured, it is time to test the Ansible playbook. Navigate to the location of your Ansible playbook. cd /opt/docker Then, execute the following command. sudo -u ansible-admin ansible-playbook /opt/docker/hello-app.yml Upon completion, you will see the successful execution outcome of your Ansible playbook. Furthermore, take a moment to visit your Dockerhub account and verify whether a new image is now visible. You should see the newly added image. This outcome confirms that your Ansible playbook is correct. into Step 14: Integrate Ansible Docker tasks the pipeline Now as Github plugin, Ansible and Docker are installed and configured, you're now able to integrate them all into the pipeline which you created in the “[module 2]: Jenkins Server” to transfer a project jar artifact from “ to and then build a new Docker image from your project and then push this Docker image onto Dockerhub. “Publish over SSH“ JenkinsServer” “AnsibleServer” To achieve it you need to follow these steps: Navigate to “Dashboard“ → “CI_CD_Pipeline“ → “Configure“ → “Post-build Actions” → “Send build artifacts over SSH” . Add command into input. “sudo -u ansible-admin ansible-playbook /opt/docker/hello-app.yml” “Exec command” Finally, click and buttons to finish the integration Ansible Docker tasks with the pipeline. “Apply” “Save” “ Step 15: Test “Ansible playbook with Docker tasks integrated into the pipeline Now you can test your upgraded pipeline to seamlessly transfer a project jar artifact from “ to then build a new Docker image from your project and then push this Docker image onto Dockerhub. To do that you need to click the button. As a result you will see a successful job result in the build history. JenkinsServer” “AnsibleServer” “Build Now” Furthermore, take a moment to visit your Dockerhub account and verify whether a new image is now visible. You should see the newly added image. This outcome confirms that your Ansible playbook with Docker tasks was successfully integrated into the pipeline. [Module 7]: Kubernetes Now let’s configure K8s on the EC2 instance. You are going to create a new EC2 instance and install their kubectl command-line tool for interacting with a cluster further. Kubernetes Step 1: Launch an AWS EC2 Virtual Server instance Use instructions from “ section of this tutorial to finish this step. Launch an AWS EC2 Virtual Server instance” Do not forget to add a security group setup. It allows and to work on port and respectively. all tools SSH 8080 22 Use the name to distinguish your EC2 Virtual Server instance. “K8sServer” You can reuse security group and for a new EC2 instance. “CI_CD_Pipeline” “CI_CD_Pipeline_Key_Pair“ “K8sServer” Step 2: Connect to an AWS EC2 Virtual Server instance Click the “Connect” button. Now you can see the EC2 Virtual Server instance online terminal. Step 3: Change the hostname of “KubernetesServer” AWS EC2 Virtual Server instance The first thing you need to do when you want to configure KuberenetesServer on EC2 virtual server instance is to change its hostname. Let’s do it. Execute this command to open hostname file: sudo vi /etc/hostname You should see something like this. Replace this hostname with then reboot it. “kubernetes-server” and sudo init 6 You will hostname was changed. Step 4: Check AWS CLI version Use this command to check the version. AWS aws --version This way you can see your current aws-cli version. If you can see version , you should download the latest version. aws-cli/1.18 Step 5: Update AWS CLI Now as you found out that you have an old aws-cli version on your EC2 instance you need to update it. To that go to AWS → Documentation → AWS Command Line Interface → User Guide for Version 2. Copy-paste . the curl command First, execute this command to download awscli version 2. curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" Wait for the downloading process to commence. You should see something like this. Second, you need to execute this command to unzip awscli version 2 archive. unzip awscliv2.zip Third, you should execute this command to install awscli version 2. sudo ./aws/install Then, reload the Kubernetes EC2 Virtual Server instance online terminal. Next, use this command to check the AWS version aws --version You can see that aws cli has aws-cli/2. Step 6: Install kubectl is a fundamental command-line tool for interacting with any Kubernetes cluster, regardless of the underlying infrastructure. It allows you to manage resources, deploy applications, configure networking, access logs, and perform various other tasks within a Kubernetes cluster. Kubectl Now you need to install kubectl command-line tool for interacting with a Kubernetes cluster further. To that you need to go to → → → → or → . AWS Documentation Amazon EKS User Guide Installing updating kubectl Linux Or just click the link . https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html First, execute this command to download kubectl. curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.27.1/2023-04-19/bin/linux/amd64/kubectl Wait for the downloading process. You will see something like this. Give permission to kubectl. chmod +x kubectl Move kubectl to /usr/local/bin folder. sudo mv kubectl /usr/local/bin Check the version of kubectl. kubectl version --output=yaml Step 7: Install eksctl is an another command-line tool which is tailored specifically to the Amazon EKS service. Eksctl can be used to create AWS EKS clusters, manage node groups, and perform tasks specific to EKS, such as integrating with IAM roles and other AWS services by abstracting away much of the AWS infrastructure setup and management. Eksctl Execute the command to download kubectl. curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp Move eksctl to /usr/local/bin folder. sudo mv /tmp/eksctl /usr/local/bin Check the version of eksctl. eksctl version You should see the version Step 8: Create IAM Role for eksctl You need to create an IAM role and attach it to your EC2 instance. “KubernetesServer” To do that you need to find EC2 in the search box. Choose EC2 Virtual Server by clicking the link /. https://us-east-1.console.aws.amazon.com/ec2 Go to → . IAM Dashboard Roles Click the button on the roles web page. “Create role” IAM Then choose And the click button. “AWS service” , “EC2”. “Next” Then, find , , , in the search box and then click the button. “AmazonEC2FullAccess” “AmazonEC2FullAccess“ “IAMFullAccess“ “AWSCloudFormationFullAccess“ “Add permissions” And then click the button. “Next” Then type into input. “Eksctl_Role” “Role name” And the click button. “Create role” The role is created finally. Step 9: Connect IAM Role with eksctl Go to the AWS EC2 instance web page. Choose Then click → → “KuberbetesServer”. “Actions” “Security” “Modify IAM Role”. Choose and then click the button. “Eksctl_Role” “Update IAM role” Now your IAM Role is connected with your and eksctl tool. “EKS_Server” Step 10: Create eksctl cluster cluster is a managed Kubernetes environment on AWS, automating intricate infrastructure tasks like setup, scaling, and maintenance. It's essential as it provides an efficient, secure, and AWS-optimized platform for deploying, managing, and scaling containerized applications, streamlining operations and freeing developers to focus on coding rather than managing underlying infrastructure. An Amazon EKS (Elastic Kubernetes Service) Now, it's time to set up your EKS cluster. To achieve this, follow these steps: Modify the following command with your specific information. eksctl create cluster --name cluster-name \ --region region-name \ --node-type instance-type \ --nodes-min 2 \ --nodes-max 2 \ --zones , For example, in my case, it looks like this. eksctl create cluster --name zufarexplainedit \ --region eu-north-1 \ --node-type t3.micro Execute the modified command and patiently await the completion of the cluster creation process. You will notice that the EKS cluster status is indicated as on the AWS CloudFormation web page. "creating" The cluster creation procedure typically takes around 20 minutes. Once completed, the terminal will display the outcome of the process. Furthermore, you can verify the successful creation status on the AWS CloudFormation web page. EKS cluster Step 11: Create Kubernetes Deployment yaml file is a configuration script written in YAML format that defines how to manage and maintain a specific application or service within a Kubernetes cluster. It encapsulates instructions for orchestrating the deployment, scaling, updating, and monitoring of containers running the application. This file includes details such as the container image, the desired number of replicas, resource limits, environment variables, networking settings, and more. When applied to a Kubernetes cluster, the Deployment YAML file ensures the desired state of the application, automatically managing the creation, scaling, and recovery of containers to maintain the desired level of availability and reliability. A Kubernetes Deployment YAML file Now, as Kubernetes cluster, eksctl, kubectl are installed and configured, you can create a Kubernetes Deployment yaml file. You can do that by executing the following command. touch hello-app-deployment.yaml Then, edit this file by executing the following command. vi hello-app-deployment.yaml Add this content to the hello-app-deployment.yaml. apiVersion: apps/v1 kind: Deployment metadata: name: zufarexplainedit-hello-app labels: app: hello-app spec: replicas: 2 selector: matchLabels: app: hello-app template: metadata: labels: app: hello-app spec: containers: - name: hello-app image: zufarexplainedit/hello imagePullPolicy: Always ports: - containerPort: 8080 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 1 Now hello-app-deployment.yaml is created and ready to use. Step 12: Create Kubernetes Service yaml file is a configuration script written in YAML format that defines a network abstraction for a set of pods, allowing them to be accessed consistently within a Kubernetes cluster. This file outlines how the service should be discovered, accessed, and load-balanced by other services or external clients. It includes specifications like the service type (ClusterIP, NodePort, LoadBalancer), port numbers, selectors to identify pods, and more. When applied to a Kubernetes cluster, the Service YAML file creates a virtual IP and port that routes traffic to the appropriate pods, abstracting the underlying pod changes and providing a stable endpoint for communication, enabling seamless connectivity and dynamic scaling. A Kubernetes Service YAML file As Kubernetes cluster, eksctl, kubectl are installed and configured, you can create Kubernetes Service yaml file. To do that you need to create a Kubernetes Service yaml file by executing the following command. touch hello-app-service.yaml Then, edit this file by executing the following command. vi hello-app-service.yaml Add this content to the hello-app-deployment.yaml. apiVersion: v1 kind: Service metadata: name: zufarexplainedit-hello-app-service labels: app: hello-app spec: selector: app: hello-app ports: - port: 8080 targetPort: 8080 type: LoadBalancer Now hello-app-service.yaml is created and ready to use. Step 13: Test Kubernetes cluster using kubectl With your Kubernetes EKS cluster successfully installed, and configured, and your Kubernetes service and deployment files ready, it's time to put things to the test using kubectl commands. Apply Deployment. Use the following command to apply the deployment configuration. kubectl apply -f hello-app-deployment.yaml This will create a deployment with the specified number of replicas and a rolling update strategy, ensuring your application's availability and manageability. 2. Apply Service. Next, apply the service configuration. kubectl apply -f hello-app-service.yaml This will set up a LoadBalancer type service, exposing your application to the internet. Note that it might take a short while for the LoadBalancer to be provisioned and acquire an external IP address. Check LoadBalancer Status. Monitor the status of your service using. kubectl get service zufarexplainedit-hello-app-service When an external IP is assigned, you're almost ready to access your application. Access Your Application. Using a web browser, enter the assigned external IP address followed by :8080. After a brief moment, the page will load, displaying the "HelloWorld" message. Keep in mind that the initial loading might take a few seconds. Cleaning Up with kubectl Commands When you need to tidy up resources within your Kubernetes environment, you can use the following kubectl commands to effectively remove deployments, pods, and services. . 1. Delete All Deployments To delete all deployments, you can use the following command. kubectl delete deployments --all This action ensures that no active deployment instances are left in your cluster. . 2. Delete All Pods If you need to delete all pods, whether they are managed by a deployment or not, you can use the following command. kubectl delete pods --all Clearing pods can help reset your cluster state or prepare for new deployments. . 3. Delete All Services To clean up services that expose your applications to the network, you can use the following command. kubectl delete services --all Removing services may involve downtime, so consider the implications before proceeding. Deleting an Amazon EKS Cluster To remove all the resources associated with the specified Amazon EKS cluster created with , including worker nodes, networking components, and other resources, you can use the following command. eksctl eksctl delete cluster --name {your cluster name} --region {your region name} For me it is. eksctl delete cluster --name zufarexplainedit --region eu-north-1 Make sure you are certain about stopping the cluster, as this action is irreversible and will result in data loss. Step 14: Add and configure a new user in “KubernetesServer” AWS EC2 Virtual Server instance Now let’s add a new user to AWS EC2 Virtual Server instance. ansible-admin “KubernetesServer” To do that use this command. sudo useradd ansible-admin Then, set the password for user. ansible-admin sudo passwd ansible-admin Also, you need to configure user privileges by editing the file. sudoers sudo visudo Add to this file. “ansible-admin ALL=(ALL) ALL” sudoers Also, you need to edit file to enable PasswordAuthentication. /etc/ssh/sshd_config sudo vi /etc/ssh/sshd_config Then you need to reload the service to make these changes. sudo service sshd reload As a result of executing this command, you will see: Now you can use this command to avoid adding sudo for all further commands. sudo su - ansible-admin Step 15: Generate key in an AWS EC2 Virtual Server instance You are planning to manage remote servers such as EC2 virtual server instance further in this article. That is why you need to set up SSH keys. K8s ssh-keygen As a result of executing this command, you will see: Now SSH keys are generated and ready to use. Passwordless SSH authentication to localhost for Ansible Step 16: Configure Ansible is designed for automating tasks on remote servers. Passwordless SSH authentication empowers Ansible to connect to those servers without the need for manual password entry. Execute this command to create a secure connection from your computer to another computer (like the one at IP address 172.31.34.41), using the ansible-admin user's SSH key. sudo ssh-copy-id -i /home/{your user name}/.ssh/id_rsa.pub {your user name}@{your host address} In my case, it looks like this. sudo ssh-copy-id -i /home/ansible-admin/.ssh/id_rsa.pub ansible-admin@172.31.34.41 Now you can see . It means that the passwordless SSH authentication installation was successfully completed. “Number of key(s) added: 1” Step 17: Configure Ansible hosts in AnsibleServer EC2 instance When you run an Ansible playbook, you specify the hosts it should run on. In this step you need to specify KubernetesServer EC2 instance host. To do that you need to repeat the same steps which you passed in “ [Module 6]: Ansible”. Let’s find out the KubernetesServer EC2 instance host by executing the following command. sudo ifconfig After you find the KubernetesServer EC2 instance host, you can add it to the Ansible hosts file by executing the following command. sudo vi /etc/ansible/hosts You can also add “[kubernetes]” as a reference Step 18: Create a new Ansible playbook for Kubernetes tasks Now that Kubernetes is all setup and good to go, you can make a new Ansible playbook with Kubernetes tasks for your pipeline. This playbook will let Ansible run your app on the Kubernetes cluster with kubectl commands. Here's how you can do it: Start by creating a new Ansible playbook file. Just use this command. touch kubernetes-hello-app.yml Then, edit the newly made file. Open it up for editing with this command. hello-app.yml vi kubernetes-hello-app.yml Inside the file, type in the following: --- - hosts: kubernetes tasks: - name: deploy regapp on kubernetes command: kubectl apply -f hello-app-deployment.yaml - name: create service for regapp command: kubectl apply -f hello-app-service.yaml - name: update deployment with new pods if image updated in docker hub command: kubectl rollout restart deployment.apps/zufarexplainedit-hello-app The Ansible playbook for Kubernetes tasks is ready to use. Step 19: Integrate a new Ansible playbook for Kubernetes tasks with Jenkins Now that Kubernetes, Ansible, and Ansible playbook for Kubernetes tasks are all set up and good to go, you can integrate it with Jenkins. Go to JenkinsServer → Jenkins dashboard → New Item. Create new Jenkins Freestyle project with the name “CD-Job”. Click “OK” button. Go to “Post-build Actions” section. Click the “Add post-build action” button. Choose “Send build artifacts over SSH” option. Choose “AnsibleServer” as SSH Server. Add this command to “exec command” input. sudo -u ansible-admin ansible-playbook /opt/docker/kubernetes-hello-app.yml Click “Apply” and “Save” buttons. Go to Jenkins dashboard → “CI_CD_Pipeline” → Configure → “Post-build Actions” section. Click the “Add post-build action” button. Choose “Build other projects” option. Go to “Build other projects” section. Choose “Trigger only if build is stable” option. Add “CD-Job” to “Project to build” input. Click “Apply” and “Save” buttons. Now you can consider that the pipeline is completely finished and ready to use. Step 20: Testing the Final Pipeline Version Now is the time to put the final pipeline version to the test. Navigate to Jenkins → Jenkins dashboard → "CI_CD_Pipeline". Click the "Build Now" button. To see the results, review the build console logs. You will notice that the "CD-Job" was triggered in the build console logs, and the final status is marked as SUCCESS. 4 Moreover, you can head to Dockerhub to confirm the addition of the new Docker image. And finally, you can access your application using a web browser. Enter the assigned external IP address followed by ":8080". After a brief moment, the page will load, showing the "HelloWorld" message. Conclusion Well done on setting up this modern CI/CD pipeline! You've done an awesome job, and you're a real hero! Thanks for all your effort! About the author is an experienced senior software engineer who is passionate about designing modern software systems. Zufar Sunagatov Developed high-performance distributed systems using Java, Kotlin, AWS, Docker, and Kubernetes for companies like Citi (London), MTS (Moscow) and others ( ). you can follow Zufar's LinkedIn profile for more information Founded Ice Latte, an open-source online marketplace from scratch (you can join Zufar’s team and contribute to his project on GitHub: ). https://github.com/Sunagatov/Online-Store Established the ZufarExplainedIT community to share his knowledge and experience with others (you can follow Zufar's IT Telegram community: ). https://t.me/zufarexplained Mentored for over 3000 minutes and conducted 60 sessions on ADPlist, nurturing aspiring talents ( ). you can get mentored by Zufar on ADPlist Engaged audiences as a speaker at conferences and IT YouTube channels, sharing invaluable insights (examples of his YouTube videos include: , ). “Тестовое собеседование Java разработчика #27 - Зуфар Сунагатов” “Aspiring Speakers Lunchtime Lightning Talks May 28th” Authored highly popular IT articles with 7000+ views each (examples of his articles are: , ). “Microservice Architecture Patterns Part 1: Decomposition Patterns” “Что значит «вариативный»? Объясняем простыми словами”

Building a CI/CD Pipeline with AWS, K8S, Docker, Ansible, Git, Github, Apache Maven, and Jenkins

Too Long; Didn't Read

@zufarexplained

Credibility

RELATED STORIES