How to Turn A DevOps Pipeline Into a DevSecOps Pipeline: A Shift Left Concept Overview

Hi, HackerNoon! My name is Sofia, and I am a DevOps/Cloud engineer. I decided to participate in the contest by HackerNoon and Aptible.

In this article, I will talk about the features of the DevSecOps pipeline and the concept of Shift Left. You will learn about the main stages of the DevSecOps pipeline, automated security checks in software development, and free and open-source tools. You'll also find tips to help you detect vulnerabilities earlier and improve application security.

This article will help you assess the maturity of your DevSecOps pipeline, develop a roadmap for its development, choose the right tools for each task, and better understand how to manage projects following the DevSecOps philosophy.

Shift Left Concept

The main goal of the DevSecOps methodology is to introduce automated security checks into the DevOps pipelines, i.e., into the software development process itself.

Not so long ago, information security specialists conducted tests after completing the development process. This approach is inefficient - if errors are discovered during security testing, the entire development cycle has to be restarted. This is time-consuming and expensive.

Take a look at the image below. You can see that the cost of error correction increases with each step.

It costs almost nothing to fix security issues discovered during development and functional testing. All necessary changes can be made to the source code immediately and sent for rechecking.

Fixing problems at the artifact build stage is at least twice as expensive. You will have to make changes to the build process, for example, edit the Dockerfile, which means you will need the help of DevOps engineers.

If an error is detected during integration testing, it will be 10 times more expensive to fix it. In this case, you must restart the development cycle and involve developers, DevOps engineers, and testers.

Security problems detected at the deployment stage can lead to user data leaks. The company can receive millions of dollars in fines and damage to its reputation, which means the cost of a mistake increases hundreds of times.

Hence, the main conclusion is that safety checks should be performed as early as possible. If you visualize it, move them to the left side of the Pipeline. This is the concept of Shift Left, which security experts love to talk about so much.

DevSecOps Pipeline Structure

DevSecOps pipelines are an automated chain of processes and tools that build, test, and deliver applications in a production environment and secure them at every stage.

The image above shows the DevSecOps Pipeline structure with all the main phases of security checks. Here are a few words about each of them:

• Pre-commit. It implies checking the security of source code before the developer commits the code to the version control system. This check allows you to detect unencrypted confidential information such as passwords or API keys.

• Pre-build. It implies checking the security of source code after it has entered the version control system. The tools perform static analysis of source code and its dependencies to detect common vulnerabilities, such as many of the OWASP Top Ten list.

• Post-build. It is the security check of an artifact built from source code, such as a Docker file, RPM package, or JAR archive. The tools analyze the application environment and dependencies, finding outdated versions of packages and libraries that already have patches in newer versions.

• Test-time. It implies testing the security of an application running from a collected artifact. Tools in this phase try to "break" the application by simulating the attackers' actions.

• Post-deployment. It implies checking the security of an application after it has been deployed in a production environment. The tools monitor open registries of known vulnerabilities in real time and notify when a threat to the application is detected.

Next, let's take a closer look at what these checks are and what tools are used to perform them.

Pre-commit Checks

Pre-commit checks allow you to analyze the source code on the developer's machine before committing changes to the local copy of the repository. If any of the statements returns an error, the commit is not performed. In this case, the developer must fix the mistake and commit again.

This check avoids the situation when unchecked code, for example, with unencrypted secrets, gets into the repository.

For performing pre-commit source code checks, it is necessary to install tools on developers' local machines. This approach has not only advantages but also disadvantages. For example, environmental differences: different versions of the instruments, various operating systems, and even processor architectures.

If developers use different versions of pre-commit tools, the results of checking will differ, and this may make it challenging to work together. Consider this when implementing pre-commit checks within a team or across the company.

Tools for Pre-Commit Checks

Many open-source tools can be used to set up pre-commit checks:

1. Gitleaks
2. git-secrets
3. Trivy Secret Scanning
4. Whispers
5. git-all-secrets
6. detect-secrets
7. gitty leaks

These are great tools for pre-commit checks.

Pre-Build Checks

In the pre-build phase, White Box Testing is performed. These checks are used to analyze the source code, like in the previous step. In this case, the application is a "white box" because we know and understand its architecture and internal structure.

There are several types of pre-build checks:

• Secret Detection
• Static Application Security Testing (SAST)
• Software Composition Analysis (SCA)

Now, let’s discuss them in detail.

Pre-build Secret Detection Check

Secret Detection is an automated check that detects unencrypted sensitive information: unencrypted secrets in source code that have entered a version control system.

Pre-build checks are performed after the source code has entered the repository of the version control system. Therefore, all unencrypted sensitive data in the storage can potentially be accessed by attackers, for example, if the repository's contents are leaked.

In addition, the process of implementing secret-detection checks may occur not from scratch but in an evolving project. In this case, unencrypted secrets may be found in old commits and different repository branches.

To solve these problems, we need to do many different things. For example, we must clean repositories of unencrypted secrets or implement various secret management tools like Hashicorp Vault.

Tools for Secret Detection

Secret Detection can be configured using Gitleaks, git-secrets, or detect-secrets. However, it is best to use services provided by well-known CI/CD platforms, for example, GitLab Secret Detection.

Pre-Build SAST Check

Static Application Security Testing (SAST) is a test when analyzers do not just check syntactic correctness but also measure code quality with the help of unique metrics. The main task of SAST scanners is security testing.

In particular, SAST-analysers contain source code for common vulnerabilities, for example, some of the OWASP Top Ten lists. It is essential to say that SAST scanners find not only the vulnerability itself but also the code fragment that caused it.

SAST analysis is called White Box Testing because the analyzer can access the application's internal structure. It is important to note that analyzers check the source code without running it, so they may generate false positives and fail to detect some vulnerabilities.

For this reason, you should not limit yourself to SAST analysis only. It is better to approach the issue comprehensively and use different types of analysis: SCA, DAST, IAST, and OAST.

SAST Tools

Proprietary solutions:

• SonarQube
• Checkmarx SAST
• Solar appScreener

The free solution is GitLab SAST.

Pre-Build Source SCA Check

Software Composition Analysis (SCA) is a methodology that secures applications by analyzing their open-source components.

SCA scanners look for outdated dependencies that contain known vulnerabilities and exploits. In addition, some SCA tools can determine the license compatibility of components and the risks of license infringement.

Source SCA analyses the source code and, more specifically, the application dependencies defined in the source code. This analysis is often referred to as Dependency Scanning.

SCA allows you to detect an issue where a vulnerability in one component can lead to security problems in all applications.

A good example is Log4Shell. This vulnerability was discovered in late 2021 in Log4j, a popular Java logging framework. It became one of the most feared vulnerabilities because it allowed attackers to execute arbitrary Java code on hundreds of millions of devices.

SCA tools

Trivy is an open-source solution.

Commercial solution with free pricing plans:

• FOSSA
• Mend SCA

As part of GitLab (available in Ultimate-version only), you can use GitLab Dependency Scanning.

Post-Build Checks: Binary SCA

The post-build phase comes after artifacts have been built from source code in the CI Pipeline: a Docker image, an RPM package, or a JAR archive. With post-build checks, you can analyze the dependencies in these binary artifacts.

Binary SCA analysis involves inspecting binary artifacts such as Docker images, RPM packages, and JAR/WAR/EAR archives. It is also often referred to as Container Scanning. It makes sense to run it when the binary artifacts are built, i.e., not before the post-build stage.

There are some exciting features when working with Docker images. Binary SCA analyzers check layers of Docker images and look for them as hash sums in public vulnerability databases such as National Vulnerability Database (NVD) or Snyk Vulnerability DB.

Some of the analyzers can not only find vulnerable components but also suggest a fix, for example, by specifying the minimum required version of a component with an already fixed vulnerability.

Examples of Popular Binary SCA Analyzers

The free solution is GitLab Container Scanning.

Open Source solutions:

• Trivia
• Clair
• Grape
• Quay
• JFrog Xray
• Docker Hub

Docker Registry with built-in analyzers - Harbour.

Test-Time Checks: DAST, IAST, and OAST

At this stage, dynamic Gray Box Testing and Black Box Testing are performed. When the application's internal structure is partially or wholly unknown, these security checks are performed by emulating the actions of an attacker who finds vulnerabilities and tries to "break" the application by exploiting them. Let`s briefly discuss each of them: DAST, IAST, and OAST.

Test-Time DAST Check

DAST scanners automatically test applications by simulating external attacks through various vulnerabilities. The application is a black box for the analyzer; nothing is known about it.

For DAST checks, it is necessary to have a running instance of the application available for the scanner. Moreover, the closer the parameters of the test instance of the application are to the production installation, the fewer false positives there will be.

For example, suppose you have deployed a test application instance accessible only via HTTP protocol, but in production, the application is accessible only via HTTP protocol.

In that case, the DAST scanner will generate some errors related to the lack of traffic encryption between the client (analyzer) and the server (application instance).

Examples of DAST Tools

Tools that are worth your attention:

• GitLab DAST — only available in the Ultimate version
• OWASP Zed Attack Proxy (ZAP) — an open-source solution, which is also used in GitLab DAST
• Acunetix
• Fortify WebInspect
• HCL Security AppScan
• Synopsys Managed DAST
• Tenable.io (Web App Scanning)
• Veracode Dynamic Analysis

Great, move on.

Test-Time IAST Check

IAST (Interactive Application Security Testing) is a Gray Box Testing that combines the advantages and strengths of White Box Testing SAST and Black Box Testing DAST.

To collect all the advantages and eliminate the disadvantages of SAST and DAST methods, developers invented IAST - a hybrid mechanism combining these methods. It increases the accuracy of dynamic testing because it gives more information about the application operation through code analysis.

It is worth remembering that besides its advantages, IAST has limitations. The most basic one is the dependence on the language in which the application under test is written. IAST solutions work at the application level and require access to the executable code to analyze its behavior.

This means that IAST solutions must be able to understand the programming language in which the application is written. It should be noted that support for a particular IAST solution may be better implemented for some languages than others.

IAST analysis also takes a long time. It depends on many factors, such as the size and complexity of the application, the number of external dependencies, and the performance of the IAST tool used.

Besides, the analysis process does not scan the code itself but checks only those fragments that are directly executed. Therefore, IAST analysis is best combined with functional testing when you can test as many application scenarios as possible.

Examples of IAST Tools

Here are great tools for you:

• Synopsys Seeker
• Veracode Interactive Analysis
• Microfocus Fortify WebInspect
• Checkmarx CxIAST
• Contrast Assess
• Hdiv Detection (IAST)

Great, move on.

Test-Time OAST Check

OAST (Out-of-band Application Security Testing) is a technique developed by PortSwigger and is an extension of DAST that finds vulnerabilities that DAST does not see, such as log4shell.

Examples of OAST tools

I recommend you:

• Burp Suite
• OWASP ZAP with OAST plugin
• OAST services BOAST, TukTuk, and interact

Move on.

Post-Deploy Checks

This is an essential stage in ensuring application security and operability. Unlike pre-commit checks, which are performed at the development stage, post-deploy checks allow you to identify possible problems already during application operation in the natural environment.

Monitoring the Safety of Artifacts

For detecting new vulnerabilities in time, it is necessary to perform regular artifact checks, including after an application has been deployed.

This can be done using special repositories or tools, such as Snyk Container, Trivy, GitLab Container Scanning, or developing your integration module.

Application Security Monitoring

Another way to ensure security is to monitor the application itself. There are several technologies available for this purpose:

• Web Application Firewall (WAF) is a traffic filtering tool. It works at the application layer and protects web applications by analyzing HTTP/HTTPS traffic.

• RASP (Runtime Application Self-Protection) is a technology that detects and blocks attacks on an application in real time. It adds a defense function to the runtime environment, allowing the application to self-protect automatically.

The main advantage of RASP over WAF is that it understands how the application works and detects attacks and illegitimate traffic. RASP can see not only typical attacks using signature matching but also attempts to exploit zero-day vulnerabilities by reacting to anomalies.

Unlike WAF, RASP works within and with the application. WAF can be fooled. If an attacker changes the application code, WAF cannot detect it because it does not understand the execution context.

RASP has access to diagnostic information about the application, can analyze it, detect anomalies, and block attacks.

RASP technology has a specialty of focusing on preventing attacks by default. The system does not require access to the source code.

RASP Fools

I recommend you:

• Fortify
• Sqreen
• OpenRASP is an open-source solution developed by Baidu
• Signal Sciences
• Jscrambler
• Hdiv
• Imperva

Great, move on.

Visualization of DevSecOps Pipelines Results and Vulnerability Management

At different stages of the Pipeline, I use many Application Security Testing/DevSecOps analyzers. Each of them generates a report in its own format, and some of them generate so-called False Positives. Because of this, it isn't easy to analyze the security of an application as a whole.

To effectively analyze vulnerabilities and manage the remediation process, specialized Vulnerability Management tools, often referred to simply as Security Dashboards, are used.

Security Dashboards solve the problem by delivering the results of DevSecOps analyses in a unified and easy-to-analyse form. In this way, security engineers can see what issues exist, which ones are most critical, and which ones need to be addressed first.

DevSecOps Tools

A wide range of tools are usually used as Security Dashboards: for example, classic SOAR/SIEM systems and the specialized DevSecOps orchestrator AppSec.Hub from Swordfish Security or the open-source vulnerability management toolkit DefectDojo.

DefectDojo is a widespread tool. It is widely used even in enterprise companies. However, despite its popularity and solid age, this tool occasionally has some initial-level defects when contributors break the basic functionality in the commit.

However, what is nice is that DefectDojo developers and maintainers help to sort out the complexities. DefectDojo developers are very responsive people - we recommend contacting them directly via Issues on GitHub.

Shift Left Concept: Let's Summarize

Once again, here is a quick recap of what was in the article.

I explained what the Shift Left concept is and how it helps reduce the cost of bug fixes and development time: the earlier in the development process you start security checks, the better.

Then, I deconstructed the DevSecOps Pipeline structure and looked at what security checks are performed at each stage of the Pipeline:

• Pre-commit. It implies checking the safety of source code before it enters the version control system.

• Pre-build. It is a check of source code security in the version control system (Secret Detection, SAST, SCA).

• Post-build. It is a security check of an artifact built from source code (Source SCA, Binary SCA).

• Test-time. It implies testing the security of the application that is running from the collected artifact (DAST, IAST, and OAST).

• Post-deploy. It implies checking application security after deployment in the production environment (WAF, RASP).

Now, you understand how the DevSecOps Pipeline works. Now, you can assess the maturity of your DevSecOps pipelines, develop a roadmap for its development, and choose the right tools for each task.