Seamless Implementation of Safety Practices in the DevOps Pipeline

Hi HackerNoon Community!

While exploring the platform, I was surprised by the number of expert articles on DevSecOps. My previous article, How to Turn A DevOps Pipeline Into a DevSecOps Pipeline: A Shift Left Concept Overview, received much attention from the community. That's why I decided to continue covering this topic, and today's article is dedicated to the seamless implementation of safety practices in the DevOps pipeline.

I also recommend you read my 2 other articles on DevSecOps:

• DevSecOps In The Cloud: How to Manage Security With CSPM
• The Benefits and Challenges of Implementing DevSecOps

At the 2023 World Economic Forum (WEF) in Davos, experts presented a ranking of global risks that will be relevant in the coming years - cyberattacks on critical infrastructure are in the top 10. At the 2022 World Economic Forum (WEF) in Davos, cybersecurity was also in the top 10 worldwide risks. This is because the development industry is growing. Today, it is already hundreds of billions of dollars, and security problems, often in the code or artifacts, are solved slowly or ignored altogether.

In large organizations, the ratio of developers to security professionals, specifically Application Security, is disastrous - 100:1. In such a situation, it is hard to expect a small team of AppSec specialists to cover all the code that developers create, catch all vulnerabilities and provide software products with reliable protection against existing cyber threats.

So, let’s discuss the seamless implementation of safety practices in the DevOps pipeline.

Industrial factors

Industrial factors must be considered as part of a safe development process. First and foremost, these include business, development, and safety.

Business

This includes functions, services, and digital applications, the need for which is increasing daily. Business is the driver of the emergence of such software products. It is the driver that stimulates the continuous growth of services and applications. Consequently, the number of customers, stakeholders, and engaged people is increasing.

What do they all care about? Of course, they care about reducing time to market (time gap from the beginning of idea development to the final launch of the solution and its release to the market).

How does it work in their understanding? If they come up with an idea, it means that it should be realized conditionally tomorrow in mobile, desktop, or web applications for end users.

Development

Under the influence of business customers, the development cycle in terms of time to market is shortening. There is a shift to microservice infrastructure, which allows rolling in the necessary changes.

Cybersecurity

Rapid development and many releases provoke the growth of technical debt (I have written about it in my previous article How to Sell A Technical Debt From a DevOps Perspective?) and the number of information security defects. If tools are used to track these defects, the number of fouls in them and the number of triggers increases. All this, together with staff shortage, creates many problems when releasing secure software.

Apart from this, malicious actors are also highly active. How are digital threats evolving? Gits, the stock development process, and coding threaten to bypass code reviews and compromise the storage of products and components borrowed from GitLab, GitHub, and other services.

Attackers also have many attack vectors once the code is in the build pipeline.

First of all, it is possible to compromise the CI system itself. If it is infected, everything that passes through it will also be compromised. Another vector is the modification of those systems that are components of CI. Other opportunities are opened by using Open Source software that can also be distributed with the functionality of CI systems.

Open Source products are often embedded with malware. I mean a compromise of external packages and dependencies that developers pull from available repositories. There are more and more cases of malware, protestware, known public vulnerabilities, zero day (0-day), and much more.

In the delivery pipeline, the attack vectors are partly similar to those listed above and partly unique to CI. These include compromising CD systems and infrastructure environments. Among the attackers’ oldest (most mature) practices is the compromise of endpoint interfaces.

Secure development will mitigate the risks from these vectors and regularly analyze the security of products as they are being built.

Goals of DevSecOps

So, what are the goals of the DevSecOps methodology? Let's take a closer look at each of them.

Time To Market (TTM)

DevSecOps helps reduce TTM. It would seem, how? New acceptance criteria have been introduced, and vulnerability scanning tools have appeared, so the TTM cannot decrease. But as experience shows, this indicator is reduced on medium and long-distance projects.

Tech debt is already there from the point of view of information security. However, within the framework of DevSecOps, the company will have an opportunity to find vulnerabilities at the earliest stages of development, eliminating them faster and at lower costs. This will have a positive impact on time to market. Besides, the maturity level of the development team will increase not only in terms of information security but also in terms of code functionality.

Compliance

Regulators have seriously started to develop and issue regulatory documents that become obligatory for some companies. First, these organizations include those that are part of critical information infrastructure (CII), such as banks that release their applications. There are no binding regulations for other companies yet, but in time, all of them will have them.

One of the goals of DevSecOps is to align development with standards from regulators and industry best practices to ensure a high level of maturity in terms of documenting the process.

Another important point for lawyers in terms of enforcing the restrictions imposed by agreements is keeping track of the license purity of freely distributed software.

It may happen that a component or a library is free, but the license agreement says that every Thursday of the month, the person using the software must send the author a case of beer. And this is not a joke but an absolutely real example. But there are more serious things when, for example, the license policy obliges companies that use a publicly available component to make their software freely distributable.

Digital Business Continuity and Security

DevSecOps practices ensure a company's software output and digital services are secure. Improving the cyber resilience of a single remote banking system or two within a large bank is something the industry has dealt with for a long time. But there are usually dozens of such systems in mature FinTech and Telecom companies. Not all of them are covered by security approaches. As a consequence, the density of defects is not reduced. But they will eventually disappear if you constantly monitor the same vulnerabilities and tell them how to avoid them.

Developing Maturity In Code Design

When Application Security is implemented, developers look at it skeptically at first. They need to evolve in Security to know how to fix a particular vulnerability. They must know why it is considered dangerous and whether the method highlighted by the tool or Application Security specialist in the library is being used in their code. In the end, adopting secure development approaches increases the maturity of the software development processes.

Methodology

Let's talk about methodology and how to approach it. To build a safe development process, you cannot focus on one thing, for example, only on tools or regulations. You need to work in three domains or vectors - tools, processes, and competencies (i.e. people themselves).

If you take one of the representatives of the Gartner quadrant on static code analysis and introduce it into the development process, it will give so much white noise that on the second or third day, you will want to turn it off. It is impossible to work with so many fouls. This situation will happen if you don't think over the process of analyzing fouls and formulate a role model (zones of responsibility, SLA conditions, and so on). That is, to work with such a tool, you need the appropriate competencies.

So, let's talk about the three domains I mentioned above. The 8 marked in the figure below symbolizes the cyclic development process, which many aspire to and some already put into practice. Different Application Security practices or classes of tools are needed at different stages of software production.

In the design phase of a new feature or system, the Open Source Analysis (OSA) practice comes in handy when the developer is just building certain external libraries through ADE. This practice keeps unsafe components out of the development loop. Considering that software today is practically not written from scratch, and external components are reused to create it, this is a very actual practice that should not be forgotten.

When it comes to application code development, Static Application Security Testing (SAST) is one of the most mature practices used in most large organizations. Building a dependency tree is another mature practice used in most large organizations. It is about identifying potential vulnerabilities, further parsing, and eliminating errors.

When the code turns into some artifact in a CI system, it is also important to look at direct and transitive dependencies present in it from the viewpoint of their security. The question may arise here as we have already restricted external unsafe libraries. They do not pass our protection profile to get into the loop. So, why do we need another check at the build stage?

First of all, vulnerabilities appear in freely distributed libraries every day. Secondly, there were cases when we blocked a library at the Open Source stage, but a developer decided that he still needed it while writing code. Such cases are caught within the Software Composition Analysis (SCA) practice.

Move on.

When a system gets into UAT, pre-production, or test, you can already play with it with Dynamic Application Security Testing (DAST) tools. You can try to illuminate it through the interface, look at API requests, find out which are false, try to imitate the actions of intruders, etc. Of course, it will not be the same as what hackers do, but obvious bugs and even not very obvious errors can be caught with the help of tools and methodology.

After that, the system gets to the operation stage, where someone has already built a microservice infrastructure and is still cutting their monolith architecture into "pieces". Here, it is important to think not only about analyzing the container images themselves but also about their inter-network interaction and network policies. There are quite a few practices that enable the so-called CSP platforms necessary for an effective, secure development process.

Application Security Orchestration and Correlation

When a company has such a huge Application Security suite in place, questions arise:

• How do we get it all up and running on time in CI?
• With what settings?
• For which team?
• How do we gather information in a single window?
• How do we track metrics and process improvement dynamics?

There are Application Security Orchestration and Correlation (ASOC) tools to answer all these questions. We'll look at them in more detail a little later.

Building Centers of Competence

Let's move on to the second domain, i.e. people. Here, I will talk about the creation and development of competence centers.

Applied training is necessary for developers. It will expand their knowledge of security and allow them to create secure code, avoid errors in architecture, and so on. You should also not forget about Security Awareness. It can be used to educate many people within the company about why security is important and why you need to do it. This will free up developers' time, make the process more mature, and improve the work of related departments. In this way, the company will gain allies rather than people who might go on an "Italian strike".

In addition, as part of creating a competence center, you can activate the work on the internal portal and put on it the unified rules that the company uses in terms of Application Security, vulnerability reviews, articles on security topics, and so on. In general, anything that can be useful. It seems like an obvious thing to do, but it is often not used in very large companies with mature processes. Also, do not forget about tools like email newsletters, seminars, meetups, webinars, and CTFs.

Activities implemented within the competence center are also useful because they allow you to select the most interested employees from among the trainees who are suitable for the role of Security Champions. These are representatives of development teams, architects, or the operations department. They perform the role of security guides in their teams. They are interested in the information security domain. They additionally evolve in it, share their knowledge with their colleagues, and thus interest them in security. In other words, they are agents of the security team, "recruiting" members of other departments into information security, considering the necessary depth of immersion.

Implementing the practice of Security Champions is not easy, but creativity can help a company. You can devise activities or rewards that will motivate people to do it—for example, a bonus or the promise of an exciting business trip.

Process building

In large companies that have been building up the maturity of their processes for a long time, there are situations where volumes of various instructions are clearly structured, written "beautifully", and accessible. Still, the problem is that they do not work. What is set out on paper does not correspond to what exists in life. There are other problems when there is no interaction, i.e., processes are not formalized, not described in regulations, and not fixed anywhere.

It is important to describe processes, but it is not necessary to write huge volumes of documentation. It will be enough to have short and clear memos that tell where and whose area of responsibility it is, how, and through which communication channel to contact Application Security. Then, this tool will be effective and can make life easier for teams.

When we build processes, we rely on BSIMM, OWASP SAMM, Open SAMM, and others. Let's consider our approach using these methodologies on the example of static analyzer implementation within the framework of expectation and reality.

Expectation. What is there difficult to implement? We choose a tool, buy it, and replicate it on the whole code base, and that's all. But it doesn't work that way.

Reality. Instead of 3 steps, we have to take, for example, about 30. These are defining goals, forming a test team, setting up rules, reducing the False Positive rate, replication, gradual change of Quality Gates, and many other things without which it is impossible to get a good result.

The Importance of Orchestration

With ASOC (Application Security Orchestration and Correlation) solutions, you can manage Application Security tools, track metrics, and correlate data. Products that implement ASOC practices enable the most seamless (as much as possible) implementation of AppSec practices into existing company processes.

The premise behind the need for such tools is that the DevOps landscape is dynamically evolving and becoming increasingly complex. The number of so-called Quality Gates is growing, and we need to be sure that our object (code, artifact), built applications, and libraries are safe and meet all requirements. But as the DevOps landscape becomes more complex, it becomes more difficult to automate Application Security practices.

Many companies are trying to do it in separate pieces, i.e., implementing partial automation. This approach leads to endless projects involving internal and external resources. The main problem is that there is no single window, ideology, or tool around which to start working. Therefore, I recommend an approach that uses solutions that implement ASOC practices. They will allow you to integrate AppSec practices into DevOps and automate their work quickly.

ASOC

So, what are the ASOC tools? They include 3 domains - orchestration, correlation, and analytics.

Orchestration

Let's consider this domain from the point of view of the tasks it implements:

• Connect all scanners with necessary rules for specific commands.
• Set up their interaction with code bases, artifact storages, etc.
• Set up tracking of changes to units and objects in these bases.
• Identify software quality control points at different stages of development.
• Launch information security pipelines.

All of this is accomplished as part of orchestration.

Correlation

Next, we see the entire process within a single window. The scanners are looking for vulnerabilities, and we can work with them in the context of the Correlation domain.

Within the Correlation domain, tools confirm vulnerabilities and eliminate false positives. The second big block in this phase is grouping and identifying root vulnerabilities. Scanners sometimes talk about the same issues but in different languages.

Understanding which root vulnerability will allow you to remove as many triggers as possible is important. It would also be great if ASOC tools had some kind of math model to highlight the probability of where the fouls are and where the real triggering is.

It's also important to talk about two-way integration with bug tracker systems. The scanners gave the entire list of triggers in a single window. We noted which of these are real problems. Then, all this is sent to the bug-tracking system as a task for developers to eliminate vulnerabilities. Here, it is necessary to establish two-way synchronization to determine whether the task is closed and whether its result will be confirmed at the next scan. This is also part of the correlation tasks.

Analytics

Metrics can be used to explain to yourself as a customer, to management, and to teams in general how the secure development process is evolving. This will allow you to evaluate work effectiveness in this direction and establish process management. You can measure everything: code base coverage, defect density, time to fix defects, etc. ASOC tools allow you to do this.

DevSecOps - Platform as a Service

Suppose it is September, Q4 (the beginning of the most important sales period), high workload. You got a call from a respected client: "Everything is gone. The management told us everyone else has AppSec, but we don't. We need to have it by the end of the year. Can we do that?"

Actually, you can. But how do you approach it?

If the task is to start and replicate Application Security practices as quickly as possible, there is no time for big planning. Forming an internal competence center, selecting all necessary tools, and piloting on different teams is the right approach. Still, it takes months and sometimes years.

You can implement DevSecOps in a Platform as a Service format as part of a quick start. There are expert teams that already have an understanding of which tools for which code bases are worth using. These products are already deployed in internal labs, and to quickly implement the secure development process, you can outsource such a team.

They will come with their tools and basic documents and immediately start replicating AppSec practices. And most importantly, these specialists will parse the results provided by the company's team and produce something that really works. But it's worth noting that a quick start is more expensive than a planned approach.

However, within the framework of DevSecOps implementation, you can run into an insidious problem. The company has paid for building a secure development process, receives a service of proper quality but does not build up internal expertise, does not hire people, and does not get into new tasks. This can lead to the organization getting "hooked" on an external provider and becoming dependent on it, leading to additional expenses.

How do you determine a company's readiness to implement Application Security and DevSecOps?

Many frameworks have DevSecOps maturity models. With the help of ready-made questionnaires, it is possible to understand on your own or with the help of external auditors what stage of maturity the company is at now. You can make a pie chart of the eight key security development vectors. This will allow you to see what is not currently applied in the organization and what is relevant and already in use. You can also create a roadmap and, based on it, move forward.

What is more important - Dev, Sec, or Ops?

DevSecOps combines development, operations, and security. The question often arises: which of these units is more important? Let's start by understanding what each of these units cares most about:

• Development. We're on fire with deadlines and KPIs! And actually, the VP won't get a bonus if we don't release this by a specific date.
• Operations. You guys are out of your minds! We can't release so many new features without fully testing them. It won't work. We can't handle this many customer requests.
• Security. The most important thing is that everything must be secure.

Thus, it is impossible to single out one thing in this trio. Everyone is important here. DevSecOps is a process that forms a single team from disparate departments, striving to improve the product.

Conclusion

Today, DevSecOps is no longer some newfangled trend you can safely do without. In today's conditions, process practices are mandatory for everyone who has something to do with software development. It is only important to decide how much to implement them. No tool is a bullet, so when implementing it, it is important to think about people and the existing processes of the company, as well as about the possibilities of their interaction with each other.

Before starting to build a secure development process, it is important to assess the maturity level of the company, understand where it is at in its development and what it wants to achieve, and then you can implement tools. Today, cyber threats are actively evolving, and there is nothing you can do about it. But you can strengthen your position and try to stay ahead of the curve.

Orchestration technologies and formulated metrics for assessing the effectiveness of the process will help with this. They will allow you to introduce new practices and analyze progress, i.e., keep your finger on the pulse.

In addition, it is necessary to develop internal expertise: train employees, implement the practice of Security Champions, etc. Without the necessary competencies of specialists, the company will not be able to get ahead. If the team does not yet have the necessary knowledge and the organization needs to urgently implement security practices, it is always possible to engage external specialists and, with their help, build all the processes, including those related to employee training.