Your CI System is a Terrible Compliance System of Record - Here are 5 Reasons Why

If Jenkins, GitLab, GitHub, or CircleCI is the engine for your DevOps it will contain a lot of information relevant to maintaining a Compliance System of Record (CSoR). However, your CI system shouldn’t form the basis for your CSoR and in this article, we’ll give you 5 reasons why.

Let’s jump straight in…

1. CI systems have lots of mutable data

The first objection to using a CI system concerns the single biggest challenge a CSoR should address: preventing modification and deletion of information. A compliance audit trail must stand up to scrutiny. Auditors need to know if process evidence can be modified by internal or external bad actors.

The problem with Continuous Integration systems is that they tend to have lots of mutable information. For example, it’s often trivial to delete stuff like pipeline runs and repositories. Data and evidence can also be lost in less obvious ways if pipeline reruns overwrite previous information, or if log retention policies delete old data. A robust CSoR doesn’t modify compliance data.

2. CI systems are hard to secure

A frequent counterargument to the mutable data objection is that you can apply security controls. However, “We could just lock down the CI server” isn’t the winning argument you might think.

Having a highly rigid security posture around your CI system makes it harder for developers to get work done because it imposes heavy processes around how CI changes. Things that used to be easy - like creating a new microservice or renaming a repository - now involve paperwork and risk theater. This leads to a lot of disillusionment in the tech teams and establishes security as an enemy.

But, beyond this cultural anti-pattern, this is still a poor compliance solution. By turning a compliance problem into a security problem you’re getting weak compliance and rigid pipelines - the worst of both worlds.

3. CI systems are not databases

The third reason why your CI system should not be your CSoR relates to using cases beyond audit trails. Compliance data is used in many contexts. It can be used in reports to regulators, in security monitoring systems, and in further operational automation such as deployment controls.

CI systems don’t have a structured data layout and query language in which to use this information. It is also difficult to add your own specific data to the historical record without abusing these systems.

4. CI systems provide poor change observability

CI systems are source-code-centric with user interfaces often scoped at the git repository level. This makes sense for a developer working on software, but the narrow focus makes it hard to reason about change in large distributed systems. Could your CI system give you good answers to the following questions?

Which versions of different microservices are running in production? What are the recent changes to production? What was in production last Wednesday evening? What deployments are missing approvals?

Beyond this, most CI systems don’t typically use reliable software identification methods like Content Addressable storage, so it’s hard to correlate the CI events with actual running software.

5. CI-based CSoR’s == DevOps lock-in

Finally, using a CI system as a System of Record relates to the overall technology strategy for your development teams. Forcing everyone into a locked CI ecosystem hurts your ability to adopt better and more distributed tooling.

If teams want to use different CI systems, or would naturally benefit from doing so, basing your audit trail on distributed and incompatible systems can make audits time-consuming and hard to navigate.

Tl;dr - Don’t do it!

It’s tempting, and seemingly logical, to base your CSoR in your CI system. But, as you can see, there are very good reasons to avoid it.

For one thing, there’s absolutely no guarantee that an auditor will be satisfied with a mutable CI system as the record of compliance proof. And if that wasn’t disqualifying enough, you risk undermining developer productivity with a rigid system that offers poor observability and locks in your tools. It’s a lose/lose/lose situation. What should a robust CSoR look like? Inverse the 5 pitfalls I describe above and you should have more success.

Succeeding with DevOps requires sharp tools for specific jobs - you always want to have round pegs in round holes. Building a successful CSoR in a DevOps way is no different. It shouldn’t come as a surprise that using your CI system for this task is full of inefficiency and risk.

Also Published here