In the beginning, a single developer worked on 100% of the code base that lived on their machine. Inevitably, at some point, they wanted to share their code, or back it up, or even create a simulated testing environment. On that day, they began the process of moving from Dev to DevOps. Since then, the world has changed drastically.
Code bases are filled with countless dependencies, distributed source management systems, and even a proper integration and deployment process. Your software can go worldwide in less than 5 minutes, and with that great power, (say it with me Spiderman) comes great responsibility. Let’s cover 5 DevOps security challenges in 2019… and beyond!
Challenge 1: Dependency Madness
We live in a crazy new world where the code we use is based on THOUSANDS of other sources and commits. Package managers exist in every successful language. The most popular language of 2018 by far was JavaScript, which is supplied by online package managers like NPM. Taking a look at the statistics and claims that come from these package managers, lets you know just how dependent our “move fast and break stuff” world of agile development has become.
According to npm Inc:
“When npm, Inc. started in 2014, a tree of a few dozen JavaScript packages was typical. These days, the average modern web application has over 1000 modules, and trees of over 2000 modules are not uncommon. In fact, 97% of the code in a modern web application comes from npm. An individual developer is responsible only for the final 3% that makes their application unique and useful.”
Wow! This sounds like speed has really increased. With all this freedom, it’s time for a gut check on what it looks like on the security side. In a popular Hacker Noon article, the author claims in the title, “I’m harvesting credit card numbers and passwords from your site. Here’s how.” While your code is dependent upon dependencies, those dependencies have their own and so on. All it takes is for one library to integrate a bad actor. It’s getting harder and harder to see your dependency graph, in a human-readable form.
Fortunately, many services exist for checking and updating you when vulnerabilities and malicious code is found. Even recently, GitHub has added a Security tab for automatically flagging such issues for free!
Keep track of dependencies and ensure you have an up-to-date tracker on your team’s codebase. These are not problems you should ignore.
Challenge 2: The Credential Juggle
As API’s grow to third-party vendors, more and more API keys for microservices and functions fill our apps. With these advancements, we are putting expensive and sensitive keys into the hands of our developers. What stops one of these team members from taking these API’s and using them, distributing them or racking up some large API fees?
The Credential Juggle, is a complex and difficult act for any large team. While each technology is vastly different, there are a few consistent methods for protecting the keys to the kingdom.
A Staging server is by far the simplest method. The keys to the Staging server are given to everyone. The Staging server is limited, and any actual API keys to external services are either mocked, or wiped, or changed frequently enough to dissuade a team member from abusing it. If possible the Staging server is inaccessible to the outside world.
In some situations, the platform doesn’t allow multiple teams. Inevitably in these situations, I’m looking at you Apple, other tools are created that help you manage your credentials. In the Apple example, I highly recommend a tool named Fastlane.
You know your system is handled right when, hypothetically, if a developer’s laptop were stolen, your main product would not suffer.
Challenge 3: Licensing After the Fact
Legal systems are hard, and more so in Software. International law isn’t something a person can throw an hour at, and walk away with deep clarity. We’ve seen licenses change to push companies out via political statements, and if you want your software to be used by certain institutions, you have to make sure your code and all your dependent code is properly licensed! What a headache!
The best way to handle this is to identify your legal license upfront and maintain it. You can start by using TLDR legal to identify the license you need. Then use a free service like FOSSA to verify each commit to your project doesn’t introduce code that violates your intended license.
Identifying and sticking to the plan early isn’t only the best plan, in code bases that have grown over years, it’s the only plan. Deep seeded dependencies are inextricable from mountains of code.
Challenge 4: Single Provider - Double Headache
You’re using a popular service, backing up, keeping versions, and building a product. All is well right? If tomorrow, one of your services disappeared, would you be able to recover? DO NOT USE the same provider of your hosting, as your backups. The Internet is filled with sad stories of companies believing in a single product and finding out all too late that it is fallible.
Most recently, here’s a story of how Digital Ocean caused an entire business to fail:
While Digital Ocean has learned from this mistake, the bigger lesson is that we should all learn from it. If your server is provided by X, then your backups should reside on Y. Relying on a single provider, while attractive for DevOps simplicity, will result in double the headache.
Challenge 5: Developers! Developers! Developers!
The last and most crucial part of security is people. You can have the longest, most quantum-secure crypto in the world, and if your team writes the passwords on sticky notes, you’re vulnerable. Proper DevOps means proper process with people.
So for Challenge 5, it’s the most complicated problem of all. It’s about DevOps planning and communication. Not just in what is best, but in what is happening. A proper DevOps process has to understand and entertain the human factor of your team.
Does your process properly communicate “Why?” Does your team feel comfortable elevating communication and being candid about what is happening? Are the roles clear and defined? The biggest technical problems at most companies, when you really dissect it, end up being a simple problem of communication. Conway’s law is the simplest and yet the most complex aspect of DevOps. As the adage goes:
“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.” — M. Conway
Review, discuss, and share these 5 DevOps Security challenges with your team.