Whether you're starting to code your first blockchain application or are a CTO overseeing an app that manages billions of dollars, security is crucial. In the world of blockchain, security breaches can lead to devastating losses. Amidst developing new features and satisfying customers, the risk of a security catastrophe is always present.

Over the last years as the CTO for Yield Protocol, I've learned how to mitigate risks and address complexities that often go unnoticed. While the risk of a security breach still exists, now it is a managed risk, that I can dial up or down as needed.

In this article, I'll share my approach to blockchain security. I'll discuss various threats and how to safeguard your application against them. Additionally, I'll provide practical ways to enhance security within the constraints of your resources.

Threats

In the blockchain sphere, threats appear to be lurking at every corner. Just a quick glance at the disaster leaderboard at Rekt will make this clear. But what causes these disasters?

I categorize threats into four groups:

Bugs can appear in many forms and are typically triggered by users shortly after new code is released. In 2021, for instance, a bug in a Compound Finance upgrade resulted in an unplanned $80M payout to users before the platform could halt operations.

In the Yield Protocol, our own release of YieldSpace-tv encountered a similar issue. Just before launching, during a final test of the new features, all funds from one pool unexpectedly moved from the protocol to a developer's wallet, prompting immediate alarms.

Hacks though originating from faulty code like bugs, are different as they require a skilled developer to exploit them. Consequently, the exploit might happen much later than the release that made the application vulnerable. Worse, they might involve a single transaction that drains all the money in seconds.

A recent case involved Euler Finance, which lost $200M due to a hack. The attacker exploited a bug introduced during an upgrade months before. The attacker deployed custom contracts that abused the liquidations system with flash loans, and suddenly extracted all the funds in the application. Our own contracts were affected as they had deposited part of the stolen assets on behalf of our users.

Errors in governance can also lead to disasters. The first version of Yield Protocol v2, for instance, used an incorrect address for WETH. Fortunately, I noticed this before user funds were at risk, but not before I had wasted 20 ETH on deploying now-useless contracts.

We've since corrected other governance errors, for example related to FRAX lending limits and liquidation parameters. While these issues had limited impact, they revealed the potential for serious consequences from such errors.

Rugs are exploits on the governance system. The most prominent example is the rug on the Ronin Network by the North Korean Lazarus group. In these scenarios, either the key-holders misuse their powers, or a malicious actor acquires governance rights due to lax security practices.

Although I've got no plans to rug anything, and I’ve never been rugged myself, this threat vector is always a significant concern. In the early days of Yield Protocol, I held the governance keys as I was deploying and setting up everything. When the total value locked (TVL) exceeded $1M, I took steps to implement a multisig system, reducing the risk of a rug. Having sole control over the funds was indeed an unsettling experience.

While thinking about all these potential threats can be daunting, categorizing them allows for targeted, effective responses. By breaking them down, we can address each threat individually, carefully choosing mitigation measures to ensure comprehensive coverage.

Mitigations

If you're reading this, you're probably already aware of some security measures for blockchain applications. Understanding which ones you need and why can be challenging. Here, I'll explain a few of them in relation to our four threat vectors.

Unit and integration testing

Primarily, unit and integration testing safeguard your application against bugs. With unit testing, you verify that specific scenarios function correctly. But remember, anything you don't test directly might slip through the cracks.

Phased Releases

Phased releases, where you set clear limits for the initial launch, are a sensible strategy. For example, you might restrict users to borrowing no more than $100K in the first week. This way, any bugs get exposed without causing massive user losses.

Invariant testing and formal verification

Though challenging to set up, invariant testing and formal verification are valuable. They help catch edge case bugs that you might not actively be looking for. Such bugs could be exploited in a hack, making these tests an essential security measure.

Code audit

Auditing involves hiring third-party experts to find common bugs and potential design failures that could lead to hacks. They use their specialized skills to spot edge case bugs you might miss.

Bug bounty

Despite the name, bug bounties are more about protecting your application from hacks than bugs. Being the last line of defense against a hack, it's advisable to have a bug bounty in place.

Governance review

At Yield, we've executed over a hundred governance actions, not without errors. Seeing no better option, we've developed our own system for internal audits of governance reviews.

Test harness

A test harness is essentially a form of unit testing applied to find governance errors. At Yield, we run each governance action in a fork, followed by a series of tests to verify the protocol's functionality under the intended configuration.

Multisig

A multisig setup can complicate an intentional governance attack, depending on the operational security of the multisig members.

Timelock

A timelock can hinder governance attacks by making the attack visible for a set period before execution. However, its effectiveness depends on someone monitoring the timelock and understanding its contents.

Decentralization

If governance is handed over to token holders, a governance attack becomes more challenging, especially if the governance tokens are widely distributed and difficult to acquire.

Permissionlessness

If the application is designed so that no changes can be made, it becomes impossible to introduce new bugs or execute governance actions. But remember, this also means existing bugs can't be fixed.

While there are other security measures and different ways to apply the ones explained above, this should give you a starting point to understand which security measures can provide decent coverage against all threat vectors.

The Swiss Cheese Security Model

Don't let the funny name fool you - the Swiss Cheese Security Model is a serious approach for safeguarding systems with imperfect defenses. According to wikipedia, this model is utilized across various sectors, including aviation safety, engineering, healthcare, and computer security.

Remember, all mitigation measures we've discussed are human-made, and humans are naturally prone to error. By layering different mitigation measures onto your application, you enhance its security. Considering multiple threat vectors allows you to ensure at least two layers of protection for each, with the hope that gaps in one layer are covered by another.

Here's our approach at Yield:

• Bugs: Unit and Integration Testing, Audits, Phased Releases.

• Hacks: Invariant Testing, Audits, Bug Bounty.

• Errors: Governance Reviews, Test Harnesses.

• Rugs: Timelock, Multisig.

  
  

Our security coverage evolved through trial and error, based on the incidents we encountered and our own ideas to prevent them in the future. Now, we're comfortable with our security coverage, and as I write this article, I understand why. We have at least two security layers for each threat vector.

This approach allows you to build your application's security measures while maximizing your resources. It should help you answer questions like, "Do I need a bug bounty, another audit, or formal verification?" If you encounter more bugs than expected, or a whitehat saves you from a hack, or a governance action goes awry, you'll know that you need to bolster security for that specific threat vector.

Continuous Security

A significant amount of effort is invested in securing a blockchain application before it goes live. We work towards total unit test coverage, perform formal verification, and have one or two audits done. After going live, we set up a bug bounty and a multisig, then monitor the TVL (Total Value Locked) charts.

If you're fortunate, your application will run for years, and your role will evolve. Instead of preparing for a grand launch, you'll work on numerous smaller, more frequent updates.

You'll handle governance requests to adjust individual parameters, minor contract upgrades to fix bugs reported via your bug bounty, the launch of new significant features involving new contracts, and many other chances to adjust your application.

Each of these interactions with the application should undergo the same security scrutiny as the original release. Arguably, these smaller updates may require even more attention, as the application's complexity increases with each modification. After a hundred changes to Yield Protocol, it is undoubtedly more intricate than it was at launch.

Furthermore, as your users and their stakes change, so does the risk. The danger varies if your application has $1M in deposits, which you can cover from the company budget, versus $200M, where a catastrophic hack could spell the end. Your public-facing developer team might not risk their reputations for $1M, but when the stake is $100M, the judgment of some may waver.

As your application evolves and grows, its security needs also change. While bugs were the primary focus before going live, the attention shifts towards governance errors and hacks as the application matures. As the TVL increases, you'll need to add more security layers to close potential gaps. It's essential to stay flexible and continually reassess your security needs.

Conclusion

Even though blockchain development has been around for more than ten years, many practices are still in their early stages. Despite the staggering amount of money in smart contracts, and the frequent occurrence of large-scale hacks, there's little literature offering a structured approach to securing a blockchain application.

In this article, I've provided a framework for understanding the potential threats to blockchain applications, as well as some possible protections. The Swiss Cheese Security Model, combined with this classification of threats and protections, leads to enhanced safety. I also discussed how the security model must adapt over time.

I use this knowledge in managing the security of the Yield Protocol. This article reflects the lessons we've learned from our security breaches over the past few years, thankfully none of them fatal. In a way, these security breaches have made me sleep better now than I ever did before, because I understand the threats, I know how to defend against them, and I realize that I'm not alone in protecting the protocol—everyone else is also busy maintaining their security layers.

If you managed to read this far, thank you. I hope that the lessons I learned will be useful to you.