Preface The first few days of this vulnerability disclosure were a very interesting ride. It started with posting (which you should definitely read). Most people blew him off because they thought he didn’t understand how certificate pinning worked in Firefox and that his setup would inherently bypass pinning by design. The funny thing about that is that they were right (while still being very wrong). Going in, didn’t have a great understanding of how certificate pinning in Firefox worked and neither did I for that matter. To add, had he done his attack on Firefox instead of Tor Browser, his result would have actually been the same even if a vulnerability didn’t exist. It’s kinda hard to blame anyone for blowing off his attack given the totality of the information available at the time. I almost did too. movrcx his attack on Tor Browser movrcx The only reason I caught onto anything being weird is that I had a vague memory of Tor Browser hardening how they handled pinning by enabling a stricter level of enforcement. Because of that hardening, ’s attack shouldn’t have worked in Tor Browser using his setup. That led me to dig in and explore the issue more deeply to the point that I could pinpoint the source of the problem, and then submit to ’s to get more eyes on it. Within hours of it going live, the root of the problem was pointed out to me via e-mail by of the Tor Project. Then, over the course of the afternoon it was determined that this problem would actually for a few days in November and then for in December into January. As of this writing, that’s still the case although Mozilla has vowed to fix their processes to prevent it from actually happening. A bug that presents itself on schedule is pretty interesting. movrcx my own post on the subject daveaitel Daily Dave mailing list Nicolas Vigier come back 5 WEEKS The point of this writeup is give the down and dirty technical specifics on how and why this attack works in detail. My first post had some speculation in it but the facts are known now and patches have already started rolling out. This will lay out the facts without all of the speculation (unless otherwise specifically noted). Plus, I will show that not only would this vulnerability have come back in the future but, more importantly, that it has a few other times in the last couple of years in both Tor Browser and Firefox without being caught. already happened Certificate Pinning Certificate Pinning is the process of forcing a browser to only use certain certificates in the validation of a TLS connection to a certain domain. This is done by either a static certificate pin list included with the browser or using a standard called HTTP Public Key Pinning (HPKP) which allows a site to push down its own pins on the first connection to it. Mozilla uses Certificate Pinning to protect connections to (AMO), which is used for the updates of most Firefox extensions. The purpose of pinning this domain is to prevent a rogue CA from being able to generate a certificate for AMO that could then be used to perform a man-in-the-middle (MiTM) attack on the extension update process. addons.mozilla.org HTTP Public Key Pinning (HPKP) has a “chicken or the egg” issue with regards to getting the certificate pin to the user. The protocol is a trust-on-first-use (TOFU) protocol. The pins are sent by a website to the user on the first connection to the site. That means that first connection is not protected by pinning. Worse yet, if an attacker were to MiTM that first connection, they could push their HPKP pins instead. To combat this, both Chrome and Firefox (IE/Edge does not support HPKP) include a HPKP pre-load list bundled into each release of the browser. The pins are included in there to protect any connection (like the first one to a website) that isn’t protected by pushed HPKP pins. Once a site pushes its own HPKP headers to the user, those pins then override the pre-load list. own You can also put a pin in this pre-load list even if the site you include doesn’t push HPKP headers on the first connection. These are referred to as static pins because they are never overrode by HPKP. This was the case for . Pinning for AMO relied solely on the static pin that was bundled with the browser. addons.mozilla.org The Vulnerability When HPKP headers are pushed to the user, they are sent with an expiration date. This is necessary because certificates need to change sometimes because they expire, are compromised, are found to be using insecure signatures, or for any other reason. If the pins didn’t expire, the site would be stuck with the certificate they pinned indefinitely. It’s up to the site to determine what the expiration date should be. This also means that the static pins need to expire. If the user doesn’t update their browser, and these pins don’t expire, a website that changes it’s certificate in the future would be unable to receive connections from the legacy browsers. That’s why it’s also important to stop enforcing pinning when they expire instead of having it fail. The vulnerability here is that Mozilla failed to set the expiration date for the static pins and HPKP pre-load list long enough into the future to last until the next release of Firefox. Let’s look specifically at the last line of the code that sets the pin expiration. The specific file is generated on and included in the source code for Firefox 45.3.0esr released on (the current and vulnerable version as of this writing which is set to be replaced by 45.4.0esr on ). StaticHPKPins.h 25 Jul 2016 02 Aug 2016 20 Sep 2016 static const PRTime kPreloadPKPinsExpirationTime = INT64_C(1472903978258000); The expiration time is expressed here in epoch time. here converts to . 1472903978258000 Sat, 03 Sept 2016 11:59:38 GMT To review, was generated on Firefox 45.3.0esr was released on The pins expired on Firefox 45.4.0esr doesn’t release until . That means from until the user installs the new version of Firefox released on , the static pins are expired and pinning is not enforced. StaticHPKPins.h 25 Jul 2016. 02 Aug 2016. 03 Sept 2016. 20 Sept 2016 03 Sep 20 Sept 2016 Tor Browser is based off the ESR release so the expiration dates and lack of pinning enforcement are the same in this case. The regular Firefox release faired a bit better because its pins didn’t expire until a week later on making it vulnerable for 10 days instead of 17 days. 10 Sep 2016 The Fix Mozilla has an automated process for updating this expiration date and has blamed this issue on that process failing. Daniel Veditz from Mozilla Security describes the issue in the Mozilla Bug Report In a , Mozilla vowed to add HPKP to . They also promised to ensure that the pins won’t expire before the next release can extend the expiration date. Here is their statement: statement obtained by Dan Goodin at ArsTechnica addons.mozilla.org We investigated this and a fix will be issued in the next Firefox release on Tuesday, September 20. We had fixed an issue with the broken automation on the Developer Edition on September 4, but a certificate pinning had expired for users of our Release and Extended Support Release versions. We will be turning on HPKP on the addons.mozilla.org server itself so that users will remain protected once they have visited the site even if the built-in pins expire. We will be changing our internal processes so built-in certificate pins do not expire prematurely in future releases. Mozilla seems to have followed through and has indeed started to roll out HPKP by turning it on for . versioncheck.addons.mozilla.org Jason Thomas from Mozilla confirming the deployment of HPKP in a Bugzilla ticket Rolling out HPKP now has helped Mozilla protect the current version of Firefox until the next version is released on . It’s not a complete fix but I personally believe it reduces the risk to a point that is well below Firefox’s threat model. 20 Sept Tor, on the other hand, has a much different threat model. They realized this and decided to take the version they had compiled for the release and release it 4 days early on . They deserve credit for their quick reaction. 20 Sept 16 Sept What’s Old is New As I mentioned in the Preface, this bug is scheduled to make a reappearance. This is explained : here Daniel Veditz of Mozilla Security describes how pins will continue to expire Had this bug gone un-checked, it would have came back for 3 days in November and for from December into January. This made me wonder how often both Firefox and Tor Browser have been vulnerable to this in the past. It coming up 3 times in 4 months seemed pretty excessive to me. 5 weeks I brought this up in the #tor-dev IRC channel on OFTC and graciously offered to compile that data for me for use in this write-up. He compiled the info for Firefox Stable, Firefox ESR, Tor Browser Alpha, and Tor Browser Stable. It is an impressive dataset and you should definitely check it out. Matt Nordhoff _firefox-pin-data - Firefox and Tor Browser pin data_github.com mnordhoff/firefox-pin-data While Firefox and Tor Browser have both been vulnerable in the past, it turns out that it hasn’t happened that often. Lets look at few of them and pull some dates. Here are the stats for Firefox ESR: As you can see, with the exception of the current release, the only other time that the pinning has expired in the past for Firefox ESR is from . However, that’s a pretty significant time window. Also, it appears actually shipped with the pins already expired. It wasn’t until the release of that pinning was reinforced. There was also only a 4 day window to update between and . 04 July 2015 to 11 August 2015 38.1.1esr 38.2.0esr 45.0.2esr 45.1.0esr Now, lets look at Tor Browser Stable: You’ll notice that Tor Browser had expired pins until the release of . This is because Tor actually backported the pinning functionality but likely weren’t aware they needed to update the expiration date. It’s an honest mistake and it’s fair to say that Tor Browser didn’t really support static pins until version . However, outside of that, Tor had not been vulnerable before until this most recent release. They actually faired pretty well. 5.0 5.0 Conclusion and Acknowledgements This has been a pretty interesting bug to track down. It shows how failures of process can be very difficult to spot when you’re dealing with a complex codebase and there is no indication that something is failing. I’d be lying if I said luck didn’t play a significant role in the discovery of this bug. If had tried his attack before or after , it would have failed in his tests. It’s only because he conducted it within that 17 day window that this was discovered. movrcx 3 Sept 20 Sept Mozilla is also so an attack like this wouldn’t work even if you were able to break certificate pinning again in the future. This vulnerability was actually only limited to extensions that updated through AMO. Extensions that follow their own update process were unaffected. The update process in the version included with Tor Browser was not affected because they do updates via instead and do signature verification of the updates once they are downloaded. However, the version delivered by AMO is likely affected the same as all of the others. looking into how to harden the extension update process itself HTTPS Everywhere EFF.org Here are some acknowledgements I’d like to make: — Without his initial discovery, none of this would have been unwrapped. He was also extremely easy to work with while we were combing through the code to figure out the root cause of this bug. movrcx — Her help while I was trying to figure things out was invaluable. She originally pointed me to the fact that Mozilla was using static pins outside of HPKP. Erinn Atwater — For publishing my initial writeup on and being generally responsive to me, even while busy in Germany. I definitely recommend subscribing to his mailing list. daveaitel Daily Dave — For being the first person to point out the expiration date issue to me via e-mail. Nicolas Vigier Georg Koppen — His swift reaction and decision to release the next Tor Browser release early should be commended. He quickly understood the issue, understood its impact, and took action. — For compiling the stats on Firefox and Tor Browser pins in such an extremely clean and usable way and for his dedication in making sure it was accurate and contained the whole picture. Matt Nordhoff #tor-dev on OFTC— These people were extremely responsive once I reached out and their transparency in talking through a problem is not something you see from most development teams. I hope this was as interesting for you as it was for me. I feel like there are a lot of lessons to be learned from this whole process.
