I would like to share a recent experience with bug bounties. I’ve been doing bug bounties for years and I just now am trying-out the big boys on the playground — bugcrowd.com and hackerone.com — and the experience is… well, different.
The incentive is there, I guess, to be able to negotiate with someone which cares on delivery times and holding accountability against the firms you are assessing, but I never felt the need of involving a third party in the process, maybe because I’m not making a living out of it and doing it out of fun and genuine care since usually I’ve been discovering vulnerabilities on websites I’m already an active user on. The extra buck, T-shirt or mere mention on a Hall-of-Fame of some sort, is mostly for bragging rights. I’ll come back to some of my own two-cent level insights at the end of this post — but first, let’s go on and tell you about the vulnerability found on Heroku.
Heroku is pretty amazing, actually. They have an integrated PaaS solution that allows developers host their apps on the cloud with maximum ease.
Large as well as small companies are using Heroku non-stop, so I guessed that’s a good place for interesting vulnerabilities that might affect them all by spotting an organization-indifferent vulnerability.
Heroku has several options of interfacing with their service. Web-based is pretty common usage and API follows. There is a Heroku Command Line Interface (CLI) tool which basically is a wrapper to API calls. I used the CLI tool in order to contact the site and started on messing with some inputs and after a time I focused on the login mechanism which looked pretty simple. It is based on a POST request, accompanied by username and password parameters.
At first, when messing with the inputs I figured nothing really matters, I was hoping for an error output or any kind of response changes between real usernames against bogus ones. Even though I didn’t see any of those error text changes — there was one more subtle change in response.
A login request looks something like that:
POST https://api.heroku.com/login HTTP/1.1
User-Agent: heroku-cli/5.2.39-010a227 (windows-386) go1.6.2
And an appropriate response would be:
HTTP/1.1 404 Not Found
Cache-Control: private, no-cache
Date: Mon, 29 Aug 2016 16:49:52 GMT
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Xss-Protection: 1; mode=block
So, as said, nothing more than a 404, but on the other hand it is somewhat generous on it’s optional headers — “so what?” you might ask. That makes the whole difference. To be specific, the thing that caught my eye was the “X-Runtime” header.
From it’s name I could only guess what it is that I found. But it needed a bit more probing to be sure about it.
First stop is looking at the 1,000,000th time at the HTTP RFCs, noting that there is nothing like it in there — so it must be something added by another component on the web server. A short scan revealed it is most probably derived by a server side component relying on Ruby/Rails.
One of those projects, named Rack, was first to come up in search, that is to say that we can’t be sure of whether that’s the Ruby code that is operating on the site, but we’ll take it as a test case which proved to be sufficient at the end.
For those of you that never heard of it (like myself), it’s GitHub page reads:
Rack provides a minimal, modular, and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.
Peering into it’s source code, the aberrant header is explained within lib/rack/runtime.rb:
Now we can be sufficiently sure of our common-knowledge understanding of ‘X-Runtime’ meaning. It is referring to the time it took the operation to complete.
Are you thinking of what I am thinking?
To confirm this thought we have to get a larger test sample set against different mix of non-existent and existent user names on the system. And take note of timing differences and if there any distinguishable kind of delta.
Existent user names inclusion yeilded the following timing annotation:
And when using non-existent user names (I’ve used the correct scheme of e-mail address, just to be safe):
All of this assessment is funneled into one final conclusion — There is a clear distinctive (~1 order-of-magnitude) difference in times between bogus and true user names on the system.
Why is that even possible? Probably because of cryptography-related operations involved in the process of checking the password against it’s true hashed password.
It is a by-the-book example of a Timing Attack. Once the server receives the parameters from the user it first checks the user to see if it is relevant, if it isn’t the operation comes to an end and respond the 404 to the user. On the other case, if the user name is found to be relevant — it progresses forward-on to check the password given by the user against it’s local hashed copy of the password, which is a very intensive operation in contrast to the short trip to username-list that it followed. Because of that difference in operation time, we can perceive these distinctive routes.
Many applications are doing the same, but mostly because of latency it is somewhat harder to determine a definite conclusion upon a timing-based attack. In this case — the Ruby server is giving us all we need to get to this conclusion — all with one simple POST request. Cool!
This can be tackled on many levels. Since I’m not aware of Heroku’s considerations on keeping this header on, I will try to cover different approaches as there can be practical limitations of nearly each one of those.
According to Bugcrowd’s terms — User Enumeration vulnerabilities are not in-scope, and they actually penalized me for that in -1 points (honestly, I should have known better. To my defense it wasn’t clear to me that there is a separate terms not only from the vendor but from BugCrowd as well that need to be adhered.)
The thing is, as I mentioned before — that wasn’t my point, as many of my peers I wasn’t doing this for profit, I was trying to make a site I evaluated for using for my own needs and if happen to be, as in this case, prompt for security vulnerabilities. Whether I get the bounty is the secondary objective, the thing that worried me is that because BugCrowd consider it out-of-scope doesn’t make this vulnerability something we shouldn’t discuss. The terms should be there to be defined that not everything is going to make you a small buck on the side, but once you find something that is out of scope for bugcrowd why should it be considered out-of-scope of mitigation for the vendor?
I truly don’t have anything against Heroku nor BugCrowd, they both are comprised of great, passionate people from my experience, but bureaucracy in this case contradicts progress and I wish those unwritten procedures should be taken care of for the benefit of our community.