The 7th anniversary of the first major hack of Mt. Gox offers an opportunity to reflect upon software defects, human error, process flaws, and the best principles and practices for solution delivery in the IT industry. In this blog and my upcoming book, Bugs: A Short History of Software Imperfection, I will chronicle some important software system failures in the past and discuss ideas for improving the future of system quality. As information technology becomes increasingly woven into Life, the quality of software impacts our commerce, health, infrastructure, military, politics, science, security, and transportation. The Big Idea is that we have no choice but to get better at delivering technology solutions because our lives depend it.
On 19 June 2011, a computer hacker breached the security of Mt. Gox, the world’s largest bitcoin (BTC) exchange at the time. Suspicious trading activity followed soon after the breach, triggering a flash crash of bitcoin’s price from $17 USD to $0.01 USD on the exchange, during which the attacker withdrew a small amount (approximately 2000 BTC), and also leaked the exchange’s user database consisting of usernames and hashed passwords. Mt. Gox eventually collapsed around 7 February 2014 under the weight of processing delays and errors, further security incidents and possible insider fraud. Although the Bitcoin system and its community embrace cryptography and decentralization in theory, in practice, there is an extensive ecosystem of third-party intermediaries that contribute to the cryptocurrency economy. Such service providers include currency exchanges, escrow services, digital wallets, mining pools, market data suppliers, and investment funds. As the prices of cryptocurrencies and their usage have surged, these intermediaries have become targets for cybercriminals and can be a major source of risk for naive individual and corporate actors. This essay will focus on the rise and fall of Mt. Gox, explain the IT matters that arose in the 2011 and 2014 hacks, and outline some risks of cryptocurrency exchanges in general.
Jed McCaleb, an American software developer, started the mtgox.com web site in 2007 with the original purpose of trading playing cards for the popular fantasy game Magic: The Gathering (in fact Mt. Gox stands for “Magic The Gathering Online eXchange). McCaleb was bitten by the bitcoin bug in 2010, and by July, the site’s order matching system for playing cards morphed into a bitcoin exchange where buyers and sellers could trade the digital currency as well as manage balances, deposits, and withdrawals. By early 2011, the site had become successful with aggregate daily transactions exceeding tens of thousands of dollars but it took up all of McCaleb’s time and was already a hacker target; needing help fast, he sold the site in March to Mark Karpeles, a French software developer for nothing upfront sharing 50% of profits for the first six months after its sale and retaining a 12% ownership stake in the firm. The exceptionally favorable terms to Karpeles should have been a red flag to him as well as the assumption of existing debts of 80,000 missing BTC, and extraordinary statements in the purchase agreement that “the Seller is uncertain if mtgox.com is compliant or not with any applicable US code or state, or law of any country… Buyer agrees to indemnify Seller against any legal action that is taken against Buyer or Seller with regards to mtgox.com or anything acquired under this agreement.”
Months after Karpeles took over, the troubles began in earnest. On 13 June 2011, Mt. Gox reported that 478 accounts were robbed of 25,000 BTC worth about $400,000 USD at the time. Then on Friday June 17, a message offering the Mt. Gox user database for sale was published to pastebin, a plaintext content hosting service used for software code review by Internet Relay Chat (IRC) members and hackers. Signed ~cRazIeStinGeR~ and tied to [email protected], the pastebin message was troubling but the company ignored the warning and did nothing. On Saturday June 18, Karpeles then reported some additional cases of theft. On Sunday, June 19 17:15:36 UTC, suspicious trading activity suddenly started on Mt. Gox. Someone had placed one or more orders to sell hundreds of thousands of BTC after which the exchange rate crashed from $17 USD to $0.01 USD. The largest trade observed for 261,383.7630 BTC was executed at 0.01 USD at 17:51:16; it sponged most of the exchange’s order book and constituted 4% of the 6.5 million BTC in circulation at the time. Later around 18:00 UTC as the news started to travel, the Mt. Gox site and other BTC exchanges experienced intense volatility with the exchange rate swinging wildly between $1 and $20. By this time, Karpeles was woken up, investigated the matter, determined that an attacker had compromised McCaleb’s old administrative account used for auditing, and shut the Mt. Gox site down. Later that same day around 19:15 UTC, someone published the complete list of Mt. Gox usernames, email addresses, and password hashes on an Internet forum. The list contained 61,016 accounts with the equivalent of more than $8.75M USD. Most of the account passwords were hashed with the UNIX MD5-based crypt() utility, but hundreds of the passwords were plain MD5 hashes (unsalted, non-iterated) that could be brute-forced easily. More on this later. Several other BTC exchanges also began voluntary shutdowns as a security precaution since many users used multiple exchanges for trading. At 21:00 UTC, Mt. Gox began disclosing the attack to its users, recommending that users who used the same password for Mt. Gox and other electronic services (e.g. email, banking, etc) should change it, warning users of possible email phishing attacks, and promising to reverse and refund the fraudulent trades. At 21:10 UTC, Mt Gox also confirmed that only 2000 BTC were stolen (valued $30K USD at the time). On June 21, Mt. Gox allowed users to submit requests to recover their accounts using their account name, email address, old password, new password, verification of the email address and optionally providing more evidence such as the last known Mt. Gox balance, copy of government ID, etc. These claim requests were verified manually by the Mt. Gox staff. On June 23, Karpeles executed a large transfer of 424242.42424242 BTC from cold storage to the exchange on Block 132749 as a confidence building measure to prove that the bitcoins were still under Mt. Gox’s control. On June 26, Mt. Gox re-opened for trading one week after it had been shut down. Fraudulent trades were indeed rolled back at the company’s expense. They introduced SHA-512 multi-iteration salted hashes of passwords as well as user verification upon first-time login using the last IP address that accessed the account, verifying the email address, account name, and old password. Users were then prompted to enter a new strong password. The exchange rate stabilized around $16.50 USD within hours and there were neither sell-offs nor mass withdrawals. All was well, right?
As bitcoin’s price continued its climb and Mt. Gox grew to handle over 70% of BTC transactions in 2013, Karpeles became a major figure in the bitcoin world, holding interviews with major news organizations, endowing the Bitcoin Foundation where he was a board member (he has since resigned), and expanding the company headquarters in Tokyo’s Shibuya neighborhood. Beneath this surface of success, the company had serious business and technology problems. First, according to former employees interviewed by Wired.com, Mt Gox did not use a software version control system which meant one engineer could accidentally modify a colleague’s code and there was neither a history of changes nor a reliable mechanism for merging or reverting to a known working copy. Second, Karpeles was the only one who could approve changes to the site’s source code; which meant that bug fixes — even security related items — could be delayed for days, sometimes weeks. Third, it only introduced a test environment for quality assurance in the spring of 2013 (years after Karpeles had taken over and the company was already a primary bitcoin hub). Fourth, Mt. Gox had no accounting system or process for reconciling the offline BTC balance for inventory, the online BTC balance for liquidity, and the cash balance for currency exchange; the need for accuracy and transparency was complicated by Karpeles paranoid insistence on keeping most of the bitcoins in offline, cold storage that would be more secure from hackers. Fifth, Mt. Gox was sued by Coinlab for $75M on May 2 for allegedly violating a contract signed in February to transfer North American customers to Coinlabs which never actually occurred. Sixth, the company had somehow not registered for a license with the US government as a money transmitter, and on May 15, the US Department of Homeland Security issued a warrant to seize money from Mt. Gox’s subsidiary account with payment processor Dwolla. Between May and July, DHS agents seized $5M USD from the subsidiary. On June 29, Mt. Gox finally received its money services business (MSB) license from the US Financial Crimes Enforcement Network (FinCEN). In summary, no bank or financial services company insisting on the highest ethical, professional standards would operate in such a slipshod manner in a post-9/11, post-Sarbanes-Oxley, and post-2008 crisis world.
On August 5 2013, Mt. Gox announced that it had incurred significant losses due to crediting deposits which had not fully cleared, and that new deposits would no longer be credited until the funds transfer was fully completed. By November, customers were experiencing delays of weeks to months in withdrawing cash from their accounts. Mt. Gox had also lost its place as the top exchange, dropping to third. On 7 February 2014, Mt. Gox stopped all BTC withdrawals and explained that it was “to obtain a clear technical view of the currency processes.” On February 10, Mt. Gox then divulged that the root cause may have been transaction malleability and possible double spending and that it was working with the bitcoin core development team to mitigate the problem. A CoinDesk poll of 3,000 Mt. Gox customers at the time suggested that 68% of polled customers were still awaiting their funds from Mt. Gox and that the median waiting time was now between one to three months. On February 20, protests outside the Mt. Gox office continued, and citing security concerns, the company moved its offices to a different location in Tokyo. On February 24, Mt. Gox suspended all trading and its website went offline. Six other major bitcoin exchanges issued a joint statement distancing themselves from Mt. Gox. On February 28, Mt. Gox filed in Tokyo for a form of bankruptcy protection from creditors, declaring that it had liabilities of about 6.5 billion yen ($65 million USD) and 3.8 billion yen in assets. The company also said that it had lost 744,408 BTC belonging to customers and around 100,000 of its own, totaling around 7% of all 12 million BTC in circulation and worth around $470 million USD at the time. Mt. Gox issued another statement saying it believed the bitcoins were likely stolen, blamed hackers, and began a search for the missing bitcoins. Two weeks later, Mt. Gox filed for bankruptcy in the USA. On March 20, Mt. Gox reported that it had discovered 200,000 BTC in an old digital wallet used prior to June 2011. Karpeles was finally arrested in August 2015 by Japanese police and charged with fraud, embezzlement, and stealing money for personal use; he remained in jail until 2016 before being granted bail. Case closed, right? Not yet. In July 2017, a Russian national named Alexander Vinnik was arrested by US authorities in Greece and charged with playing a key role in the laundering of bitcoins stolen from Mt. Gox. Furthermore, Vinnik was charged by Greek authorities for laundering approximately $4 billion in bitcoin. The US FBI also raided BTC-e, a rival exchange that Vinnik was alleged to be associated with, as part of the investigation.
There are a multitude of lessons that can be learned from this case study that are useful to IT professionals as well as the individual and corporate actors interested in cryptocurrencies.
McCaleb was never charged with wrongdoing in the Mt. Gox saga; he went on to launch another successful cryptocurrency called Ripple and also started a company called Lightyear.io based on the Stellar payment system to allow cross-border monetary transactions between fiat and digital currencies. Karpeles is now out of jail, but still remains on trial in Tokyo. According to an April 2018 interview with Bloomberg news, he has lost his faith in bitcoin, but due to quirks in Japanese financial laws may still recover money from the liquidation of Mt. Gox since the trustee had agreed to sell the BTC on hand to make all creditors whole. Note even the trustee’s sale of BTC has its darker side. It is widely known that exchanges have a close circle of large investors or “whales” they cater to. The whales get special treatment and private information that regular traders do not which is the textbook definition of insider trading. Since bitcoin is unregulated, the public has no way guarantee that this did not happen. Once the initial orders from the Mt. Gox trustee were being made, the exchanges could have easily leaked the information to the inner circle and allowed them to trade before others. As for Vinnik, Greek law enforcement received intelligence on plans for his assassination via poisoning with the help of criminals. He remains in jail, but was transferred to a maximum security area,where he has reduced his contact with other inmates and was forbidden from receiving anything, including food, or water from any people he did not know.
I will continue to write further about Bitcoin, incorporating analysis and research from the financial markets, computer security, and IT best practices. In the meantime, watch your bitcoin wallet.