The 7th anniversary of the first major hack of offers an opportunity to reflect upon software defects, human error, process flaws, and the best principles and practices for solution delivery in the IT industry. In this blog and my upcoming book, , I will chronicle some important software system failures in the past and discuss ideas for improving the future of system quality. As information technology becomes increasingly woven into Life, the quality of software impacts our commerce, health, infrastructure, military, politics, science, security, and transportation. The Big Idea is that we have no choice but to get better at delivering technology solutions because our lives depend it. Mt. Gox Bugs: A Short History of Software Imperfection On 19 June 2011, a computer hacker breached the security of Mt. Gox, the world’s largest bitcoin (BTC) exchange at the time. Suspicious trading activity followed soon after the breach, triggering a flash crash of bitcoin’s price from $17 USD to $0.01 USD on the exchange, during which the attacker withdrew a small amount (approximately 2000 BTC), and also leaked the exchange’s user database consisting of usernames and hashed passwords. Mt. Gox eventually collapsed around 7 February 2014 under the weight of processing delays and errors, further security incidents and possible insider fraud. Although the Bitcoin system and its community embrace cryptography and decentralization in theory, in practice, there is an extensive ecosystem of third-party intermediaries that contribute to the cryptocurrency economy. Such service providers include currency exchanges, escrow services, digital wallets, mining pools, market data suppliers, and investment funds. As the prices of cryptocurrencies and their usage have surged, these intermediaries have become targets for cybercriminals and can be a major source of risk for naive individual and corporate actors. This essay will focus on the rise and fall of Mt. Gox, explain the IT matters that arose in the 2011 and 2014 hacks, and outline some risks of cryptocurrency exchanges in general. Jed McCaleb, an American software developer, started the mtgox.com web site in 2007 with the original purpose of trading playing cards for the popular fantasy game Magic: The Gathering (in fact Mt. Gox stands for “Magic The Gathering Online eXchange). McCaleb was bitten by the bitcoin bug in 2010, and by July, the site’s order matching system for playing cards morphed into a bitcoin exchange where buyers and sellers could trade the digital currency as well as manage balances, deposits, and withdrawals. By early 2011, the site had become successful with aggregate daily transactions exceeding tens of thousands of dollars but it took up all of McCaleb’s time and was already a hacker target; needing help fast, he sold the site in March to Mark Karpeles, a French software developer for nothing upfront sharing 50% of profits for the first six months after its sale and retaining a 12% ownership stake in the firm. The exceptionally favorable terms to Karpeles should have been a red flag to him as well as the assumption of existing debts of 80,000 missing BTC, and extraordinary statements in the purchase agreement that “ .” the Seller is uncertain if mtgox.com is compliant or not with any applicable US code or state, or law of any country… Buyer agrees to indemnify Seller against any legal action that is taken against Buyer or Seller with regards to mtgox.com or anything acquired under this agreement Months after Karpeles took over, the troubles began in earnest. On 13 June 2011, Mt. Gox reported that 478 accounts were robbed of 25,000 BTC worth about $400,000 USD at the time. Then on Friday June 17, a message offering the Mt. Gox user database for sale was published to pastebin, a plaintext content hosting service used for software code review by Internet Relay Chat (IRC) members and hackers. Signed ~cRazIeStinGeR~ and tied to auto36299386@hushmail.com, the pastebin message was troubling but the company ignored the warning and did nothing. On Saturday June 18, Karpeles then reported some additional cases of theft. On Sunday, June 19 17:15:36 UTC, suspicious trading activity suddenly started on Mt. Gox. Someone had placed one or more orders to sell hundreds of thousands of BTC after which the exchange rate crashed from $17 USD to $0.01 USD. The largest trade observed for 261,383.7630 BTC was executed at 0.01 USD at 17:51:16; it sponged most of the exchange’s order book and constituted 4% of the 6.5 million BTC in circulation at the time. Later around 18:00 UTC as the news started to travel, the Mt. Gox site and other BTC exchanges experienced intense volatility with the exchange rate swinging wildly between $1 and $20. By this time, Karpeles was woken up, investigated the matter, determined that an attacker had compromised McCaleb’s old administrative account used for auditing, and shut the Mt. Gox site down. Later that same day around 19:15 UTC, someone published the complete list of Mt. Gox usernames, email addresses, and password hashes on an Internet forum. The list contained 61,016 accounts with the equivalent of more than $8.75M USD. Most of the account passwords were hashed with the UNIX MD5-based crypt() utility, but hundreds of the passwords were plain MD5 hashes (unsalted, non-iterated) that could be brute-forced easily. More on this later. Several other BTC exchanges also began voluntary shutdowns as a security precaution since many users used multiple exchanges for trading. At 21:00 UTC, Mt. Gox began disclosing the attack to its users, recommending that users who used the same password for Mt. Gox and other electronic services (e.g. email, banking, etc) should change it, warning users of possible email phishing attacks, and promising to reverse and refund the fraudulent trades. At 21:10 UTC, Mt Gox also confirmed that only 2000 BTC were stolen (valued $30K USD at the time). On June 21, Mt. Gox allowed users to submit requests to recover their accounts using their account name, email address, old password, new password, verification of the email address and optionally providing more evidence such as the last known Mt. Gox balance, copy of government ID, etc. These claim requests were verified manually by the Mt. Gox staff. On June 23, Karpeles executed a large transfer of 424242.42424242 BTC from cold storage to the exchange on Block 132749 as a confidence building measure to prove that the bitcoins were still under Mt. Gox’s control. On June 26, Mt. Gox re-opened for trading one week after it had been shut down. Fraudulent trades were indeed rolled back at the company’s expense. They introduced SHA-512 multi-iteration salted hashes of passwords as well as user verification upon first-time login using the last IP address that accessed the account, verifying the email address, account name, and old password. Users were then prompted to enter a new strong password. The exchange rate stabilized around $16.50 USD within hours and there were neither sell-offs nor mass withdrawals. All was well, right? As bitcoin’s price continued its climb and Mt. Gox grew to handle over 70% of BTC transactions in 2013, Karpeles became a major figure in the bitcoin world, holding interviews with major news organizations, endowing the Bitcoin Foundation where he was a board member (he has since resigned), and expanding the company headquarters in Tokyo’s Shibuya neighborhood. Beneath this surface of success, the company had serious business and technology problems. First, according to former employees interviewed by Wired.com, Mt Gox did not use a software version control system which meant one engineer could accidentally modify a colleague’s code and there was neither a history of changes nor a reliable mechanism for merging or reverting to a known working copy. Second, Karpeles was the only one who could approve changes to the site’s source code; which meant that bug fixes — even security related items — could be delayed for days, sometimes weeks. Third, it only introduced a test environment for quality assurance in the spring of 2013 (years after Karpeles had taken over and the company was already a primary bitcoin hub). Fourth, Mt. Gox had no accounting system or process for reconciling the offline BTC balance for inventory, the online BTC balance for liquidity, and the cash balance for currency exchange; the need for accuracy and transparency was complicated by Karpeles paranoid insistence on keeping most of the bitcoins in offline, cold storage that would be more secure from hackers. Fifth, Mt. Gox was sued by Coinlab for $75M on May 2 for allegedly violating a contract signed in February to transfer North American customers to Coinlabs which never actually occurred. Sixth, the company had somehow not registered for a license with the US government as a money transmitter, and on May 15, the US Department of Homeland Security issued a warrant to seize money from Mt. Gox’s subsidiary account with payment processor Dwolla. Between May and July, DHS agents seized $5M USD from the subsidiary. On June 29, Mt. Gox finally received its money services business (MSB) license from the US Financial Crimes Enforcement Network (FinCEN). In summary, no bank or financial services company insisting on the highest ethical, professional standards would operate in such a slipshod manner in a post-9/11, post-Sarbanes-Oxley, and post-2008 crisis world. On August 5 2013, Mt. Gox announced that it had incurred significant losses due to crediting deposits which had not fully cleared, and that new deposits would no longer be credited until the funds transfer was fully completed. By November, customers were experiencing delays of weeks to months in withdrawing cash from their accounts. Mt. Gox had also lost its place as the top exchange, dropping to third. On 7 February 2014, Mt. Gox stopped all BTC withdrawals and explained that it was “to obtain a clear technical view of the currency processes.” On February 10, Mt. Gox then divulged that the root cause may have been transaction malleability and possible double spending and that it was working with the bitcoin core development team to mitigate the problem. A CoinDesk poll of 3,000 Mt. Gox customers at the time suggested that 68% of polled customers were still awaiting their funds from Mt. Gox and that the median waiting time was now between one to three months. On February 20, protests outside the Mt. Gox office continued, and citing security concerns, the company moved its offices to a different location in Tokyo. On February 24, Mt. Gox suspended all trading and its website went offline. Six other major bitcoin exchanges issued a joint statement distancing themselves from Mt. Gox. On February 28, Mt. Gox filed in Tokyo for a form of bankruptcy protection from creditors, declaring that it had liabilities of about 6.5 billion yen ($65 million USD) and 3.8 billion yen in assets. The company also said that it had lost 744,408 BTC belonging to customers and around 100,000 of its own, totaling around 7% of all 12 million BTC in circulation and worth around $470 million USD at the time. Mt. Gox issued another statement saying it believed the bitcoins were likely stolen, blamed hackers, and began a search for the missing bitcoins. Two weeks later, Mt. Gox filed for bankruptcy in the USA. On March 20, Mt. Gox reported that it had discovered 200,000 BTC in an old digital wallet used prior to June 2011. Karpeles was finally arrested in August 2015 by Japanese police and charged with fraud, embezzlement, and stealing money for personal use; he remained in jail until 2016 before being granted bail. Case closed, right? Not yet. In July 2017, a Russian national named Alexander Vinnik was arrested by US authorities in Greece and charged with playing a key role in the laundering of bitcoins stolen from Mt. Gox. Furthermore, Vinnik was charged by Greek authorities for laundering approximately $4 billion in bitcoin. The US FBI also raided BTC-e, a rival exchange that Vinnik was alleged to be associated with, as part of the investigation. There are a multitude of lessons that can be learned from this case study that are useful to IT professionals as well as the individual and corporate actors interested in cryptocurrencies. The and of an organization are its foundation and most important success factors. While McCaleb and Karpeles may have possessed individual engineering talents and entrepreneurial instincts, they knew little about running a high-performance software development team, let alone a sophisticated financial services organization. People, Leadership, Values Mt. Gox managed to violate nearly every important computer security principle, and I refer the interested reader to the excellent book, Building Secure Software by Gary McGraw and John Viega. . The June 2011 hack was made possible because McCaleb’s old administrative account was kept active to facilitate his ability to audit and verify exchange commissions. There are conflicting reports as to whether a SQL injection or cross-site forgery attack was the root cause for the account’s compromise. In any case, both vulnerabilities are well-known, and the site’s software should have been audited by an engineering group independent of the development team. Moreover, the account had too many privileges across Mt. Gox; it was used to login remotely, dump the user database, and it consequently allowed the attacker to perform the flash crash in an attempt to circumvent the daily withdrawal limit of $1000. The remote login account should have been read-only at most and provisioned without the authority to extract the user database. That would have prevented the ensuing damage. . Once the user database was copied, the attacker only had to decrypt some of the unsalted MD5 hashed passwords to generate sufficient trading volume to crash the BTC/USD exchange rate. MD5 is a 1-way hash function producing a 128-bit value and was designed by Ronald Rivest in 1991; security flaws have been found by other researchers that make it vulnerable to collisions, brute force attacks, and extension length attacks. The community consensus in both industry and government by 2010 was that MD5 was insecure and useful only for checksums or message digests; Mt. Gox should have been using SHA-2 which was first published in 2001 by the NSA and mainstream by 2011, already implemented in SSL, PGP, SSH, and IPSec. Furthermore, user login should have used two-factor authentication (the user-chosen password as well as a one-time PIN sent by email or text). . Subsequent investigations of the 2014 hack by Nilsson of WizSec, Decker and Wattenhofer of the university of ETH Zurich, and other bitcoin security researchers suggests that most of the lost bitcoins were due to the theft of Mt. Gox’s hot wallet private keys in September 2011 and . The unencrypted wallet.dat file gave the attacker access to a large number of BTC owned by the company, but more importantly its private keys. Asymmetric cryptography is a core building block of Bitcoin’s design; the private keys allow users to spend authorized transactions by generating unique signatures mathematically possible only by the key owner. In effect, the private keys in the hands of a skillful, patient attacker could be used to slowly transfer BTC to other secure addresses and without proper reconciliation as was the case at Mt. Gox, could be interpreted as part of normal business processes of moving BTC to cold storage. WizSec identified Vinnik as the owner of the wallets into which many stolen bitcoins had been transferred. Key management becomes incredibly important in the Bitcoin system, and Mt. Gox could have prevented the theft of the hot wallet by following several best practices: minimize the size and number of keys in the hot wallet, move the rest into cold storage and savings, backup the entire hot wallet, encrypt the online backup, use many secure locations, and make regular backups. The Security flaws at Mt. Gox were not in Bitcoin’s design itself, but in the company’s accounting system and its naive approach to database, network, and user security. First, Mt. Gox did not secure the weakest link, granted too many privileges to get work done, and did not compartmentalize its system security Second, Mt. Gox’s system defenses were shallow Third, they forgot that hiding secrets is hard not by transaction malleability as alleged by Mt. Gox and matters whether its reasoning about bitcoin security or financial accounting systems. One cannot reasonably expect twenty-something amateurs to design a secure accounting and trading system based on the original commodity of playing cards for nerds. This part of the story would be absurdly funny as a hypothetical, if it was not sadly true. Complexity Design matter to customers, especially during a crisis. While Mt. Gox notified customers within hours of the 2011 hack, it took weeks during the 2014 incident. The bottom line is that if you are breached, then you should be prompt, clear, and transparent in your response. Skip the excuses and blaming others; they are a waste of time. Communications and are fundamental principles and practices of high-performance software development organizations. Mt. Gox should have used version control, setup separate test environments from production used by a dedicated QA team, institutionalized change control through a committee-based review rather than its process by personality, and conducted penetration tests into its security profile. Quality Devops abound for the individual and corporate actors interested in digital cryptocurrencies and especially those using exchanges which operate as a de-facto centralized authority — again a remarkable irony that many ignore at their peril. Research into bitcoin exchange security by Moore and Christin shows that of the 80 exchanges operational through the period 2010–2015, nearly half (38) closed and 26 experienced security breaches. Besides the closure and security risks, there are others related to legal and regulatory action by governments not to mention market volatility and the irrevocability of transactions. Risks McCaleb was never charged with wrongdoing in the Mt. Gox saga; he went on to launch another successful cryptocurrency called Ripple and also started a company called Lightyear.io based on the Stellar payment system to allow cross-border monetary transactions between fiat and digital currencies. Karpeles is now out of jail, but still remains on trial in Tokyo. According to an April 2018 interview with Bloomberg news, he has lost his faith in bitcoin, but due to quirks in Japanese financial laws may still recover money from the liquidation of Mt. Gox since the trustee had agreed to sell the BTC on hand to make all creditors whole. Note even the trustee’s sale of BTC has its darker side. It is widely known that exchanges have a close circle of large investors or “whales” they cater to. The whales get special treatment and private information that regular traders do not which is the textbook definition of insider trading. Since bitcoin is unregulated, the public has no way guarantee that this did not happen. Once the initial orders from the Mt. Gox trustee were being made, the exchanges could have easily leaked the information to the inner circle and allowed them to trade before others. As for Vinnik, Greek law enforcement received intelligence on plans for his assassination via poisoning with the help of criminals. He remains in jail, but was transferred to a maximum security area,where he has reduced his contact with other inmates and was forbidden from receiving anything, including food, or water from any people he did not know. I will continue to write further about Bitcoin, incorporating analysis and research from the financial markets, computer security, and IT best practices. In the meantime, watch your bitcoin wallet. References https://web.archive.org/web/20110919162635/https://mtgox.com/press_release_20110630.html http://blog.zorinaq.com/major-attack-on-the-worlds-largest-bitcoin-exchange/ http://hackingdistributed.com/2014/03/01/what-did-not-happen-at-mtgox/ https://www.wired.com/2014/03/bitcoin-exchange/ https://blockonomi.com/mt-gox-hack/ https://arxiv.org/pdf/1706.00916.pdf http://www.nemode.ac.uk/wp-content/uploads/2015/02/2015-Bit-Coin-risk-analysis.pdf https://tylermoore.utulsa.edu/toit17.pdf https://blog.acolyer.org/2018/02/15/a-survey-on-security-and-privacy-issues-of-bitcoin/ https://spectrum.ieee.org/tech-talk/computing/networks/the-mt-gox-bitcoin-debacle-an-update https://blog.wizsec.jp/2017/07/breaking-open-mtgox-1.html https://www.tik.ee.ethz.ch/file/7e4a7f3f2991784786037285f4876f5c/malleability.pdf https://coinmarketcap.com/ https://www.bis.org/publ/arpdf/ar2018e5.htm https://www.bloomberg.com/news/articles/2018-04-05/former-mt-gox-ceo-karpeles-is-no-longer-a-bitcoin-believer https://blockchain.info/charts https://www.bbc.co.uk/programmes/b09tdyjz http://fortune.com/longform/bitcoin-mt-gox-hack-karpeles/ https://sputniknews.com/europe/201805101064329910-greece-police-russian-national-plan-kill/ https://bitcoin.org/en/secure-your-wallet#everyday
