When You “Git” in Trouble: a Version Control Story

Thank you for laughing at my extremely funny title. But do you know what’s not funny? When you push a commit to your git repo, and you see this in : GitHub Desktop Yes, I know that the use and that the cool people just use the . We’re really cool people, so we’re going to use the command line to solve this problem. In fact, we have no choice — and that’s the adventure you’re joining me on in this article: fixing a git repo that suddenly becomes damaged through absolutely and despite having whatsoever. But at least you have a visualization of my panic. cool people Git Tower really command line no fault of your own no command line git expertise Step one is diagnosing the problem. If you’re like me, however, you also have a , and that comes from relying on a tool you barely understand, which makes difficult. Finding sympathetic experts to help you resolve your specific case is also a challenge. You might not even know who to ask or what the questions should be. Nevertheless, you can learn how things work and how to resolve your own dilemmas piecemeal—if you’re patient, systematic, and willing to . And that, friends, entailed for me a trip to the , where I discovered the command, which I dutifully ran in my repo’s root directory, and which resulted in the following (truncated) output: meta problem diagnosis learn git reference documentation git-fsck > git fsck...error: object file .git/objects/67/99ddac675cab54060cdfb066dbfadb6708fc3f is emptyerror: object file .git/objects/67/99ddac675cab54060cdfb066dbfadb6708fc3f is emptyfatal: loose object 6799ddac675cab54060cdfb066dbfadb6708fc3f (stored in .git/objects/67/99ddac675cab54060cdfb066dbfadb6708fc3f) is corrupt So what we have here, and what you probably have if one of your projects comes down with a case of the , is a . Oh. Good to know. Now what? When you want to know the meaning of life, you ask . In this case, I consulted an , which happens to address a similar situation. blinkies corrupted repository God ancient email from Linus Torvalds A git repo is really a of various kinds of : blobs, trees, and commits. Blob objects are , well, blobs of your data, each representing one of your files. These blob objects are independent of one another, but they are also linked by so-called tree objects, which effectively group the blobs into an arrangement that’s analogous to a file system’s directory structure. Finally, there are commit objects, which contain the information necessary to track the changes in your trees and blobs. The commit objects are also linked, sequentially (as you might expect). graph binary objects cryptographically-hashed Git stores all of these objects in a series of nested directories, located in , according to the first few characters of their commit ids (as you can see above). For example, the object is stored in a directory called as the file ; that is, the full object name is a combination of the directory it's stored in and a specific file in that directory. .git/objects/ 6799ddac675cab54060cdfb066dbfadb6708fc3f 67/ 99ddac675cab54060cdfb066dbfadb6708fc3f So, if one of the commit objects becomes corrupted, your whole repo may turn into a , because the chain of linked commits will have been broken. That’s the bad news. The good news is that for the very reason your repo is a collection of discrete files, you may be able to restore it to health, even if one of the objects is corrupted beyond repair — if you can perform a . useless pile of bytes precise enough surgery That’s what I tried to do. Following Linus’s advice, I moved the corrupted commit object file somewhere else. You can park your damaged objects . They will probably end up in the trash, anyway. ./git/objects/67/99ddac675cab54060cdfb066dbfadb6708fc3f wherever you like I happened to get the same error message for blob object , so I moved that one too. Then I tried the file system check again: 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf > git fsckChecking object directories: 100% (256/256), done.Checking objects: 100% (8970/8970), done.broken link from tree 03a88f876eb3f6157f76461a3ae6cb18bbb86561 to blob 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcfdangling commit 76814e15074b540bc2f7e78daf3f5175a8759523missing commit 6799ddac675cab54060cdfb066dbfadb6708fc3fmissing blob 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcfdangling blob 2a60520000698ad964e4e61fab31f9b862763550dangling commit 41634cd81964068acb153bfa355d63bd80fc7cefdangling commit 5bf415e2bdbc47822ae99b64c2a0f6b4f288eefb Note that Linus suggests using , but this is the default behavior now. git fsck --full Ignoring the “dangling commit” messages, the “broken link” message tells me which tree object points to the blob object I just removed. In effect, I broke the link on purpose to reveal this information. Tree object expects to point to blob , but the blob isn't there. Commit object , the other one I moved, is also reported missing. So far, so good. 03a88f876eb3f6157f76461a3ae6cb18bbb86561 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf 6799ddac675cab54060cdfb066dbfadb6708fc3f Continuing with Linus’s advice, I now had enough information to use the command to list the contents of the tree object called out above: git-ls-tree > git ls-tree 03a88f876eb3f6157f76461a3ae6cb18bbb86561100644 blob 312d8994f1005a9563a9410c592b27000c201101 building-test.js100644 blob f84006fd14c6d4b2ccc3ef22b2fe02abf535bd1a folds-test.js100644 blob 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf index.js100644 blob 8b1f47bce7ec989dff7e936279d63d1d02f6a92d indexing-test.js100644 blob 3f2b45f8cd9dfd486c8e821ee672ed66a34768df inf-test.js100644 blob 0e6d1985aa59d17e2115bc6c7936d2ac88b00457 list-test.js100644 blob 40310b5df53691d0e1ba4118c0e3ab66ed766990 reducing-test.js100644 blob 8244d5fb2768ad5c7c33890ee26c797c2df6262b searching-test.js100644 blob ff34a2f15faf6eec7d9c9635e79d2a0abdadfb42 sub-test.js100644 blob ac0c029cc64870c7445c4bdd9d7fe20646b5cc33 trans-test.js100644 blob 99781d303e90b7aa4de8d630c1053a42f87e8331 zip-test.js Scanning the list, I found the culprit blob and its associated file: . So now I knew the source of the problem, but I didn't know of that file created the problem to begin with. Back to the command line: 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf index.js which version > git log --raw --allcommit cf63a71497e027d96614cfff6ba1d297f1a1a26eAuthor: Steven Syrek Date: Mon Jul 18 11:55:40 2016 -0400 Add tests for set operations on lists:100644 100644 67a45ac... c1c2f99... M test/list/index.js:000000 100644 0000000... 23c47fe... A test/list/set-test.jscommit f3bc2c55b22deb889f99cdd45663c20a8e8e79c1Author: Steven Syrek Date: Mon Jul 18 11:14:13 2016 -0400 Add tests for list zipping and unzipping functions and remove exponentiation operator from tests and examples:100644 100644 01af47b... 3b1bf35... M source/list/zip.js:100644 100644 21206e2... 67a45ac... M test/list/index.js:000000 100644 0000000... 99781d3... A test/list/zip-test.js The command, with the and options, will show the entire commit history of a repo. I only show the relevant parts from mine above. What we can see here is that object precedes the corrupted object , which is in turn followed in the subsequent commit (they are listed in reverse order) by object . git-log --raw --all 21206e20386e0365bc6f15d0ccd372b1c72b5667 67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf c1c2f99072ef41aca89e963cfb0143f897e0de78 At this point, Linus says that I’m done, because I discovered which versions of the file preceded and followed the corrupted commit: If you can do that, you can now recreate the missing object with and your repository is good again! git hash-object -w Unfortunately, after I tried this, my repository was not good again. Now things . Past Linus was out of advice, and present Linus (now also past Linus) probably had than help me. I was therefore left to follow the astute troubleshooting process that use every day: started to get hairy better things to do professional developers Google. Google. Stack Overflow. omg it’s down This throw-everything-at-the-wall approach led me to try a few things, starting with the command. If I couldn’t automatically re-create the missing object, I reasoned desperately, perhaps I could do it manually: git-diff > git diff 206e20386e0365bc6f15d0ccd372b1c72b5667..c2f99072ef41aca89e963cfb0143f897e0de78fatal: ambiguous argument '206e20386e0365bc6f15d0ccd372b1c72b5667..c2f99072ef41aca89e963cfb0143f897e0de78': unknown revision or path not in the working tree. Oops. I forgot the leading characters: > git diff 21206e20386e0365bc6f15d0ccd372b1c72b5667..c1c2f99072ef41aca89e963cfb0143f897e0de78diff --git a/21206e20386e0365bc6f15d0ccd372b1c72b5667..c1c2f99072ef41aca89e963cfb0143f897e0de78 b/c1c2f99072ef41aca89e963cfb0143f897e0de78index 21206e2..c1c2f99 100644--- a/21206e20386e0365bc6f15d0ccd372b1c72b5667..c1c2f99072ef41aca89e963cfb0143f897e0de78+++ b/c1c2f99072ef41aca89e963cfb0143f897e0de78@@ -25,3 +25,7 @@ export * from './sub-test'; export * from './searching-test'; export * from './indexing-test';++export * from './zip-test';++export * from './set-test'; Above are the lines in that changed between the two commits on either side of the corrupted commit. They are marked with a , with a few surrounding lines also shown for context. Deleted lines, if there had been any, would have been marked with a . Since two identical files, when , should produce identical hash keys, I thought I'd try to brute force a solution by deleting the changed lines and re-creating the commit by hand: index.js + - hashed > git hash-object -w ./test/list/index.js2a60520000698ad964e4e61fab31f9b862763550 Nope. Try again, maybe just deleting the lines marked . + > git hash-object -w ./test/list/index.js21206e20386e0365bc6f15d0ccd372b1c72b5667 Nope, but interesting. I managed to recreate the original state of the object before the corrupted commit happened, but I guess what I was really trying to do was re-create the correct intermediate state? There weren’t too many possibilities, fortunately, since I had uncharacteristically been going through a phase. So I changed the file once more, adding to it only those lines marked with a that I recalled adding before everything went bits up: good git hygiene + git hash-object -w ./test/list/index.js67a45ac2f58a444fa4db11cd9ab7e024a8e35dcf Yay. > git fsckChecking object directories: 100% (256/256), done.Checking objects: 100% (8970/8970), done.dangling commit 76814e15074b540bc2f7e78daf3f5175a8759523missing commit 6799ddac675cab54060cdfb066dbfadb6708fc3fdangling blob 2a60520000698ad964e4e61fab31f9b862763550dangling commit 41634cd81964068acb153bfa355d63bd80fc7cefdangling commit 5bf415e2bdbc47822ae99b64c2a0f6b4f288eefb Oh right, I have a healthy blob now, but I’m still missing the commit object that points to it. Now what? It’s to the rescue! git-gc > git gcerror: Could not read 6799ddac675cab54060cdfb066dbfadb6708fc3ferror: Could not read 6799ddac675cab54060cdfb066dbfadb6708fc3fwarning: reflog of 'HEAD' references pruned commitswarning: reflog of 'refs/heads/restructure' references pruned commitserror: Could not read 6799ddac675cab54060cdfb066dbfadb6708fc3ffatal: Failed to traverse parents of commit b267a6a8264c0cdc72d047049610fc91e9f7c06ferror: failed to run repack Or not. That was supposed to all the… garbage. And fix… all the things. I don’t know why I thought that. But I hoped. I really, truly hoped. And then I . Noting the “reflog” message above, I was on to my next brilliant idea: garbage collect imprecated > git reflog expire --all --stale-fixerror: Could not read 6799ddac675cab54060cdfb066dbfadb6708fc3ffatal: Failed to traverse parents of commit b267a6a8264c0cdc72d047049610fc91e9f7c06f As everyone knows, when using , the more options you add, the more masculine you are. . Real men don’t read pages: they just . Plus, I rather liked the idea of re-flogging my repo. But no. That didn’t work, either. command line tools It doesn’t matter if you don’t know what they do man move fast and break things I had by now of trying absolutely anything, without regard for or . I turned back to the logs, which always feels one step shy of admitting defeat and finding a corner in which to quietly weep. But perhaps a solution would miraculously present itself, something I missed before but was there the whole time for all to see? waded well into the waters sense soundness > git log 6799ddac675cab54060cdfb066dbfadb6708fc3ffatal: bad object 6799ddac675cab54060cdfb066dbfadb6708fc3f Nope. > git ls-tree 6799ddac675cab54060cdfb066dbfadb6708fc3ffatal: not a tree object Nope. I mean, duh. Somehow, I then had the bright idea of examining the logs for just the branch of my repo, which is the one I had been working on when the fatal blinking cursor entered my life: restructure > tail -n 40 .git/logs/refs/heads/restructure...44dc22e706fb029a9c96f3bd125755fd55ac882b 6799ddac675cab54060cdfb066dbfadb6708fc3f Steven Syrek 1468788351 -0400 commit: Replace isEq function in all tests with should.eql6799ddac675cab54060cdfb066dbfadb6708fc3f b267a6a8264c0cdc72d047049610fc91e9f7c06f Steven Syrek 1468789759 -0400 commit: Separate out functions in Ord tests... Huh. “Maybe the same diff thing I did for the blob objects will work on the commit objects,” I thought. So: > git diff b267a6a8264c0cdc72d047049610fc91e9f7c06f..44dc22e706fb029a9c96f3bd125755fd55ac882b...(bunch of irrelevant stuff) OK. No. But at least I still had a commit hash, , to do with. It was the last good one before my arch nemesis, , darkened my world. I consulted the docs. I consulted the Internet. And I took one more shot in the dark: 44dc22e706fb029a9c96f3bd125755fd55ac882b something 6799ddac675cab54060cdfb066dbfadb6708fc3f > git branch -l rewrite-tests 44dc22e706fb029a9c96f3bd125755fd55ac882b What I did here was to create a new branch called , using the commit—i.e. the last good one—as its start point, in accordance with the pattern specified in the . I am not actually sure what the option is for, or even whether it's necessary. Someone said to use it. Shrug. rewrite-tests 44dc22e706fb029a9c96f3bd125755fd55ac882b git branch [--set-upstream | --track | --no-track] [-l] [-f] [ ] git-branch docs -l I then moved all the files out of the repo and did one of these: git checkout rewrite-tests The command sets to the specified branch. In other words, I told git that I wanted to work on the branch. Then, I just copied all the files back over, re-committed them, and left the branch to wither and die. git-checkout HEAD rewrite-tests restructure And just like that, to my astonishment, I was done. The worst was over and none too soon: I was starting to see in hash keys. I had a new branch to develop, and none of my work was lost (despite the untimely deaths of a few intervening commits). Eventually, I squashed everything back into , though I now tend to avoid working on that branch directly, in any repo, in the event one of these arises again. master kerfuffles I still visit from time to time, just to gloat. Actually, no, I don’t do that. But you can visit my wounded-and-repaired repo yourself, if you like: it contains my project. It’s a substantial amount of code that I worked on entirely alone. You can imagine how I felt when I thought I might have ruined it. And how I felt when I figured out how to fix it. Ms. Blinky maryamyriameliamurphies.js At the beginning of this article, I suggested that a damaged git repository — since it is composed of discrete objects — could potentially be recovered through careful surgery. We have seen two possibilities for such an operation. The first, in the case of damaged blob objects, is to excise the offending blob(s) and then suture over the wound through a rehashing of the original file. The second, if the first fails (or if the problem is a damaged commit object, not just a blob), is to amputate the wounded branch at the point of its corruption, graft a new branch onto the stump, and recommit any files that were casualties of the procedure. These are different solutions but similar in that they both entail repairing a data structure at a rather low level, even if it’s only manipulating files. In fact, the repair operations are possible precisely a git repo is stored as a series of files. Insofar as a file system is really just a large data structure with an interface, the command line, a git repo is also a file system-like data structure with an interface, the git command and its various sub-commands and options. If you can learn how to use a file system from the command line, in other words, you can learn how to use git, too. because I wish this story had resolved into a set of specific instructions for solving a problem that you yourself might one day encounter. Unfortunately, I only have some trite encouragement to offer: you can do it! Because it’s so hard to know the cause of these sorts of errors, not to mention the best way to fix them, it’s also hard to generalize about them. All you can do is dig in and fight back against entropy. Learn your tools, don’t fear them. Try things yourself, if only for the experience, and before you beg one of your friends to help you. Just remember to back up first! scientician If there is an obvious moral here, it’s the one that makes recovery much, much easier and far less costly should you ever be confronted by that dreaded, blinking status bar or its obtuse command line equivalent: commit early, and commit often. , after all, is often the best medicine. Preventive care