A Gentle Start A few months ago I was prototyping some KVM-based virtual machine management code. The goal was to boot a linux vm over iSCSI using pci-passthrough of virtual functions on an Intel NIC. I expected it to be straightforward. What followed was weeks of some of the hairiest debugging I have ever faced, leading to a bug I have named “The Hydra Bug”, because, like a hydra, each head I removed caused a new one to sprout. The vm boot uses a piece of open source software called . To understand my basic workflow, check out and of my tutorial on KVM + iSCSI. The first issue I faced was during the transition to the virtual function as described in . Read on for the story. iPXE Part I Part II KVM + iSCSI Part II After verifying that I can boot my image properly using a virtio NIC, I switch over to the prepared virtual function. I see iPXE starting in my VNC console, but it quickly prints an . I check the the and turn on debugging for the intel driver ( ), but none of the messages look suspicious. Maybe, I reason, if I compare the debug output to similar output from the virtio driver, I can see what it is doing differently. error iPXE docs DEBUG=intel,intelxvf I build iPXE with the virtio model number and debug turned on ( ) and specify that qemu should use it by passing a romfile parameter. Strangely, i see the same error in my console. make bin/1af41000.rom DEBUG=virtio-net What the? I go back and forth a few times trying to figure out what I might have done wrong and on the second try with the virtio NIC, it boots successfully. Oddly, though, there is no debug output in the console. At this point, I’m convinced I’m going insane. Through sheer dumb luck, I notice that I had forgotten to include the romfile parameter in the command line. My debug build was not being loaded, and it was automatically falling back to the older virtio iPXE option rom that came with my qemu package. Aha! So an old version of iPXE works, and the new one is broken. At this point, the easy solution would be to go back to the old working commit. Unfortunately, I can’t just use the old version because virtual function support was added to iPXE relatively recently. Besides, I had built iPXE at some point in the past year and I was pretty sure it was working then. I deduce that it must have been a fairly recent commit that broke things. If only i had an easy way to determine what broke things. Git-bisect to the rescue! Brief Aside: Git Bisect is a fantastic tool for finding the point in history where a bug was introduced. It uses a binary search to find the offending commit in the most efficient way possible. Bisecting is much easier if every commit in the tree is build-able, which is an excellent reason to keep your git history clean. Git-bisect Although the bisect process can be scripted, it is fairly easy to do manually. With the broken commit checked out and the sha of an older, working commit, you begin the bisect: git bisect startgit bisect bad # current HEAD doesn’t workgit bisect good $SHA # SHA is the working commit It will checkout the commit for you and tell you how many steps remain. You can then build the software and test it. If it works you type . Otherwise you type . It will then automatically check out a new commit and you repeat the process. Eventually you will zero in on the exact commit that introduced the bug. git bisect good git bisect bad To speed up the testing cycle, I write a script to build iPXE and boot the vm. Even though there had been thousands of commits between the working build and the broken one, bisecting quickly reveals the that broke things had only been merged two weeks before. commit The fix is anything but obvious (the bug is in some incomprehensible low level assembly for BIOS disk reads), so I report it and resolve to use iPXE from the commit right before it broke for my further testing. With my new iPXE build in hand, I am able to successfully boot my cirros image. upstream Now that the build is working, I am primed to discover the hydra bug itself. I discover the bug and a workaround in . detours into a related bug. The bug is fully analyzed and fixed in . Part I Part II Part III
