How To Run Your First Program in a Nano Virtual Machine

Most people associate a virtual machine (VM) with something slow and complex. Here I will show you two things - 1. VMs are not slow and 2. VMs can be super simple to use.

We will use KVM (https://www.linux-kvm.org/page/Main_Page) to run our code. This means that in order to execute this code you will need a Linux machine with KVM installed. Alternatively you can use Google Cloud Compute Engine with nested virtualization enabled (https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances).

Following is the code that we will run in our VM:

	global start

start:	mov dx, 0x3f8
	add al, bl
	add al, `0`
	out dx, al
	mov al, `\n`
	out dx, al
	hlt

The above code takes two values in registers al and bl and outputs the value via a serial port. We add "0" (ASCII value of character zero) to al register so we can display correct character. out instructions are normally used by a CPU to output a value to an IO port.

Since our VM does not have any IO ports this instruction will cause what is known as VMEXIT and signal our host OS to help with handling this instruction.

The above code when compiled produces the following sequence of bytes:

const uint8_t code[] = { 0xba, 0xf8, 0x03, 0x00, 0xd8, 0x04, '0', 0xee, 0xb0, '\n', 0xee, 0xf4 };

KVM API[1] is a set of ioctls that operate on file descriptors. The API is relatively simple, easy to work with and well documented - https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt.

Let's now try run this code in a VM!

1. obtain a handle to the KVM subsystem. This is done by opening /dev/kvm file.

int kvm = open("/dev/kvm", O_RDWR | O_CLOEXEC);

2. Use KVM_GET_API_VERSION to check if API version is exactly 12.

int ver = ioctl(kvm, KVM_GET_API_VERSION, NULL);
if (ver != 12) {
	printf("KVM_GET_API_VERSION expected 12 
 but got %d. Exiting.\n", ver);
}

3. Using KVM_CHECK_EXTENSION API call we can check if some capability is supported. In our example we will use KVM_CAP_USER_MEMORY. This capability will enable our host to specify memory contents (our example code) for our VM.

if(ioctl(kvm, KVM_CHECK_EXTENSION, KVM_CAP_USER_MEMORY) == -1) {
  printf("KVM_CAP_USER_MEMORY not available. Exiting."
  return -1;
}

4. Now it's time to create our VM. For that we will use KVM_CREATE_VM ioctl with machine type 0.

int vmfd = ioctl(kvm, KVM_CREATE_VM, (unsigned long)0);
if(vmfd == -1) {
  printf("There was a problem creating VM. KVM_CREATE_VM exit code: %d\n", vmfd);
  return -1;
}

5. Next we'll allocate a chunk of memory and place our code into it

void *mem = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
memcpy(mem, code, sizeof(code));

6. Use KVM_SET_USER_MEMORY_REGION to create a guest physical memory slot

struct kvm_userspace_memory_region region = {
  .slot = 0,
  .guest_phys_addr = 0x1000,
  .memory_size = 0x1000,
  .userspace_addr = (uint64_t)mem,
};
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

Here we tell KVM that the memory we allocated (mem) is the memory that will be associated with slot 0. It will be accessible starting at location 0x1000 from within the VM and it will be 0x1000 bytes (4Kb) in size.

7. We now need to add VCPU to our VM. To do that we can use KVM_CREATE_VCPU ioctl.

int vcpufd = ioctl(vmfd, KVM_CREATE_VCPU, (unsigned long)0);
if(vcpufd == -1) {
  printf("Could not create VCPU for VM %d. Error code: %d", vmfd, vmcpufd);
  return -1;
}

8. VCPU communicates with host OS via a shared memory region. To get the size of that region we can use KVM_GET_VCPU_MMAP_SIZE.

size_t mmap_size = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, NULL);
struct kvm_run *run = (struct kvm_run*) mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, vcpufd, 0);

9. In order for our VCPU to read the code form the right location we need to configure special register CS (code segment). We need to set base and selector value to 0.

To get special registers we use KVM_GET_SREGS and to write back changes we use KVM_SET_SREGS ioctl.

struct kvm_sregs sregs;
ioctl(vcpufd, KVM_GET_SREGS, &sregs);
sregs.cs.base = 0;
sregs.cs.selector = 0;
ioctl(vcpufd, KVM_SET_SREGS, &sregs);

The above configuration will ensure that if the code is found at address 0x1000 it is actually read from that physical address. In case CS is not set to 0 code would be read from a different physical location.

If you're interested why you may want to learn more about segmentation in x86 CPUs[*].

10. Before we can execute our program in the new VM we must configure the following registers

1. RIP (instruction pointer) needs to be set to 0x1000. This is where we placed our code to execute.
2. RAX, RBX need to be set to some value. Values in those two registers will be added and the result will be written to stdout.
3. RFLAGS needs to be set to 0x2. This is specified by x86 architecture. Not setting this register to 0x2 will cause VM to fail.

struct kvm_regs regs = {
  .rax = 2,
  .rbx = 2,
  .rip = 0x1000,
  .rflags = 0x2,
};
ioctl(vcpufd, KVM_SET_REGS, &regs);

11. Finally we are ready to run our VM. To do that we use KVM_RUN ioctl.

ioctl(vcpufd, KVM_RUN, NULL);

12. While running a VM some operations will not be able to complete and will require help from our host OS. In the case of our example program those are IN, OUT and HLT instructions. IN and OUT instructions will exit with reason KVM_EXIT_IO while HLT will exit with KVM_EXIT_HLT. At minimum those are exits that we want to handle. Once the VM exits you can run it again using KVM_RUN ioctl.

while (1) {
  ioctl(vcpufd, KVM_RUN, NULL);
  switch (run->exit_reason) {
  case KVM_EXIT_HLT:
    // handle HLT
  case KVM_EXIT_IO:
    if (run->io.direction == KVM_EXIT_IO_OUT && run->io.size == 1 && run->io.port == 0x3f8 && run->io.count == 1) {
      putchar(*(((char *)run) + run->io.data_offset));
      putchar('\n');
    }
    else {
      printf("unhandled KVM_EXIT_IO\n");
    }
    break;
  }
}

Full source code is available at: https://gitlab.com/mvuksano/kvm-playground.

Hopefully this shows you how easy it is to run a piece of code in a VM. You can use this boilerplate code to run a wide range of programs. You can also use it to learn more about how CPUs work or using nested virtualization.

In the next article I will show how to load arbitrary code and run it in a VM. Until then keep exploring KVM and keep learning about virtualization.