Running Arbitrary Workloads in a nano VM

This post will show you how you can run an arbitrary piece of code in your nano VM. This post assumes that you already have a nano VM running (here). A full version of the program can also be found here (gitlab).

At this time we will write our application using assembly language and compile using nasm compiler (here).

Our program is will be very simple - it will take two numbers in two registers and output a result via serial port.

Following is our program written in assembly (Intel syntax):

	global start

start:  mov dx, 0x3f8
	add eax, ebx
	add eax, '0'
	out dx, al
	hlt

Firstly we load 0x3f8 into dx register. This address is commonly used as serial port. Take a note that 0x3f8 is in 1MiB of memory. Next we add numbers in eax and ebx registers (and store result in eax).

Following we add ascii value of zero to our result in eax register. Finally we write first byte (least significant byte, aka LSB) from the eax register (abbreviated as al) to the address stored in dx - serial port.

out instruction will cause our VM to exit (this is know as VM exit). VM exit will be handled by our host OS and then control will be returned to the VM. hlt instruction will then cause another VM exit which will be again handled by our host VM which will terminate the VM.

Before we can run the program we need to compile it. To do that we will use nasm compiler.

Assuming the program is stored in `program.asm` you can run the following command:

nasm program.asm -o program.bin

Next we need to load the program into memory that is will be mapped into the VM.

To do that we'll use two helper functions: get_file_size and read_into_buffer.

#define BUFFER_SIZE 32

off_t get_file_size(int fd) {
    struct stat buf;
    if(fstat(fd, &buf) < 0) {
            return -1;
    }
    return buf.st_size;
}

int read_into_buffer(int fd, char* buf) {
        char temp[BUFFER_SIZE];
        int r = read(fd, &temp, BUFFER_SIZE);
        int total_bytes_copied = 0;
        while(r != 0) {
                memcpy(buf + total_bytes_copied, temp, r);
                total_bytes_copied += r;
                r = read(fd, &temp, BUFFER_SIZE);
        }
        return total_bytes_copied;
}

get_file_size uses fstat function to get some information about file including file size.

read_into_buffer takes a file descriptor and a pointer to a chunk of memory into which the program will be placed. It then uses read function to read the file and copy it into the buffer.

1. Using the above two functions we can now load a file that's passed as an argument to our program as follows:

int main(int argc, char **argv) {
    int payload_fd = open(argv[1], O_RDONLY);
    off_t fsize = get_file_size(payload_fd);
    void *buf = malloc(fsize);
    if(read_into_buffer(payload_fd, buf) != fsize) {
            return -1;
    }
    ....
}

2. The next part of the puzzle is to copy contents of the buffer buf into memory area that will be used by VM as physical memory. To do that we use memcpy function:

    void *mem = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
    memcpy(mem, buf, fsize);
    struct kvm_userspace_memory_region region = {
        .slot = 0,
        .guest_phys_addr = 0x1000,
        .memory_size = 0x1000,
        .userspace_addr = (uint64_t)mem,
    };
    ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

3. Compile the main program and run the workload in a VM:

# Run the following command as sudo
./main program.bin

Conclusion

In this part you saw how to write a small program in assembly, compile it and run it in a VM. This is not as fancy as running an arbitrary C++/golang/rustlang/xyz program but it is something. In order for us to be able to run more complex programs we need to work a bit more on configuring our VM and VCPU.

Stay tuned for the next post when I will outline next steps that we will take in order to get us closer to running more complex programs produced by compilers such as C, C++ or even rust.

Full program is available at: https://gitlab.com/mvuksano/kvm-playground/-/blob/master/02b-load-payload/main.c