kvm

Yesterday, I somehow arrived at Firecracker and ended up reading its design documents. Firecracker is a project by AWS which helps in creation + management of MicroVMS. This is of great interest to me, as I always wondered how AWS Lambda works and I have been interested in secure execution of code on servers for a long long time.

Firecracker’s docs describe that it uses KVM (Kernel-based Virtual Machine) behind the scenes to operate. hmm, KVM huh? I have heard of it before as an alternative to QEMU, Virtual Box etc. Well, is it really the alternative? hmm not sure. Well then what exactly is it?

I consider this to be a good start of learning about various virtualization technologies and probably this will give me good idea about things like, “what is KVM?”, “what are Containers?” etc.

So, here we go! A series of Wikipedia pages to read now :D

(oh I forgot, I haven’t used Firecracker or KVM practically yet. So I will see if I could give them a try during this time.)

Hoping on

First line in KVM Wikipedia page states

Kernel-based Virtual Machine (KVM) is a virtualization module in the Linux kernel that allows the kernel to function as a hypervisor.

ok, wait here…

This arises more questions.

  1. If it is the virtualization module in Linux Kernel, what other modules does the kernel have?
  2. What is a Hypervisor?

Kernel modules

After googling a bit, I arrived at this awesome link which teaches about kernel modules.

Basically it is where action is! You can add functionality to the kernel by loading modules to it instead of writing down all the code in the main kernel code. Having this modular approach avoids ending up with a monolithic kernel which has bigger size and complexity.

Ok, let us see what kernel modules are running in my pi. (lsmod is our friend)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
pi@raspberrypi:~/vishnubharathi.codes $ lsmod   
Module Size Used by
rfcomm 49152 4
bnep 20480 2
hci\_uart 36864 1
btbcm 16384 1 hci\_uart
serdev 20480 1 hci\_uart
bluetooth 368640 29 hci\_uart,bnep,btbcm,rfcomm
ecdh\_generic 28672 1 bluetooth
fuse 110592 3
joydev 20480 0
hid\_logitech\_hidpp 36864 0
brcmfmac 307200 0
brcmutil 16384 1 brcmfmac
hid\_logitech\_dj 20480 0
cfg80211 573440 1 brcmfmac
rfkill 28672 6 bluetooth,cfg80211
snd\_bcm2835 32768 1
snd\_pcm 98304 1 snd\_bcm2835
snd\_timer 32768 1 snd\_pcm
snd 69632 5 snd\_timer,snd\_bcm2835,snd\_pcm
evdev 24576 6
fixed 16384 0
uio\_pdrv\_genirq 16384 0
uio 20480 1 uio\_pdrv\_genirq
i2c\_dev 16384 0
ip\_tables 24576 0
x\_tables 32768 1 ip\_tables
ipv6 425984 42

I could see .ko extension files is the artifact that is needed to load a module. So, if you are writing a kernel module you would end up building a .ko (kernel object) file. To load this, use the insmod command.

I played around a while to see all the available kernel objects that came with my installation and here is what I found.

1
2
3
4
5
6
7
8
9
10
pi@raspberrypi:/lib/modules/4.14.79-v7+/kernel $ ls | xargs -n1   
arch
crypto
drivers
fs
kernel
lib
mm
net
sound

The above directories are used to categorize the kernel modules and fs is a category, which I suppose is responsible for file systems.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
fs  
├── 9p
│   └── 9p.ko
├── binfmt\_misc.ko
├── btrfs
│   └── btrfs.ko
├── cifs
│   └── cifs.ko
├── dlm
│   └── dlm.ko
├── ecryptfs
│   └── ecryptfs.ko
├── fuse
│   ├── cuse.ko
│   └── fuse.ko
├── gfs2
│   └── gfs2.ko
├── hfs
│   └── hfs.ko
├── hfsplus
│   └── hfsplus.ko
├── isofs
│   └── isofs.ko
├── jffs2
│   └── jffs2.ko
├── jfs
│   └── jfs.ko
├── nfs
│   ├── blocklayout
│   │   └── blocklayoutdriver.ko
│   └── flexfilelayout
│   └── nfs\_layout\_flexfiles.ko
├── nfsd
│   └── nfsd.ko
├── nilfs2
│   └── nilfs2.ko
├── nls
│   ├── nls\_cp1250.ko
│   ├── nls\_cp1251.ko
│   ├── nls\_cp1255.ko
│   ├── nls\_cp737.ko
│   ├── nls\_cp775.ko
│   ├── nls\_cp850.ko
│   ├── nls\_cp852.ko
│   ├── nls\_cp855.ko
│   ├── nls\_cp857.ko
│   ├── nls\_cp860.ko
│   ├── nls\_cp861.ko
│   ├── nls\_cp862.ko
│   ├── nls\_cp863.ko
│   ├── nls\_cp864.ko
│   ├── nls\_cp865.ko
│   ├── nls\_cp866.ko
│   ├── nls\_cp869.ko
│   ├── nls\_cp874.ko
│   ├── nls\_cp932.ko
│   ├── nls\_cp936.ko
│   ├── nls\_cp949.ko
│   ├── nls\_cp950.ko
│   ├── nls\_euc-jp.ko
│   ├── nls\_iso8859-13.ko
│   ├── nls\_iso8859-14.ko
│   ├── nls\_iso8859-15.ko
│   ├── nls\_iso8859-1.ko
│   ├── nls\_iso8859-2.ko
│   ├── nls\_iso8859-3.ko
│   ├── nls\_iso8859-4.ko
│   ├── nls\_iso8859-5.ko
│   ├── nls\_iso8859-6.ko
│   ├── nls\_iso8859-7.ko
│   ├── nls\_iso8859-9.ko
│   ├── nls\_koi8-r.ko
│   ├── nls\_koi8-ru.ko
│   ├── nls\_koi8-u.ko
│   └── nls\_utf8.ko
├── ntfs
│   └── ntfs.ko
├── ocfs2
│   ├── cluster
│   │   └── ocfs2\_nodemanager.ko
│   ├── dlm
│   │   └── ocfs2\_dlm.ko
│   ├── dlmfs
│   │   └── ocfs2\_dlmfs.ko
│   ├── ocfs2.ko
│   ├── ocfs2\_stackglue.ko
│   ├── ocfs2\_stack\_o2cb.ko
│   └── ocfs2\_stack\_user.ko
├── overlayfs
│   └── overlay.ko
├── quota
│   ├── quota\_tree.ko
│   ├── quota\_v1.ko
│   └── quota\_v2.ko
├── reiserfs
│   └── reiserfs.ko
├── squashfs
│   └── squashfs.ko
├── ubifs
│   └── ubifs.ko
├── udf
│   └── udf.ko
└── xfs
└── xfs.ko

30 directories, 73 files

Yeah! I could see some well know file system names. Does that mean a file system is just a kernel module and to write a file system, all I need to do is write down code and generate a .ko file and load it? (Strong guess from me is YES, but I will only know for sure if I attempt writing one or is someone who have attempted could tell me – let me know if you know the answer for this!)

Hypervisor

According to wikipedia,

A hypervisor or virtual machine monitor (VMM) is computer software, firmware or hardware that creates and runs virtual machines.

I have heard of this word “hyper” in the Windows world like “Turn on Hyper-V”. Well I turned it on, but without knowing what exactly it is. Now I have the answer. Hyper-V is the a hypervisor built into Windows (just like how KVM is for linux)

Paravirtualization

While reading about KVM, I came across a concept named Paravirtualization. Here is the Wikipedia verses,

A hypervisor provides the virtualization of the underlying computer system. In full virtualization, a guest operating system runs unmodified on a hypervisor. However, improved performance and efficiency is achieved by having the guest operating system communicate with the hypervisor. By allowing the guest operating system to indicate its intent to the hypervisor, each can cooperate to obtain better performance when running in a virtual machine. This type of communication is referred to as paravirtualization.

And there is a line in KVM Wikipedia that states about the support for paravirtualization in KVM

KVM provides paravirtualization support for Linux, OpenBSD,[12] FreeBSD,[13] NetBSD,[14] Plan 9[15] and Windows guests using the VirtIO[16] API. This includes a paravirtual Ethernet card, disk I/O controller,[17] balloon device, and a VGA graphics interface using SPICE or VMware drivers.

This line is important, because it is helping me understand the practical sense of paravirtualization. Best example: when I had run VMs previously, I noticed that the VM could connect to the host operating system and access the internet via a network connection through the host. This means the Guest operating system is able to speak with the host operating system and communicate its intent. I think this kind of capability is called Paravirtualization.

KVM Structure

kvm-structure

libvirt

I first encountered libvirt while checking out Digital Ocean’s libvirt Go package. libvirt is an open-source API written in C (with bindings in other languages) to manage virtualization. Hypervisors will be using this library to actually create and manage virtual machines.

libvirt

Time for action

I think I kind of went through some basic reading materials. Now I am going to try to create a virtual machine using the kvm interface.

I am on a raspberry pi and my lsmod already revealed that I don’t have kvm kernel module loaded. KVM’s support as per the official page is for CPUs running on Intel or AMD. So, I going to spin up a virtual machine in Digital Ocean to play around.

Doing an lsmod in my DO (Digital Ocean) box revealed that my kvm.ko and kvm_intel.ko are loaded.

1
2
3
4
root@dev:~# lsmod | grep kvm   
kvm\_intel 212992 0
kvm 598016 1 kvm\_intel
irqbypass 16384 1 kvm

libvirtd seems to be an important component, because it is the daemon that exposes a socket for accessing libvirt API to manage VMs

1
2
root@dev:~# file /var/run/libvirt/libvirt-sock  
/var/run/libvirt/libvirt-sock: socket

virsh is the command line client that connects to this socket and provides a CLI interface for managing the VMs

1
2
3
root@dev:~# virsh list --all   
Id Name State
\----------------------------------------------------

Currently no VMs are present as per virsh.

Let us try creating one!

The process dealt with installing some packages namely virt-manager, libvirt-bin, libosinfo-bin

After some strong belief on virsh and virt-install tools, I just successfully created a VM running alpinelinux3.8

virsh-alpine

I used virt-install to create the VM

1
2
3
4
5
6
7
virt-install --memory=128 \\  
--vcpus=1 \\
--cpu=host \\
--name=a38 \\
--cdrom=alpine-virt-3.8.0-x86\_64.iso \\
--os-variant=alpinelinux3.8 \\
--disk size=5

To manage this VM,

virsh list
virsh connect a38
virsh shutdown a38
virsh destroy a38

Before closing, I would like to check one more thing, “Where does the kvm kernel object sit?”

1
2
root@dev:/lib/modules/4.15.0-52-generic/kernel# find . -name kvm.ko  
./arch/x86/kvm/kvm.ko

I also noticed this thing called irqbypass.ko which is being used by kvm.ko