Virtualisation and Networking (xVM, crossbow)

The last year, since joining the Engineering organisation at Sun, I’ve been working on what started as the xen project (a port of the opensource xen project to Solaris) but, for reasons I won’t go into, had to be renamed to xVM, a name which has since taken on almost a life of its own and is now an umbrella for a lot of the virtualisation technology at Sun.

A few words on what xVM is and how it works: xVM is a technology for running more than one instance of an operating system (such as (Open)Solaris or Linux) on the same physical machine at the same time. To provide isolation for the running OS instances, also called (guest) domains, or guests, from one another, xVM provides a thin layer called the hypervisor which acts like a virtual machine towards the running domains. The hypervisor exposes a set of interfaces to these domains by means of which the domains can access HW and communicate with one another and the outside world. This implies that the OS in question be made aware of the fact that it’s running on top of the hypervisor, this is called paravirtualisation.
There are also provisions to run an OS completely agnostic of the underlying hypervisor; in xen parlance, this an HVM or fully virtualised guest. These require an additional emulation layer for I/O, which costs a performance penalty.
Finally, there are paravirtualised drivers for otherwise unmodified guests to work around the cost of the aforementioned emulation layer.

The hypervisor itself is minimal, for administrative stuff (like starting guest domains) and for things like access to network cards etc, it requires a privileged guest, dom0. All other guest domains are called domUs. (There is work underway to let domUs access eg. I/O HW directly, so the above description is a bit out of date, but it’s sufficient for the following discussion.) hosts a xen community page, where you can read a lot of good stuff about this effort.


Coming, as I was, from the Networking group, I worked on – surprise! –
networking stuff, specifically, on network communication between
Currently, domUs cannot talk to one another directly
(something our friends from the LDoms league have already achieved), but have to go via dom0. You can see an illustration of the data flow here (and thx to my colleague David Edmondson for letting me share this). If you’re familiar with the way this looks in Linux, you’ll notice some difference. This is due to vnics, technology we borrowed from the crossbow project at an early stage, another part of our overall virtualisation strategy that’s coming along nicely. More below. A word on names: in Xen parlance, a communication channel consists of a frontend and a backend. The frontend lives in domU, the backend lives in dom0 (therefore xnf and xnb)

There’s two distinct mechanisms that can be used for transporting data between domains: page flipping and hypervisor copying. For page flipping, the hypervisor sets aside some of its own (fairly limited) memory, which can then be used to transfer data between domains, involving repeated mappings into and out of the domains’ address space. For hypervisor copy (HVcopy), OTOH, guest domains set aside their memory and indicate this memory to the hypervisor, which copies data into this memory at another domain’s request (in our case, only dom0 – this means that for now HVcopy is only used for moving data into a domU, not out)

When I joined, only page flipping had been implemented for xVM, but it was found desirable to also have hypervisor copy. The reasons I can dredge up from memory are:

  1. in high-load situations, the hypervisor would run out of pages available for page flipping, which would cause tons of error messages on the console and a significant drop in NW throughput.
  2. some Windows (drivers, AFAIK Windows itself has not been paravirtualised) only "speak" HVcopy, and we intend (Open)Solaris to be able to host Windows domUs as well.
  3. expected performance increase.

I implemented HVcopy for xnb and xnf, and am happy to report that we successfully addressed issues 1 and 3. We could not find Windows drivers that would interoperate with Solaris as domU (corrected:) dom0, so that is as yet unresolved.


As I mentioned above, the vnic code we got from the crossbow project which we incorporated into the xVM code, and which was putback to Nevada in build 75, was a very early version, and this code has seen some significant change in the crossbow project itself since. Recently, I updated the relevant parts of the xVM code (xnbo, xnbu) to the new "look and feel" of the crossbow API – this means that as of about mid-January, it is possible to for the crossbow "bits" to be booted as dom0, to start domUs, and – this is what it’s all about, obviously, to successfully move network traffic in and out of the guest domains. As crossbow itself is not a part of Nevada yet, this change has only taken place in the crossbow "gate" (the copy of the Nevada gate where the crossbow development happens). Please refer to the crossbow community pages for when you can download what.

Update: Nicolas Droux pointed me to a more detailed document about network virtualisation in Solaris. Thx.

This entry was posted in Sun. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s