I’ve been working with the oVirt API recently in order to build a customised control panel for internal (and eventually external) use. The API has mostly been pretty easy to work with, albeit somewhat poorly documented. Most of the time I’ve had to refer to the RSDL or error messages returned by syntax errors in order to see which properties are available and which of those are required. I’ve got by with little difficulty - just minor holdups - until this recent undertaking. Embedding a VNC console in the control panel.
Introduction to WebSocket, noVNC and websockify
We wanted to allow our internal users to be able to connect in to the console of a VM without any requirement for ops to help them out. This didn’t seem like it would pose a challenge, at the end of the day the console connection is just VNC (in our deployment, we only use VNC consoles and not SPICE consoles for various reasons). I even already knew of a project to bundle a VNC viewer in the browser, noVNC. So far, so good.
It is worth noting at this point that until oVirt 3.6.5 there was a bug with the API which caused requests for the “proxy ticket” (detailed later in this post) to fail when using a VNC connection. Apparently it worked for SPICE connections but I was unable to test this. BZ1305837.
On initial glance, it looks like noVNC would immediately achieve my goals. However, it is when you begin to read the documentation that you see it requires a “WebSocket Proxy”. The author of noVNC has released a project under the name of websockify which works hand in hand with noVNC to provide the required proxy.
A proxy is required in this instance as the browser cannot make a direct TCP connection to the VNC server. In order to combat this limitation, the WebSocket standard was formed and browsers have adopted it wonderfully. WebSockets allow raw TCP sockets between a browser and server, but they do require the server to be “WebSocket aware” as it needs to deal with the WebSocket handshake to set up the connection. This is where websockify comes in, it performs the WebSocket handshake and then happily forwards raw TCP traffic in both directions. There are VNC servers with support for WebSocket, but the one that oVirt uses does not have this support. Refer to the websockify project for some of the compatible VNC servers.
The oVirt developers have very kindly provided us with a pre-baked WebSocket proxy as part of the install. They have a page on their wiki detailing this functionality, including how to install it if you didn’t during your initial deployment. Their proxy appears to be a slightly modified version of the aforementioned websockify.
This proxy serves a second purpose too. As we are running on a cluster of hosts, we don’t necessarily know which host the guest is on at any one point. The proxy provides a central place (i.e. on the engine in most cases) to determine the correct host, find the VNC port required and hold the session open to the VNC server.
A tertiary purpose (or benefit, if you will) of this proxy is allowing just a single point of entry through your firewalls to the console. Instead of opening ports on every host to the outside world (they still need to be accessible to the proxy) we can open up just the WebSocket proxy.
With the basic knowledge of the architecture, I’ll now detail three “gotchas” I came across as I was implementing this feature and using the WebSocket proxy.
Issue 1: lack of understanding the websocket proxy path
Assuming you have a fairly standard install, the proxy will be available for use at wss://engine.example.com:6100 (in case you are unfamiliar with WebSockets, wss:// is the protocol for a TLS encrypted WebSocket). This on its own is not enough to talk to the proxy server though, a path is required to use this as a URL. In the websockify documentation their server runs with a path of /websockify, but oVirt’s does not. This lead to some initial confusion, but some digging around the source of oVirt’s own noVNC implementation revealed the requirement for a “proxy ticket”.
This proxy ticket is a very long base64 encoded JSON object. The JSON object has a handful of fields, including the actual useful data and a signature among other things. A proxy ticket can be obtained from the oVirt API at engine.example.com/ovirt-engine/api/vms/XXX/graphicsconsoles/YYY/proxyticket where XXX is the VM’s ID and YYY is the ID of the graphics console. This requires data to be posted which can simply be an empty <action/> tag.
The key fact is that it is this proxy ticket (in base64 encoding) which is the path that must be appended to the WebSocket URL. So we end up with a working URL of wss://engine.example.com:6100/AVERYLONGBASE64STRING. Cool, progress.
Issue 2: untrusted certificates
Now this issue has come to be solely due to me being tired after trying to get this to work for hours and expecting things to work differently then they did.
First, a quick aside to detail the configuration of the oVirt WebSocket proxy. It is configured somewhere in /etc/ovirt-engine/ovirt-websocket-proxy.conf.d, probably a file called 10-setup.conf if your install was anything like mine. The PROXY_PORT setting should be self explanatory, as should the SSL_CERTIFICATE and SSL_KEY settings. I am still unsure what the CERT_FOR_DATA_VALIDATION setting means, but have happily left it at the default. SSL_ONLY sets whether it will accept insecure WebSocket connections (via the protocol ws://).
With that in mind, we can see the default configuration allows only secure connections. Great. So we fire up our noVNC client and attempt to connect… Only to get a “Server Disconnected” error. Digging in the console doesn’t really help much, except reveal that the WebSocket connection never got established fully. So we dig into the server logs to see INFO msg:824 handler exception: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca. Seeing this error here is what threw me off track for so long as I expected it to be a server issue given that it was logged on the server.
Another quick aside, this time into the WebSocket handshake. Without going into too much detail, the handshake is performed over good ‘ol HTTPS. So we can deduce that a request is made to https://engine.example.com:6100/something.
With that covered, you can probably see where I’m going with this issue. We are running the standard install which includes an untrusted SSL certificate and due to this the browser is refusing to connect to the server. The error message for this isn’t particularly useful and I’m still perplexed as to how this is logged on the server (if you know, please email me, my address is on the homepage). An easy method of adding an exception is using the knowledge that the proxy will respond to HTTP and visiting the HTTPS URL and adding a certificate exception with your browser. With that done, the WebSocket connection will be established successfully!
Issue 3: obtaining a VNC password
This really isn’t an issue and is only labelled such to continue with the theme of this post. There isn’t much documentation from oVirt about this so it took a little bit of digging and I think fits this post.
The final hurdle is realising that we need a password to connect to the VNC server. This can be achieved with a simple API call to the engine.example.com/ovirt-engine/api/vms/XXX/ticket endpoint, posting XML similar to:
<action>
<ticket>
<value>password</value>
<expiry>60</expiry>
</ticket>
</action>
Obviously the content of the <value> tag is the password. The <expiry> field is the amount of time in seconds that the password is valid for. This only requires the VNC connection to be established in this time period and the connection will remain established even after the expiry of the password. For a system like ours where this API call is automated, we set this to a very low value like 5 seconds for security.
Wrap up
So there’s the three issues that bugged me during the implementation of an embedded VNC console to an oVirt guest.
- Lack of documentation regarding oVirt’s WebSocket proxy path.
- Lack of intelligence (on my part!) regarding untrusted SSL certificates.
- Lack of documentation regarding setting a VNC password.
Hopefully this blog post will serve to correct the first and final points, and maybe I’ll get around to editing the oVirt wiki to include this knowledge if they think it would be valuable.
I’ll eventually follow up this post with a working example codebase.
I welcome all comments and questions by email. My address is on the homepage of this site.