Proxying Proxmox 4 with Apache

In some of our locations, we've found it's easier to grab a dedicated server and carve up the resources ourselves. This allows us to scatter some of our secondary services around the globe to improve the resilience of our network.

Our tool of choice thus far has been Proxmox for its ease of use and the fact that it's based on our favorite operating system, Debian (why we use Debian). The web interface is intuitive and makes it easy to handle everyday tasks related to our virtual machines.

To help secure our infrastructure we hide Proxmox behind an Apache reverse proxy. This allows us to put the battle tested Apache webserver on the front lines, and it allows us to more easily customize the behavior as we're more familiar with the Apache configuration. Apache allows us to customize our SSL configuration and require a client certificate for connection.

With Proxmox 3 this was super easy, any website that described how to set up a reverse proxy was a sufficient resource. There was one caveat however: using the standard Debian 7 packages you could not tunnel the noVNC console as it uses WebSockets and Apache 2.2 doesn't know how to proxy WebSockets; mod_proxy_wstunnel doesn't show up until Apache 2.4.5. We don't often need to view the console of our VMs so we made do with the Java applet.

Having a working noVNC console is nice for a few reasons:

1. If we break the network somehow, we've got console access. This is true of the Java VNC client as well, but see point #2.
2. It uses functionality natively provided by modern browsers instead of requiring an external plugin. I love Java. I used to be a Java developer. I don't want to run a Java applet ever nor do any of my browsers.

With our upgrade to Proxmox 4 we were looking forward to properly tunneling WebSockets and having a working console using Debian 8's Apache 2.4.10 packages. So we took our existing Apache config and added the two lines that seem to be listed on every example configuration:

• ProxyPass wss://localhost:8006/ retry=0
• ProxyPassReverse wss://localhost:8006/

At first it seemed as though everything worked great, but then we found that Paul wasn't able to log in. Then randomly I couldn't log in either. The biggest problem here was that sometimes I could log in and sometimes I couldn't; there didn't seem to be a clear pattern as to when I could and when I couldn't.

Not being able to log in is clearly more of a problem than not having a working noVNC console. As far as I could tell, no one else was having this problem. Either they'd found some other magic incantation or they'd switched to nginx. We're not really familiar with nginx so I was hesitant to use it in front of a somewhat critical piece of infrastructure without really understanding it.

The symptoms seemed to indicate that the Proxmox webserver was not receiving the username/password credentials we were providing. I found that by adding some debugging code (aka dumping stuff to file) to /usr/share/perl5/PVE/APIServer/AnyEvent.pm I could verify that when our log ins failed the request body was in fact empty.

I then looked at what Apache was seeing by turning on mod_dumpio. Key point when using mod_dumpio: the LogLevel directive must be LogLevel dumpio:trace7. If you have multiple LogLevel directives (virtual hosts for example), they all need to be set to the same value. I found it easier to just add a single directive in the main configuration and to temporarily comment out all the rest. Once you have mod_dumpio turned on, you'll see that Apache has the full, correct body content. You might also notice that it upgraded the connection to a WebSocket...

So that's interesting. Why would it use a WebSocket? Shouldn't we only need those for the noVNC console? As a test I removed the new ProxyPass lines and everyone was able to consistently log in. It seems we've identified the problem. I'm not sure if this is a bug in Apache, Proxmox, or even Firefox, but for now we just want to work around it.

As far as I know, there's only one URL that really needs WebSocket support, and that's the URL for the noVNC console. So we should be able to narrow the scope of the WebSockets proxy such that it does not affect the login requests. After a bit more trial and error, we ended up with the following configuration:

<VirtualHost *:443>
  ServerName <our secret server name>
  ServerAdmin <our secret admin email>
  DocumentRoot /var/www

  SSLEngine On
  SSLCertificateFile <our secret certificate file>
  SSLCertificateKeyFile <our secret certificate key>
  SSLCACertificateFile <our secret CA file>
  SSLHonorCipherOrder On
  SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"
  SSLProxyEngine on
  SSLProxyVerify none

  ProxyRequests off
  ProxyPreserveHost on

  <Location />
    ProxyPass https://localhost:8006/
    ProxyPassReverse https://localhost:8006/

    SSLVerifyClient require
    SSLVerifyDepth 1
    Order allow,deny
    allow from all
  </Location>

  <Location /api2/json/nodes/>
    <IfVersion >= 2.3>
      ProxyPass wss://localhost:8006/api2/json/nodes/ retry=0
    </IfVersion>

    SSLVerifyClient require
    SSLVerifyDepth 1
    Order allow,deny
    allow from all
  </Location>

  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  ErrorLog ${APACHE_LOG_DIR}/error.log
</VirtualHost>

After deploying this new configuration to all of our Proxmox servers, everything was awesome.