Use Apache 2.4.x as a secure reverse proxy for JBoss Wildfly

Quite some time ago, I wrote a post about using Apache as a secure reverse proxy for JBoss AS5. The development of both web servers and application servers has not come to a stand-still, so I felt it was time for a follow-up - not in the least because I have to configure such a set-up again myself.

This time around, Apache httpd has advanced to version 2.4 (the Ubuntu distro I'm using, 14.04 LTS, comes with version 2.4.7; the latest version available is 2.4.9); the JBoss application server has been going through versions AS6, AS7 and finally - reflecting a new naming scheme - Wildfly8 (I'm using version 8.1.0.Final).
For me, Apache is installed in /etc/apache2/, and you may install Wildfly anywhere (I'll use {$wildfly-home-dir} to denote the path).

Note: The JBoss documentation expresses a preference for mod_cluster when putting the app server behind an Apache. However, this component is - as far as I can tell - only available for httpd 2.2.x (x >= 8), and trying to included the precompiled modules in an httpd 2.4 led to errors.
I decided to keep the default Apache installation and use mod_proxy_ajp instead; that was possible since I didn't need the advantages mod_cluster advertises to have over its alternatives.

Securing the connection

To enable SSL security on the connection, enable the following parts in the /etc/apache2 directory by creating a symbolic link in the *-enabled subdirectories that point to the *-available subdirectories:

• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/ssl.conf
• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/ssl.load
• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/socache_shmcb.load
• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/rewrite.load
• /etc/apache2/sites-enabled$ sudo ln -s ../sites-available/default-ssl.conf

To make sure that all calls are made secure, redirect calls to the normal HTTP endpoints by adding an appropriate entry in the default virtual host configuration:

• In /etc/apache2/sites-enabled/000-default.conf:

    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Note 1: The preferred RedirectPermanent directive didn't seem to work properly for me. Maybe I just mis-configured something for that.

Note 2: An official SSL certificate needs to be purchased and installed to remove the warning browsers issue when landing on a page on the server. I'm not delving deeper into this issue here, see the mod_ssl description for details.

Enable the AJP protocol in Wildfly

To be accessible for the calls through the reverse proxy, Wildfly must expose a port on which it listens for traffic following the AJP protocol.

• Add an entry to the {$wildfly-home-dir}/standalone/configuration/standalone.xml file, in the undertow subsystem within the default-server section:

    <name="ajpListener" scheme="http" socket-binding="ajp"/>

That's all, because the corresponding socket binding is enabled by default (see bottom of that file), on port 8009.

Setting up Apache as a secure reverse proxy for Wildfly

Enable the following modules in order to be able to use mod_proxy_ajp:

• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/proxy.conf
• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/proxy.load
• /etc/apache2/mods-enabled$ sudo ln -s ../mods-available/proxy_ajp.load

Activate the secure reverse proxy to the application server:

• Add a proxying entry to the /etc/apache2/mods-enabled/proxy.conf file:

    ProxyPass / ajp://localhost:8009/

• Enable secure proxying in the /etc/apache2/sites-enabled/default-ssl.conf file:

    SSLProxyEngine on

Taking it for a test run

Now fire up the application server:

• {$wildfly-home-dir}/bin/standalone.sh

and (re)start the webserver:

• sudo service apache2 restart

and go to the root of your installation (which may be  http://localhost/  if you're trying this out locally). What you should see now is a warning from your browser, telling you the certificate that's used by the site you're trying to access is not trusted. If you choose to ignore this warning - you can, i.e. if you trust your own server - then you should be redirected to the landing page of the Wildfly installation (or anything you have deployed in the root context instead), served over a secure SSL connection.

"Too many certificates in chain"? It just may be a corrupt keystore!

Recently we ran into some trouble with the keyword expansion functionality that CVS offers (way too much false positives in the comparisons), and we simply decided to turn it off for our sources. That meant changing the ASCII/Binary property of all files from "ASCII -kkv" to "ASCII -kk" (keyword compression). Problem solved, albeit in a rather crude way.

Well, that sure bit us in the proverbial back end. The point is that when you change this property, you should be careful not to change it for files that were designated "binary". But we did.

One side effect of this was that a keystore file, stored in CVS, now became ASCII as well; effectively corrupting the file for further use. When trying to read a key from it, I got the following stack trace:

Caused by: java.io.IOException: Too many certificates in chain
   at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
   at java.security.KeyStore.load(Unknown Source)

In an attempt to locate the source for this, I stumbled upon this:

http://docjar.com/docs/api/sun/security/provider/JavaKeyStore.html#engineLoad(InputStream,%20char)

(just scroll down a bit, the top bar covers the important part!)

So from that code I gather that the occurring problem is actually an OutOfMemoryError (which I feel is kind of creepy) that is caused by the keystore implementation trying to load a corrupted keystore file. I think it's unlikely that there will be so many chained certificates in any practical keystore that this will really lead to running out of memory, so next time I see this error message I surely will think 'keystore file corruption'!

An inconsistency between explicit and implicit hashing when signing in Java security?

For the connection to a certain other system within our network, the program I'm working on needs to verify that it indeed is what it claims to be: an authorized client. A common way to accomplish is through PKI: it signs the message it sends using a private key, and the other system can verify this signature using the corresponding public key. See e.g. this article for an explanation of how this works.

In our case, there are three steps in signing a message:

• calculation of the message digest through a hashing algorithm,
• calculation of the digital signature using the private key, and
• coding the result to base64.

The last step is not part of the normal signing process, but we need to send the result as a string inside an XML message. Using the 'raw' signature would result in weird characters in the XML, very likely choking up the parser.

As I was coding away, I was lulled into performing each of these steps separately, so I started off with implementing the hashing using the java.security.MessageDigest class. I instantiated it with the "SHA-1" algorithm and simply called the digest method with the message to obtain
its hash. Pretty straightforward stuff.

Then I turned to the java.security.Signature class to supply me with the subsequent signing functionality. It occurred to me that there are algorithm choices that include hashing algorithm names, so I quickly found out that it is possible to let the Signature class take care of
both the first and second step of my signing process. While that struck me as quite convenient, I decided to stick to the original plan and not use the hashing possibility here. I chose the "NONEwithRSA" algorithm, and after feeding the message and private key the sign method provided me with an answer.

Then I encoded it in base64 (using the Apache Commons Codec library, which I also could have used for the hashing functionality) and presto! So I thought, at least...

But then...

The first test we performed immediately indicated something was wrong. And after checking everything else (like making sure code page encodings were correct and what not) we came to the conclusion that the signature itself had to be the culprit.

So I decide to put the 'my' way of creating a signature side by side with the signing method using the implicit hashing possibility, to see whether there might be a difference in the outcome:

public void test(byte[] data, PrivateKey privateKey) throws Exception {
   // Explicit hash and separate signing:
   byte[] hashedData = MessageDigest.getInstance("SHA-1").digest(data);
   byte[] signedData = signData(hashedData, privateKey, "NONEwithRSA");

   // Signing with implicit hashing:
   byte[] signedHashedData = signData(data, privateKey, "SHA1withRSA");

   System.out.println("Encoded data (explicit hashing) = "
           + new String(Base64.encodeBase64(signedData)));
   System.out.println("Encoded data (implicit hashing) = "
           + new String(Base64.encodeBase64(signedHashedData)));
}

private static byte[] signData(byte[] data, PrivateKey privateKey, String algorithm) throws Exception {
   Signature signature = Signature.getInstance(algorithm);
   signature.initSign(privateKey);
   signature.update(data);
   return signature.sign();
}

And even though:

• from the code it seems that both paths should lead to the same result: no other configuration than the algorithm names are given, so all else should be default, and
  
• from the Java security documentation for the signing options "NONEwithRSA" is stated to 'not use a digesting algorithm', so it should act as "SHA1withRSA" without the SHA-1 hashing,
  

there definitely is a difference between the outcomes!

In our situation, the implicit hashing turned out to deliver the correct result (at least, with regards to what the other system expected), so a minimal code change (getting rid of the explicit hashing step) did the trick. We use an external configuration file to set the signing algorithm through a system property, so changing that was easy.

What's causing this?

Now why is there a difference between the two approaches? I've tried to find an answer using Google, but that quest didn't turn up any answers.
So I did what any self-respecting developer would do: step through the implementation in a debugger. Unfortunately my toolkit didn't allow me to see everything I wanted to; however I could see that the input of the signing step was identical in both cases and the same implementation for signing is used under the hood (sun.security.rsa.RSACore). I cannot see what is happening with respect to internediate padding of the byte arrays, however, so I'm guessing that the 'default' settings of the two approaches - driven by different SignatureSpi implementations - differ in this respect.

If anyone could point out the actual difference to me, that would be greatly appreciated. For now I'll have to be content with knowing that the two approaches do return different results and that picking one at random may lead to problems...