Sorcerer's Tower

Configuring Jetty for HTTPS with Let's Encrypt

The Jetty documentation for Configuring SSL/TLS is long and daunting, but makes no mention of how to work with the EFF's Let's Encrypt certificate authority, which provides free automated certificates with the aim of having the entire web available over HTTPS.

This article provides the steps for obtaining a Let's Encrypt certificate, importing it into Jetty, enabling HTTPS using the certificate, and handling renewals.

It assumes you have Jetty setup in a home/base configuration, serving over HTTP for one or more Internet-facing domain names.

As with all such guides, it is recommended to read all steps before making any changes, and ensure you have backups for any existing files you may modify.


1. Getting a Certificate with Certbot

Certbot is a command line tool for requesting Let's Encrypt certificates.

Obtaining a certificate is as easy as certbot certonly --webroot -w {webroot} -d {domain}

For multiple domains, specify multiple webroot/domain pairs in order, for example:

certbot certonly --webroot \
 -w /var/www/example.com -d www.example.com \
 -w /var/www/example.net -d www.example.net

(The backslashes here and in later examples escape the neighbouring newline characters, allowing copy-paste whilst maintaining readability. If typing the command on a single line omit the backslashes.)

The first time you run Certbot, it prompts you to create your Let's Encrypt ACME account - simply follow the prompts. Use a valid email address to receive a notification email sent 20 days before a certificate expires - i.e. letting you know if renewal hasn't worked.

Validation

To validate that the domain(s) in the certificate resolves to the server asking for it, Certbot creates a temporary file in each webroot, at {webroot}/.well-known/acme-challenge/{validationfile} and the Let's Encrypt validation server requests them via http://{domain}/.well-known/acme-challenge/{validationfile}

If you have Jetty URL rewriting in effect, you need to ensure the validation request is excluded. This can be as simple as prefixing your regex with (?!/\.well-known/)

If the validation does not succeed, you can update your configuration and repeat the command. (Do not undo changes afterwards - revalidation will occur at each renewal.)

Once validation is succesful, Certbot creates various certificate files inside the directory /etc/letsencrypt/live/www.example.com - there is one directory per certificate, named after the first domain specified in the command.


2. Importing Let's Encrypt PEM certificates into Jetty

In an ideal world, this stage would be as simple as enabling a module and Jetty would look in the Let's Encrypt directory and use the certificates it found.

Unfortunately, this is not the case, and there are two extra steps.

Convert the certificates from PEM format to PKCS12 format

openssl pkcs12 -export \
    -inkey privkey.pem -in fullchain.pem \
    -out jetty.pkcs12 -passout pass:p

The insecure password of p is assigned to the file jetty.pkcs12 because Java's keytool (next step) cannot import files without passwords.

Import the PKCS12 certificates into a Java keystore file

keytool -importkeystore -noprompt \
    -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -srcstorepass p \
    -destkeystore keystore -deststorepass storep

Again, we set a trivial password of storep because keytool cannot export files without passwords either, and has a six character minimum.

Having these files password protected matters if they will be on a machine with untrusted users. If that is your case, replace the values with something suitably secure.

It is the keystore file generated by the keytool command which Jetty must be configured to use, so move the keystore to {jetty-base}/etc/keystore and use chown/chmod to give it appropriate permissions.


3. Enabling HTTPS in Jetty

Enabling both https and ssl modules can be done by adding --module=https to your start.ini (or equivalent start.d config), and optionally the default port of 8443 can be changed by specifying jetty.ssl.port=443

To use the keystore requires an SslContextFactory, which is done via XML configuration.

Create {jetty-base}/etc/jetty-ssl-context.xml with the following content:

<Configure id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">
    <Set name="KeyStorePath">etc/keystore</Set>
    <Set name="KeyStorePassword">storep</Set>
</Configure>

(You can optionally add the XML and Jetty DOCTYPE lines, but it works without them.)

Both KeyStorePath and KeyStorePassword must be explicitly specified, even if using default values - whilst a missing KeyStorePassword prevents Jetty from starting, a missing KeyStorePath does not (but it will fail to serve HTTPS).

(If your password needs to be secret, make sure untrusted users are prevented from reading the XML file.)

Launch Jetty and your startup log should contain a line for oejus.SslContextFactory referencing the domain(s) in the certificate and the location of the keystore.

INFO:oejus.SslContextFactory:main: x509=X509@12a3bc4d(1,h=[www.example.com, www.example.net],w=[]) for Server@56efab78[provider=null,keyStore=/opt/jetty-base/etc/keystore,trustStore=null]

Confirm Jetty is serving requests with curl, for example:

curl -IsS https://www.example.com

You should see a 200 OK (or whatever you've configured the default response to be).

If you get certificate errors that disappear after adding -k to the above, this indicates Jetty is working but your certificate is not correct - pay attention to the error message.


4. Certificate renewal

Renewing Let's Encrypt certificates is really simple - run certbot renew and it renews any certificates that will expire soon (for the default 90 day certificates this means when there is less than 30 days to expiry).

This command can be scheduled to run every 24 hours and it only processes when it needs to, but note that EFF recommend not running it at midnight, or other significant times, to avoid unnecessary server congestion. Having it run daily (rather than every 30 days) ensures that it has plenty of opportunity to retry renewal if there are server issues.

Again, it would be great if it were possible to point Jetty at the PEM files and have it detect Certbot's renewals without restarting, but that does not work. We need to re-convert the certificates after each renewal - i.e. run step 2 again - and then restart Jetty for the new keystore to take effect, and we don't want to be doing that every night.

Certbot Deploy Hook

Fortunately, Certbot provides the ability to run a script when a certificate is succesfully renewed. This can either be specified as part of the renewal command (certbot renew --deploy-hook /path/to/deploy-hook-script) or by putting the script within the /etc/letsencrypt/renewal-hooks/deploy directory.

The Certbot renewal documentation includes an example script - here it is modified to use the commands from step 2:

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS
do

    case $domain in
        www.example.com)
            pkpass=p
            storepass=storep
            jettybase=/opt/jetty-base

            umask 077

            cd "$RENEWED_LINEAGE"

            openssl pkcs12 -export \
             -inkey privkey.pem -in fullchain.pem \
             -out jetty.pkcs12 -passout "pass:$pkpass"

            keytool -importkeystore -noprompt \
             -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -srcstorepass "$pkpass" \
             -destkeystore keystore -deststorepass "$storepass"

            chown jetty:web keystore
            chmod 400 keystore
            mv keystore "$jettybase/etc/keystore"

            # service jetty restart
            # above line superceded by ssl-reload module; see update below
        ;;
    esac

done

Save that as /etc/letsencrypt/renewal-hooks/deploy/convert-jetty-certs.sh with execute permission and it will run when certbot renew succesfully renews the certificates.

The final service jetty restart is for the new certificate to take effect, but of course it may be better to replace it with something to schedule the restart during the next quiet period, or instructs a loadbalancer to failover, or triggers an email notification, or any other process that makes sense - the script is an example to be updated as necessary.

I recommend not scheduling the certbot renew command immediately but setting a reminder to manually run it and ensure everything works smoothly at a time when any issues can be dealt with. (It's also wise to backup the existing keystore file so it can be restored if, for any reason, there's an issue with the new one.)

If you choose to go down that route, you should check whether Certbot has already setup a scheduled task as part of installation - see Certbot's documentation for further information.

Once you're happy that the hook script is behaving correctly, go ahead and [re]enable scheduled renewals, using either cron or systemd.

Update: Jetty SSL Reload Module

Since version 9.4.31, Jetty can be configured to monitor the keystore file and automatically apply the renewed certificate without a full restart.

The remainder of the deploy hook script is still needed — Jetty still does not handle PEM files — but instead of restarting the Jetty service, only the SSL keystore is reloaded and without the server going offline.

The functionality is activated by enabling the ssl-reload module, and the frequency of checks is controlled by jetty.sslContext.reload.scanInterval setting, so to check the keystore for changes once an hour would be:

--module=ssl-reload
jetty.sslContext.reload.scanInterval=3600

The Jetty documentation for this is at: https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_sslcontextfactory_keystore_reload


5. Recap

1. Get the certificate

certbot certonly --webroot \
 -w /var/www/example.com -d www.example.com \
 -w /var/www/example.net -d www.example.net

2. Convert and import

cd /etc/letsencrypt/live/www.example.com

openssl pkcs12 -export \
 -inkey privkey.pem -in fullchain.pem \
 -out jetty.pkcs12 -passout pass:p

keytool -importkeystore -noprompt \
 -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -srcstorepass p \
 -destkeystore keystore -deststorepass storep

chown jetty:web keystore
mv keystore /opt/jetty-base/etc/keystore

3. Update start.ini and etc/jetty-ssl-context.xml

cd /opt/jetty-base

echo -e '\n--module=https\njetty.ssl.port=443' >> start.ini
echo -e '\n--module=ssl-reload\njetty.sslContext.reload.scanInterval=3600' >> start.ini

echo -e '<Configure id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">' \
 '\n\t<Set name="KeyStorePath">etc/keystore</Set>\n\t<Set name="KeyStorePassword">storep</Set>' \
 '\n</Configure>' \
 > etc/jetty-ssl-context.xml

4. Automatic renewal.

Verify Jetty is serving over HTTPS (i.e. steps 1..3 worked ok), then add the deploy hook script: /etc/letsencrypt/renewal-hooks/deploy/convert-jetty-certs.sh

You may want to manually execute the first certbot renew to ensure everything works smoothly - setup a reminder (or two) for thirty days before the certificate expires.

Check the Certbot automated renewal documentation.



That's it! Hopefully this guide is both effective and easy to follow, but feedback is welcome.

I also monitor the jetty-users mailing list, which is likely the best avenue for any general Jetty questions.