Mutual TLS (mTLS) is an enhancement of TLS where both the server and the client present and validate X.509 certificates. Let us delve into understanding how to execute mTLS calls effectively.
1. Introduction
In standard TLS, the server presents a certificate so the client can verify the server’s identity. In mTLS, the client also presents a certificate that the server verifies. This provides mutual authentication and is commonly used for machine-to-machine communication and API security.
- Server certificate: proves the server’s identity to clients.
- Client certificate: proves the client’s identity to the server.
- Trust anchors: both sides validate each other’s certificates against configured CA/truststore.
2. Setting up mTLS on Nginx
Below are simplified steps to configure Nginx as an mTLS-enabled server. This assumes you have a server certificate/key and a CA used to sign client certificates.
2.1 Generate a CA, server certificate, and a client certificate (example with OpenSSL)
First, we generate a Certificate Authority (CA), server certificate, and a client certificate using OpenSSL commands:
# Create CA key and cert openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -subj "/CN=Example-CA" -out ca.crt # Server key and CSR openssl genrsa -out server.key 2048 openssl req -new -key server.key -subj "/CN=api.example.com" -out server.csr openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256 # Client key and CSR openssl genrsa -out client.key 2048 openssl req -new -key client.key -subj "/CN=java-client" -out client.csr openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365 -sha256 # Convert client cert+key to PKCS#12 for Java openssl pkcs12 -export -out client.p12 -inkey client.key -in client.crt -certfile ca.crt -name "client" -passout pass:changeit
These commands generate a root CA, create server and client keys and certificates, and finally export the client certificate in PKCS#12 format for Java applications.
2.2 Nginx Configuration (server block)
Next, configure Nginx to present the server certificate and require client certificates signed by your CA:
server {
listen 443 ssl;
server_name api.example.com;
ssl_certificate /etc/nginx/ssl/server.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
# Trust store containing CA that signed client certs
ssl_client_certificate /etc/nginx/ssl/ca.crt;
ssl_verify_client on; # require client certificate
ssl_verify_depth 2;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header X-Client-Cert $ssl_client_cert; # optional
}
}
This Nginx server block sets up SSL, configures the server certificate, and enforces client certificate verification using the CA certificate. Reload Nginx after placing the certs in /etc/nginx/ssl/:
sudo nginx -t && sudo systemctl reload nginx
The above command tests the Nginx configuration and reloads the server to apply the changes.
2.3 Code Example
We’ll demonstrate a Java 11+ client using the built-in HttpClient and a custom SSLContext loaded from a PKCS#12 keystore (the client.p12 generated above).
// File: MtlsHttpClient.java
import java.io.FileInputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.security.KeyStore;
import javax.net.ssl.*;
public class MtlsHttpClient {
public static void main(String[] args) throws Exception {
String p12Path =
"./client.p12"; // path to PKCS#12 (contains client cert + key)
char[] p12Password = "changeit".toCharArray();
// Load client key/cert into KeyManager
KeyStore keyStore = KeyStore.getInstance("PKCS12");
try (FileInputStream fis = new FileInputStream(p12Path)) {
keyStore.load(fis, p12Password);
}
KeyManagerFactory kmf =
KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(keyStore, p12Password);
KeyManager[] keyManagers = kmf.getKeyManagers();
// Load truststore (CA certs) - here we use the default truststore and also
// show explicit truststore
KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(
null); // empty -- we'll import CA programmatically if needed
// If you created a truststore.jks with your CA, load it instead:
// try (FileInputStream tfs = new FileInputStream("truststore.jks")) {
// trustStore.load(tfs, "changeit".toCharArray());
// }
// Use default trust managers (system CA) OR explicit
TrustManagerFactory tmf = TrustManagerFactory.getInstance(
TrustManagerFactory.getDefaultAlgorithm());
tmf.init((KeyStore) null); // system default
TrustManager[] trustManagers = tmf.getTrustManagers();
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);
HttpClient client = HttpClient.newBuilder().sslContext(sslContext).build();
HttpRequest request =
HttpRequest.newBuilder()
.uri(URI.create("https://api.example.com/secure-endpoint"))
.GET()
.build();
HttpResponse response =
client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Status: " + response.statusCode());
System.out.println(response.body());
}
}
2.3.1 Code Explanation
The MtlsHttpClient Java class demonstrates how to create an HTTP client that performs mutual TLS (mTLS) authentication. First, it specifies the path to a PKCS#12 file (client.p12) containing the client certificate and private key, and loads it into a KeyStore. A KeyManagerFactory is then initialized with this key store to provide the client credentials for the SSL handshake. Next, a trust store is set up to hold trusted CA certificates; in this example, it demonstrates both an empty programmatic trust store and the option to load an explicit truststore.jks. A TrustManagerFactory is initialized either with the default system trust store or a custom trust store to validate the server certificate. An SSLContext is then created and initialized with the client key managers and trust managers to enable secure TLS communication. Using this SSL context, an HttpClient instance is built, which is then used to create an HttpRequest pointing to a secure HTTPS endpoint. The client sends the request and receives a HttpResponse, printing both the HTTP status code and the response body to the console. Overall, this code establishes a secure, authenticated connection where both client and server certificates are verified, ensuring confidentiality and integrity of the communication.
2.3.2 Code Run and Output
The code, when executed, will return output like the following if Nginx accepts the client certificate and the upstream server responds with JSON:
$ java -cp target/myapp.jar MtlsHttpClient
Status: 200
{"user":"java-client","message":"hello from secure endpoint","time":"2025-09-07T10:12:34Z"}
2.3.2.1 Missing or empty keystore
If you run the client without a keystore (or with an empty KeyManager) and the server requires client certs, you typically get an SSL handshake/connection failure from the JVM:
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_required
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2054)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1126)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:708)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.security.ssl.SSLSocketInputRecord.decrypt(SSLSocketInputRecord.java:199)
... (stack trimmed)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1041)
... (stack trimmed)
2.3.2.2 Unknown CA for client certificate
If you present a client cert signed by an unknown CA (server’s ssl_client_certificate doesn’t include that CA), Nginx will reject it — the client sees an SSL handshake failure like:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:531)
at java.net.http/jdk.internal.net.http.HttpClientImpl$SslChannelWriter.write(HttpClientImpl.java:xxx)
at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:xxx)
... (stack trimmed)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:285)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:??)
... (stack trimmed)
2.3.2.3 Server hostname mismatch
If the server certificate CN/SAN doesn’t match the requested host (e.g., you connect to https://api.example.com but server cert is for other.example.com), you’ll see:
javax.net.ssl.SSLHandshakeException: hostname in certificate didn't match: <api.example.com> != <other.example.com>
at java.base/sun.security.ssl.HostnameChecker.match(HostnameChecker.java:??)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:??)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:??)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:??)
at java.base/sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:??)
... (stack trimmed)
2.4 Common pitfalls & troubleshooting
- Certificate chain issues: Ensure the server has a full chain and clients have the CA that signed the peer certs.
- Hostname verification: By default, the JVM verifies hostnames. Don’t disable it in production; if using test certs, set the CN/SAN properly.
- Keystore formats: Java often prefers JKS or PKCS12 for keystores. Convert with
keytooloropenssl pkcs12. - Permissions: The private key file (if used directly) should not be world-readable.
- Nginx logging: If Nginx rejects the client cert, inspect the error log (often shows verification failure reasons).
3. Conclusion
mTLS adds strong authentication by requiring both parties to present and validate certificates. With Nginx acting as the TLS terminator, you can centralize client certificate validation and forward the verified client identity to upstream services. On the Java side, modern HttpClient APIs make it straightforward to load a PKCS#12 keystore and build an SSLContext that performs client certificate authentication.
