Disabling SSL verification for Elastic search Restclient not working in Java

I am trying to connect to elastic search hosted in a gcp box. To connect to this, there is a SSL check and I need a certificate.

However from these discussions, it is possible that we can turn off this verification.

How to disable SSL verification for Elasticsearch RestClient v6.7.0 in Java

https://discuss.elastic.co/t/host-name-does-not-match-the-certificate/186618

I had tried to remove the verification. But it throwing the following error

javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:947)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:229)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1732)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1694)
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1090)
    at org.dexter.lab.elasticUtils.ESUtils.getLastIndexedTimeStamp(ESUtils.java:44)
    at org.dexter.lab.druidUtils.DruidDelayChecker.main(DruidDelayChecker.java:357)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528)
    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197)
    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:970)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:967)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
    ... 9 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
    ... 17 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 23 more

This is what I did:

I did a override to verify function and returned true. So that it won’t check. This is the code for that

restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                @Override
                public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                    return httpClientBuilder.setSSLHostnameVerifier(new HostnameVerifier() {
                        public boolean verify(String s, SSLSession sslSession) {
                            return true;
                        }
                    });
                }
            });

I tried with another way as well as mentioned in the elastic thread

restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                @Override
                public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                    return httpClientBuilder.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
                }
            });

I had generated a self signed certificate and it worked. I had enabled debug logs to see why a certificate is needed.

These are the logs after adding certificate:

***
Found trusted certificate:
[
[
  Version: V3

This was followed by details about certificate.

The problem is I cannot generate self signed certificate for production environment and there I am getting exceptions for not using certificate even after overriding.

Is there any possible work around for this ? Any help is greatly appreciated.

Thanks in advance

Answer

In your example you only disabled the hostname verification. The server (ElasticSearch or something on top of ElasticSearch) is sending you the public key/certificate and your Restclient tries to validate that during the ssl handshake. What you need to do is tell to your RestClient that it is OK to receive a certificate from anyone but when it receives one it “should not really validate it”. So you need a custom trustmanager which has the task to validate the certificate but actually it doesn’t validate at all. What you need is an UnsafeX509ExtendedTrustManager, see below for the code snippets and the usage with the RestClient:

Option 1

/**
 * An insecure {@link UnsafeX509ExtendedTrustManager TrustManager} that trusts all X.509 certificates without any verification.
 * <p>
 * <strong>NOTE:</strong>
 * Never use this {@link UnsafeX509ExtendedTrustManager} in production.
 * It is purely for testing purposes, and thus it is very insecure.
 * </p>
 * <br>
 * Suppressed warning: java:S4830 - "Server certificates should be verified during SSL/TLS connections"
 *                                  This TrustManager doesn't validate certificates and should not be used at production.
 *                                  It is just meant to be used for testing purposes and it is designed not to verify server certificates.
 */
class UnsafeX509ExtendedTrustManager extends X509ExtendedTrustManager {

    public static final UnsafeX509ExtendedTrustManager INSTANCE = new UnsafeX509ExtendedTrustManager();
    private static final Logger LOGGER = LoggerFactory.getLogger(UnsafeX509ExtendedTrustManager.class);
    private static final X509Certificate[] EMPTY_X509_CERTIFICATES = new X509Certificate[0];
    private static final String CLIENT_CERTIFICATE_LOG_MESSAGE = "Accepting a client certificate: [{}]";
    private static final String SERVER_CERTIFICATE_LOG_MESSAGE = "Accepting a server certificate: [{}]";

    private UnsafeX509ExtendedTrustManager() {}

    @Override
    public void checkClientTrusted(X509Certificate[] x509Certificates, String authType) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(CLIENT_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public void checkClientTrusted(X509Certificate[] x509Certificates, String authType, Socket socket) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(CLIENT_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public void checkClientTrusted(X509Certificate[] x509Certificates, String authType, SSLEngine sslEngine) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(CLIENT_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public void checkServerTrusted(X509Certificate[] x509Certificates, String authType) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(SERVER_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public void checkServerTrusted(X509Certificate[] x509Certificates, String authType, Socket socket) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(SERVER_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public void checkServerTrusted(X509Certificate[] x509Certificates, String authType, SSLEngine sslEngine) {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug(SERVER_CERTIFICATE_LOG_MESSAGE, x509Certificates[0].getSubjectDN());
        }
    }

    @Override
    public X509Certificate[] getAcceptedIssuers() {
        return EMPTY_X509_CERTIFICATES;
    }

}

The above logging statements are not required at all, but handy if you want to see actually which certificate are received by the trustmanager.

The above trustmanager can be supplied to the RestHighLevelClient with the following snippet:

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, new TrustManager[]{ UnsafeX509ExtendedTrustManager.INSTANCE }, null);

RestClientBuilder restClientBuilder = RestClient
        .builder(new HttpHost("localhost", 9200, "https"))
        .setHttpClientConfigCallback(httpClientBuilder -> 
                httpClientBuilder.setSSLContext(sslContext)
                                 .setSSLHostnameVerifier((host, session) -> true));

By the way, I don’t recommend you or anyone else to use UnsafeX509ExtendedTrustManager. It is unsafe and should not be used at all in production.

Option 2

If you don’t want to add the custom code to your code base but just only want to easily disable the ssl verification, you might want to give the following snippet a try. It is a library to easily generate the SSLContext or other ssl materials and it has the option to disable the ssl verification.

<dependency>
    <groupId>io.github.hakky54</groupId>
    <artifactId>sslcontext-kickstart</artifactId>
    <version>6.6.0</version>
</dependency>

Usage

SSLFactory sslFactory = SSLFactory.builder()
          .withTrustingAllCertificatesWithoutValidation()
          .withHostnameVerifier((host, session) -> true))
          .build();

RestClientBuilder restClientBuilder = RestClient
        .builder(new HttpHost("localhost", 9200, "https"))
        .setHttpClientConfigCallback(httpClientBuilder -> 
                httpClientBuilder.setSSLContext(sslFactory.getSslContext())
                                 .setSSLHostnameVerifier(sslFactory.getHostnameVerifier());