Get Elasticsearch integration test with gradle

Elasticsearch provide some test facilities officially. ESIntegTestCase allow you to start a local elasticsearch cluster from test container, so that you can test elasticsearch index/search/aggregation without mocking. However, there are actually quite a few pitfalls in order to make it work.

Basic setup:
Gradle dependencies:

testCompile 'org.elasticsearch.test:framework:6.5.2'
testCompile 'org.elasticsearch.plugin:transport-netty4-client:6.5.2'
testCompile 'org.apache.logging.log4j:log4j-slf4j-impl:2.9.0'

Also in order to use java 8 features in kotlin in integration test cases, the java version has to be declared explicitly. Note in gradle, you need separation declaration for class compilation and test class compilation，which is different from maven

compileTestKotlin {
        sourceCompatibility = JavaVersion.VERSION_1_8
        targetCompatibility = JavaVersion.VERSION_1_8

        kotlinOptions {
            jvmTarget = "1.8"
            apiVersion = "1.2"
            languageVersion = "1.2"
        }
    }

Setup test case:
The test case needs to inherit from ESIntegTestCase. It automatically starts an elasticsearch cluster in "beforeClass" or "before" method depending on the cluster scope (suite or test). @ESIntegTestCase annotation specifies how cluster is setup, in this case, we need only a single data node. Without specifying the node settings, ESIntegTestCase might start a cluster which random number of cluster nodes. @ThreadLeakScope annotation prevents error messages printed on console when elasticsearch server performs thread leak check. Once the test case is set up, it should be able to use "client()" method to get a elasticsearch client instance for various operation in test case.

@ESIntegTestCase.ClusterScope(numDataNodes=1, numClientNodes=0)
@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
public class RuleIntegratedTest : ESIntegTestCase()

Support remote connection:
We are using the new high level rest client in favor of the deprecated transport client. However, ESIntegTestCase does not expose a remote port for http connection. It only starts a local mock transport. To resolve this issue, we need to customize the node building by add a netty4 transport plugin. This can be achieved by overriding "nodeSettings" and "nodePlugins" method

override fun nodeSettings(nodeOrdinal : Int) : Settings {
    val randomPort = Random().ints(1, 15000, 20000).findFirst().asInt
    ruleLogger.debug("Random port is {}", randomPort)
    return Settings.builder().put(super.nodeSettings(nodeOrdinal))
            .put(NetworkModule.HTTP_ENABLED.key, true)
            .put(NetworkModule.HTTP_TYPE_KEY, "netty4")
            .put(HttpTransportSettings.SETTING_HTTP_PORT.key, randomPort).put("network.host", "127.0.0.1")
            .build()
}

override fun nodePlugins(): MutableCollection<Class<out Plugin>> {
    val result = mutableListOf<Class<out Plugin>>()
    result.add(Netty4Plugin::class.java)
    return result
}

To build the rest client, we need to know the elasticsearch server's port. This can be achieved by exploit the low level "client()" exposed by ESIntegTestCase. It sends a request to server to obtain cluster information, and host/port information can be obtained from there.

fun buildHighLevelClientFromClient(client: Client): RestHighLevelClient {
    val nodesInfoResponse = client.admin().cluster().prepareNodesInfo(*arrayOfNulls(0)).get() as NodesInfoResponse
    val nodes = nodesInfoResponse.nodes
    val httpHosts = ArrayList<HttpHost>()
    for (node in nodes) {
        logger.debug("Finding next node: {}",  node)
        if (node.http != null) {
            val publishAddress = node.http.address().publishAddress()
            val address = publishAddress.address()
            httpHosts.add(HttpHost(NetworkAddress.format(address.address), address.port))
        }
    }

    logger.info("Completed building hosts: {}", httpHosts)

    return RestHighLevelClient(
            RestClient.builder(*httpHosts.toTypedArray()))
}

I was getting a strange netty 4 processor issue when running test in maven

java.lang.IllegalStateException: available processors value [1] did not match current value [3]
        at org.elasticsearch.transport.netty4.Netty4Utils.setAvailableProcessors(Netty4Utils.java:90)
        at org.elasticsearch.http.netty4.Netty4HttpServerTransport.<init>(Netty4HttpServerTransport.java:252)
        at org.elasticsearch.transport.Netty4Plugin.lambda$getHttpTransports$1(Netty4Plugin.java:98)

So a system properties "es.set.netty.runtime.available.processors" has to be set to "false" to resolve this issue. However, i was not getting the same error in gradle, so it is not absolutely necessary.

test {
    testLogging {
        showStandardStreams = true
        events "PASSED", "STARTED", "FAILED", "SKIPPED", "STANDARD_OUT", "STANDARD_ERROR"
    }

    systemProperties = [
        "es.set.netty.runtime.available.processors": "false",
        "tests.security.manager": "false"
    ]
}

After this, some security error emerges when running the test case

[2019-01-26T12:47:14,358][ERROR][i.n.u.c.D.rejectedExecution] [node_sd3] Failed to submit a listener notification task. Event loop shut down?
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "setContextClassLoader")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_191]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_191]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_191]
        at java.lang.Thread.setContextClassLoader(Thread.java:1474) ~[?:1.8.0_191]
        at io.netty.util.concurrent.GlobalEventExecutor$2.run(GlobalEventExecutor.java:228) ~[netty-common-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.GlobalEventExecutor$2.run(GlobalEventExecutor.java:225) ~[netty-common-4.1.30.Final.jar:4.1.30.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
        at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:225) ~[netty-common-4.1.30.Final.jar:4.1.30.Final]

If you notice, in elasticsearch module folder: elasticsearch-6.5.4\modules\transport-netty4, there is a customized java security policy file: plugin-security.policy, which defines customized security right when start netty. I imported it into the test folder and use it when running elasticsearch integration test.

Gradle:

test {
    testLogging {
        showStandardStreams = true
        events "PASSED", "STARTED", "FAILED", "SKIPPED", "STANDARD_OUT", "STANDARD_ERROR"
    }

    systemProperties = [
        "es.set.netty.runtime.available.processors": "false",
        "java.security.policy": "${projectDir}/plugin-security.policy"
    ]
}

Maven:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <systemPropertyVariables>
                    <java.security.policy>${project.basedir}/plugin-security.policy</java.security.policy>
                    <es.set.netty.runtime.available.processors>false</es.set.netty.runtime.available.processors>
                    <buildDirectory>${project.build.directory}</buildDirectory>
                </systemPropertyVariables>
            </configuration>
        </plugin>
    </plugins>   
</build>

For maven, this is enough to make it work normally, But in gradle, importing a policy file is not sufficient to fully resolve the security issue. Gradle forks a separate process to run the test case, and due to the error, the test runner hangs indefinitely and no error will be shown in the console! Even a java thread dump does not show any useful information because the forked process dies and the test runner simply wait forever a response that is never returned. The correct approach is to disable java security policy completely by setting system property "tests.security.manager" = "false"

Jar Hell Issue:
Elasticsearch will perform jar hell check on startup. Because elasticsearch uses a plugin architecture, the jars ported with plugin could potentially cause library conflict. Jar hell check is to check whether conflict jar version occurs in classpath. When starting elasticsearch integration test in gradle, jar hell check fails. It complains about some hamcrest version issue. Googling on web, I found two solutions to this issue. I adopted the second solution.

1. Exclude hamcrest depedencies when compiling test case:

testCompile (group: 'junit', name: 'junit', version: '4.11') {
    exclude group:'org.hamcrest' //also included in es test framework
}

2. Disable jar hell check: create a JarHell check class with empty implementation and replace the default one:

package org.elasticsearch.bootstrap;

import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.util.Collections;
import java.util.Set;
import java.util.function.Consumer;

public class JarHell {
    private JarHell() {}
    public static void checkJarHell(Consumer<String> output) throws IOException, URISyntaxException {}

    public static Set<URL> parseClassPath() { return Collections.emptySet(); }

    public static void checkJarHell(Set<URL> urls, Consumer<String> output) throws URISyntaxException, IOException {}

    public static void checkVersionFormat(String targetVersion) {}

    public static void checkJavaVersion(String resource, String targetVersion) {}

}

Reference:

https://discuss.elastic.co/t/elasticsearch-5-1-1-simple-integration-test-hangs-running-gradle-test/70485

https://stackoverflow.com/questions/33975807/elasticsearch-jar-hell-error