Cannot run payara with Kubernetes Discovery Mode

Hi,

I want to setup payara full server 5.2021.7 in kubernetes. I setup hazelcast kubernetes discovery mode based on documentation (Domain Data Grid Discovery Modes :: Payara Community Documentation)

I use official docker image - payara/server-full:5.2021.7-jdk11.
I setup Kubernetes Discovery Mode with asadmin command


set-hazelcast-configuration --clustermode kubernetes --kubernetesServiceName \$\{ENV\=KUBERNETES_SERVICE_NAME\} --kubernetesNamespace \$\{ENV\=KUBERNETES_NAMESPACE\} 

When I run payara in kubernetes, I see in my log:

[#|2021-10-08T16:03:07.600+0200|SEVERE|Payara 5.2021.7|com.hazelcast.internal.cluster.impl.DiscoveryJoiner|_ThreadID=32;_ThreadName=RunLevelControllerThread-1633701783434;_TimeMillis=1633701787600;_LevelValue=1000;|
  [10.1.15.3]:4900 [development] [4.2] com/hazelcast/cluster/Address
java.lang.NoClassDefFoundError: com/hazelcast/cluster/Address
	at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.createAddress(KubernetesApiEndpointResolver.java:101)
	at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.addAddress(KubernetesApiEndpointResolver.java:83)
	at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.getSimpleDiscoveryNodes(KubernetesApiEndpointResolver.java:76)
	at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:62)
	at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:108)
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:72)
	at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:71)
	at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:60)
	at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:136)
	at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:96)
	at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)
	at com.hazelcast.instance.impl.Node.join(Node.java:810)
	at com.hazelcast.instance.impl.Node.start(Node.java:473)
	at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:123)
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
	at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
	at fish.payara.nucleus.hazelcast.HazelcastCore.bootstrapHazelcast(HazelcastCore.java:504)
	at fish.payara.nucleus.hazelcast.HazelcastCore.getInstance(HazelcastCore.java:239)
	at fish.payara.nucleus.config.ClusteredConfig.postConstruct(ClusteredConfig.java:100)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:667)
	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
	at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:759)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:722)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:692)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.<init>(ContainerMapper.java:113)
	at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.configureHttpProtocol(GlassfishNetworkListener.java:165)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configureSubProtocol(GenericGrizzlyListener.java:456)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configureProtocol(GenericGrizzlyListener.java:445)
	at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:341)
	at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:126)
	at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:673)

I also tried to copy hazelcast library (https://mvnrepository.com/artifact/com.hazelcast/hazelcast/4.2) to glassfish/lib folder and also hazelcast-kubernetes library (https://mvnrepository.com/artifact/com.hazelcast/hazelcast-kubernetes/2.2.2)

But, I got the same result.

Any help? Thanks

Hi,

There seems to be some classpath issues since

com.hazelcast.kubernetes.KubernetesApiEndpointResolver

is located within hazelcast-kubernetes-2.2.jar and

com/hazelcast/cluster/Address

is in hazelcast 4.2.

If there are no Hazelcast related jars within your application or added in the container image (since all hazelcast jar are in /modules directory by default), can you open a Github issue with the reproducer info and report it as a bug?

Regards
Rudy

Hi Rudy,

The libraries are inside docker container, in folder /modules.

It seems that is similar bug like this → Bug Report: Hazelcast kubernetes DNS discovery fails · Issue #5379 · payara/Payara · GitHub