Context:
We have a 2-node Payara cluster that hosts couple of applications built using Java 11. The cluster is managed through Node01, which connects to Node02 remotely to perform any actions like stop/start applications.
We have regular server patching exercise that includes installation of JDK patches. Recently, after update of JDK 11 from 11.0.20.0.8 to 11.0.21.0.9, one of the applications in the Payara cluster fails to start.
Environment:
Linux kernel: 3.10.0-1160.105.1.el7.x86_64
Payara version: 5.2021.8
JDK version: 11.0.21.0.9
Error:
[2024-01-15T08:00:51.111+0000] [Payara 5.2021.8] [SEVERE] [javax.enterprise.resource.corba] [tid: _ThreadID=22 _ThreadName=RunLevelControllerThread-1705305626740] [timeMillis: 1705305651111] [levelValue: 1000] [[
iiop.createreference_exception]]
[2024-01-15T08:00:51.116+0000] [Payara 5.2021.8] [SEVERE] [NCLS-CORE-00026] [javax.enterprise.system.core] [tid: _ThreadID=22 _ThreadName=RunLevelControllerThread-1705305626740] [timeMillis: 1705305651116] [levelValue: 1000] [[
Exception during lifecycle processing
java.lang.RuntimeException: EJB Container initialization error
at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:237)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:286)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:104)
at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:218)
at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:334)
at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:570)
at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:416)
at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:248)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Unable to create reference
at org.glassfish.enterprise.iiop.impl.POARemoteReferenceFactory.createRef(POARemoteReferenceFactory.java:425)
at org.glassfish.enterprise.iiop.impl.POARemoteReferenceFactory.createHomeReference(POARemoteReferenceFactory.java:378)
at com.sun.ejb.containers.BaseContainer.initializeHome(BaseContainer.java:1393)
at com.sun.ejb.containers.StatelessSessionContainer.initializeHome(StatelessSessionContainer.java:184)
at com.sun.ejb.containers.StatelessContainerFactory.createContainer(StatelessContainerFactory.java:63)
at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:225)
… 20 more
Caused by: org.omg.CORBA.OBJ_ADAPTER: WARNING: 02310829: ReferenceFactoryManager createReference failed vmcid: OMG minor code: 829 completed: No
at com.sun.proxy.$Proxy326.rfmCreateReferenceFailed(Unknown Source)
at com.sun.corba.ee.impl.oa.rfm.ReferenceFactoryManagerImpl.createReference(ReferenceFactoryManagerImpl.java:548)
at com.sun.corba.ee.impl.oa.rfm.ReferenceFactoryImpl.createReference(ReferenceFactoryImpl.java:62)
at org.glassfish.enterprise.iiop.impl.POARemoteReferenceFactory._createRef(POARemoteReferenceFactory.java:443)
at org.glassfish.enterprise.iiop.impl.POARemoteReferenceFactory.createRef(POARemoteReferenceFactory.java:406)
… 25 more
Caused by: org.omg.PortableServer.POAPackage.AdapterNonExistent: IDL:omg.org/PortableServer/POA/AdapterNonExistent:1.0
at com.sun.corba.ee.impl.oa.poa.POAImpl.doActivate(POAImpl.java:1096)
at com.sun.corba.ee.impl.oa.poa.POAImpl.find_POA(POAImpl.java:1029)
at com.sun.corba.ee.impl.oa.rfm.ReferenceFactoryManagerImpl.createReference(ReferenceFactoryManagerImpl.java:545)
… 28 more
]]
Observations:
After rollback of the Java update package, and switching back to JDK 11.0.20.0.8, the issue is not seen anymore.
Request:
Can I get some help to understand what scenario leads to such an error? What should be my next step to investigate this issue?