Payara 6.2023.7 start-domain failed

We are running Payara 6.2023.7 via cargo-maven3-plugin and OpenJDK 17 as part of our GitHub Actions workflow.

Starting from a few days ago, Payara is refusing to start, reporting the errors as shown below.

Any idea?
Thank you in advance.

Listening for transport dt_socket at address: 8000
Launching Payara Server on Felix platform
Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishRuntime@39afc5bd in service registry.
Completed shutdown of GlassFish runtime
We are in non-embedded mode, so fish.payara.server.core.nucleus.glassfish [75] has nothing to do.
lug 20, 2023 2:11:51 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner createBundleProvisioner
INFORMAZIONI: Create bundle provisioner class = class com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.
lug 20, 2023 2:11:52 PM com.sun.enterprise.glassfish.bootstrap.LogFacade log
AVVERTENZA: Failed to install file:/home/ilgrosso/work/syncope/syncope/fit/build-tools/target/cargo/extract/payara-6.2023.7/payara6/glassfish/modules/libpam4j-repackaged.jar.
org.osgi.framework.BundleException: Unable to cache bundle: file:/home/ilgrosso/work/syncope/syncope/fit/build-tools/target/cargo/extract/payara-6.2023.7/payara6/glassfish/modules/libpam4j-repackaged.jar
        at org.apache.felix.framework.Felix.installBundle(Felix.java:3258)
        at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:147)
        at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.install(BundleProvisioner.java:448)
        at com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.installBundles(BundleProvisioner.java:208)
        at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:142)
        at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:161)
        at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:114)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:136)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:121)
        at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54)
Caused by: java.util.zip.ZipException: Invalid CEN header (invalid zip64 extra data field size)
        at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1728)
        at java.base/java.util.zip.ZipFile$Source.checkExtraFields(ZipFile.java:1261)
        at java.base/java.util.zip.ZipFile$Source.checkAndAddEntry(ZipFile.java:1212)
        at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1667)
        at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1445)
        at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1407)
        at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:716)
        at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:250)
        at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:179)
        at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:193)
        at org.apache.felix.framework.util.SecureAction.openZipFile(SecureAction.java:802)
        at org.apache.felix.framework.util.WeakZipFileFactory$WeakZipFile.<init>(WeakZipFileFactory.java:169)
        at org.apache.felix.framework.util.WeakZipFileFactory$WeakZipFile.<init>(WeakZipFileFactory.java:151)
        at org.apache.felix.framework.util.WeakZipFileFactory.create(WeakZipFileFactory.java:78)
        at org.apache.felix.framework.cache.JarRevision.<init>(JarRevision.java:83)
        at org.apache.felix.framework.cache.BundleArchive.createRevisionFromLocation(BundleArchive.java:804)
        at org.apache.felix.framework.cache.BundleArchive.reviseInternal(BundleArchive.java:491)
        at org.apache.felix.framework.cache.BundleArchive.<init>(BundleArchive.java:155)
        at org.apache.felix.framework.cache.BundleCache.create(BundleCache.java:457)
        at org.apache.felix.framework.Felix.installBundle(Felix.java:3254)
        ... 13 more

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:121)
        at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54)
Caused by: A MultiException has 2 exceptions.  They are:
1. com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [fish.payara.server.internal.batch.glassfish-batch-connector [383]], State = [NEW]
2. java.lang.IllegalStateException: Could not load descriptor SystemDescriptor(
        implementation=org.glassfish.batch.spi.impl.BatchRuntimeConfigurationInjector
        name=batch-runtime-configuration
        contracts={org.glassfish.batch.spi.impl.BatchRuntimeConfigurationInjector,org.jvnet.hk2.config.ConfigInjector}
        scope=jakarta.inject.Singleton
        qualifiers={org.jvnet.hk2.config.InjectionTarget}
        descriptorType=CLASS
        descriptorVisibility=NORMAL
        metadata=@table-suffix={optional,default\:,datatype\:java.lang.String,leaf},@data-source-lookup-name={optional,datatype\:java.lang.String,leaf},@table-prefix={optional,default\:,datatype\:java.lang.String,leaf},@schema-name={optional,default\:APP,datatype\:java.lang.String,leaf},@executor-service-lookup-name={optional,default\:concurrent/__defaultManagedExecutorService,datatype\:java.lang.String,leaf},target={org.glassfish.batch.spi.impl.BatchRuntimeConfiguration},Bundle-SymbolicName={fish.payara.server.internal.batch.glassfish-batch-connector},Bundle-Version={6.2023.7}
        rank=0
        loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [fish.payara.server.internal.batch.glassfish-batch-connector [383]], State = [NEW],1890097328)
        proxiable=null
        proxyForSameScope=null
        analysisName=null
        id=126
        locatorId=0
        identityHashCode=2078237101
        reified=false)

        at org.jvnet.hk2.internal.ServiceLocatorImpl.loadClass(ServiceLocatorImpl.java:2262)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:433)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:452)
        at org.jvnet.hk2.config.DomDocument$InjectionTargetFilter.matches(DomDocument.java:184)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getDescriptors(ServiceLocatorImpl.java:342)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getDescriptors(ServiceLocatorImpl.java:384)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getBestDescriptor(ServiceLocatorImpl.java:392)
        at org.jvnet.hk2.config.DomDocument.buildModel(DomDocument.java:135)
        at org.jvnet.hk2.config.ConfigModel.parseValue(ConfigModel.java:959)
        at org.jvnet.hk2.config.ConfigModel.<init>(ConfigModel.java:875)
        at org.jvnet.hk2.config.DomDocument.buildModel(DomDocument.java:114)
        at org.jvnet.hk2.config.DomDocument.getModelByElementName(DomDocument.java:162)
        at org.jvnet.hk2.config.ConfigParser.handleElement(ConfigParser.java:172)
        at org.jvnet.hk2.config.ConfigParser.parse(ConfigParser.java:108)
        at org.jvnet.hk2.config.ConfigParser.parse(ConfigParser.java:102)
        at org.glassfish.config.support.DomainXml.parseDomainXml(DomainXml.java:271)
        at org.glassfish.config.support.DomainXml.run(DomainXml.java:121)
        at org.jvnet.hk2.config.ConfigurationPopulator.populateConfig(ConfigurationPopulator.java:58)
        at org.glassfish.hk2.bootstrap.HK2Populator.populateConfig(HK2Populator.java:83)
        at com.sun.enterprise.module.common_impl.AbstractModulesRegistryImpl.populateConfig(AbstractModulesRegistryImpl.java:190)
        at com.sun.enterprise.module.bootstrap.Main.createServiceLocator(Main.java:249)
        at org.jvnet.hk2.osgiadapter.HK2Main.createServiceLocator(HK2Main.java:96)
        at com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishRuntime.newGlassFish(EmbeddedOSGiGlassFishRuntime.java:95)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishRuntimeDecorator.newGlassFish(GlassFishRuntimeDecorator.java:68)
        at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntime.newGlassFish(OSGiGlassFishRuntime.java:91)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:137)
        ... 6 more
Caused by: com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [fish.payara.server.internal.batch.glassfish-batch-connector [383]], State = [NEW]
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:193)
        at org.jvnet.hk2.osgiadapter.OsgiPopulatorPostProcessor$1.loadClass(OsgiPopulatorPostProcessor.java:54)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.loadClass(ServiceLocatorImpl.java:2254)
        ... 31 more
Caused by: org.osgi.framework.BundleException: Unable to resolve fish.payara.server.internal.batch.glassfish-batch-connector [383](R 383.0): missing requirement [fish.payara.server.internal.batch.glassfish-batch-connector [383](R 383.0)] osgi.wiring.package; (osgi.wiring.package=com.ibm.jbatch.spi) [caused by: Unable to resolve fish.payara.server.internal.batch.payara-jbatch [229](R 229.0): missing requirement [fish.payara.server.internal.batch.payara-jbatch [229](R 229.0)] osgi.wiring.package; (osgi.wiring.package=org.glassfish.weld) [caused by: Unable to resolve fish.payara.server.internal.web.weld-integration [4](R 4.0): missing requirement [fish.payara.server.internal.web.weld-integration [4](R 4.0)] osgi.wiring.package; (&(osgi.wiring.package=org.glassfish.web.deployment.descriptor)(version>=6.2023.0)(!(version>=7.0.0))) [caused by: Unable to resolve fish.payara.server.core.web.glue [366](R 366.0): missing requirement [fish.payara.server.core.web.glue [366](R 366.0)] osgi.wiring.package; (&(osgi.wiring.package=com.sun.enterprise.security.ee)(version>=6.2023.0)(!(version>=7.0.0))) [caused by: Unable to resolve fish.payara.server.core.security.ee [134](R 134.0): missing requirement [fish.payara.server.core.security.ee [134](R 134.0)] osgi.wiring.package; (&(osgi.wiring.package=org.jvnet.libpam)(version>=5.12.0)(!(version>=6.0.0)))]]]] Unresolved requirements: [[fish.payara.server.internal.batch.glassfish-batch-connector [383](R 383.0)] osgi.wiring.package; (osgi.wiring.package=com.ibm.jbatch.spi)]
        at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4398)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:2308)
        at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:1006)
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.startBundle(OSGiModuleImpl.java:224)
        at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:185)
        ... 33 more

Command start-domain failed.

Hi @ilgrosso,

We are aware of this issue when running Payara on Java 17.0.8 and 11.0.20, we believe it was caused by this JDK commit.

Our engineering team is looking into a permanent solution, however for now we would recommend trying the system property described in the screenshot below, or using JDK 11.0.19 until this is fixed.

Thank you @JamesHillyard adding

-Djdk.util.zip.disableZip64ExtraFieldValidation=true

did the trick.

Regards.

1 Like