Hi,
apparently I have issues with CORBA/IIOP where one Payara node fails to call a remote implementation (on a different Payara node) when the servers have been sitting idle overnight.
The error that shows up then is a “CORBA.COMM_FAILURE” with an underlying “Caused by: java.io.IOException: Broken pipe”. Subsequent calls seem to work after this “initial” error.
The architecture is a central business-logic Payara app, that is called via CORBA/IIOP/RMI from an outside Payara that offers some REST services.
The Payara versions we’re using is 5.201.
Very often the first (or first few) REST calls in the morning fail with this broken pipe error at the lowest level.
My suspicion is that somehow after a long idling time the IIOP connection between the servers is cut but the client is not correctly detecting this broken connection.
Only when the next communication over this supposedly working server connection would be required it realizes that it’s broken and does not recover internally but fail the whole call.
It was already tried to set the Payara System Property “fish.payara.SO_KEEPALIVE” to “true” hoping that it keeps those connections alive or allows for a transparent recovery.
But apparently this does not work (at least not as excpected) and the errors continue to happen.
I have dug through a lot of Payara and also Oracle/Glassfish documentation already around Corba/ORB/IIOP and Client/Server communication but I was unable to find any pointers on how/if this could be configured.
If it is helpful, see below a typical stacktrace of a failing call on the Client Payara instance when a REST call is attempted. When the REST call is repeated a second or third time it eventually succeeds as expected.
Any help or further pointers would be very appreciated.
Thanks very much!
Andreas
2022-03-23 08:48:16,624 [http-thread-pool::http-listener-1(29)] ERROR be_rest_admin 10.236.202.193 VSO01 [com.company.rest.api.security.R6OperatorRealm] java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
javax.ejb.EJBException: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
at com.company.base.mandator._MandatorManagement_Wrapper.retrieveMandator(com/company/base/mandator/_MandatorManagement_Wrapper.java) ~[mandator-management-interface-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at com.company.base.servicelocator.ServiceLocator$ServiceProxy.invoke(ServiceLocator.java:302) ~[company-commons-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.sun.proxy.$Proxy456.retrieveMandator(Unknown Source) ~[na:na]
at com.company.api.base.mandator.MandatorManagementDelegate.retrieveMandator(MandatorManagementDelegate.java:93) ~[company-api-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.company.rest.api.security.R6OperatorRealm.verifyMandatorExists(R6OperatorRealm.java:81) [rest-api-core-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.company.rest.api.security.R6OperatorRealm.doGetAuthenticationInfo(R6OperatorRealm.java:60) [rest-api-core-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:245) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.2.jar:1.4.2]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343) [spring-web-3.2.14.RELEASE.jar:3.2.14.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260) [spring-web-3.2.14.RELEASE.jar:3.2.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) [web-core.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) [web-core.jar:na]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) [web-core.jar:na]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) [web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) [web-core.jar:na]
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) [web-glue.jar:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) [web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371) [web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238) [web-core.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) [kernel.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) [kernel.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [nucleus-grizzly-all.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:259) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:201) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:140) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:226) ~[glassfish-corba-orb.jar:na]
at com.company.base.mandator.__MandatorManagement_Remote_DynamicStub.retrieveMandator(com/company/base/mandator/__MandatorManagement_Remote_DynamicStub.java) ~[mandator-management-interface-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
... 63 common frames omitted
Caused by: org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent
at com.sun.proxy.$Proxy398.writeErrorSend(Unknown Source) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.sendWithoutLock(ConnectionImpl.java:774) ~[na:na]
at com.sun.corba.ee.impl.encoding.BufferManagerWriteStream.sendFragment(BufferManagerWriteStream.java:145) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.BufferManagerWriteStream.overflow(BufferManagerWriteStream.java:85) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_2.grow(CDROutputStream_1_2.java:266) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_2.alignAndReserve(CDROutputStream_1_2.java:229) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.internalWriteOctetArray(CDROutputStream_1_0.java:454) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeString(CDROutputStream_1_0.java:407) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_string(CDROutputStream_1_0.java:384) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_codebase(CDROutputStream_1_0.java:1255) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeValueTag(CDROutputStream_1_0.java:1303) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:682) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:762) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.writeObjectField(IIOPOutputStream.java:766) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputClassFields(IIOPOutputStream.java:837) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.defaultWriteObjectDelegate(IIOPOutputStream.java:236) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputObject(IIOPOutputStream.java:609) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.simpleWriteObject(IIOPOutputStream.java:187) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueInternal(ValueHandlerImpl.java:238) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueWithVersion(ValueHandlerImpl.java:219) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValue(ValueHandlerImpl.java:179) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.callWriteValue(CDROutputStream_1_0.java:711) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:696) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.writeObjectField(IIOPOutputStream.java:766) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputClassFields(IIOPOutputStream.java:837) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.defaultWriteObjectDelegate(IIOPOutputStream.java:236) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputObject(IIOPOutputStream.java:609) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.simpleWriteObject(IIOPOutputStream.java:187) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueInternal(ValueHandlerImpl.java:238) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.callWriteValue(CDROutputStream_1_0.java:711) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:696) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$14.write(DynamicMethodMarshallerImpl.java:393) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeArguments(DynamicMethodMarshallerImpl.java:465) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:186) ~[glassfish-corba-orb.jar:na]
... 66 common frames omitted
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_191]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_191]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_191]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_191]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_191]
at com.sun.corba.ee.impl.transport.NioBufferWriter.write(NioBufferWriter.java:23) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.writeUsingNio(ConnectionImpl.java:507) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.write(ConnectionImpl.java:479) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.writeTo(CDROutputObject.java:225) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.sendWithoutLock(ConnectionImpl.java:758) ~[na:na]
... 112 common frames omitted
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueWithVersion(ValueHandlerImpl.java:219) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValue(ValueHandlerImpl.java:179) ~[na:na]