How does Payara Micro with active Hazelcast handle a killed pod?
Let’s say there are 3 pods running with Payara Micro. Kubernetes kills one of them, and no graceful shutdown was configured.
A common issue occurs: the remaining pods still consider the killed pod as active, which leads to errors.
Shouldn’t the cluster handle this situation automatically?
Or is there something I need to configure manually?
Thank you.
Greetings @Pedro, apologies for the late reply on your question. To answer your question, when running on a Payara Micro cluster in Kubernetes, as long as the Payara Micro instances are correctly configured using the kubernetesdiscovery mode and the requirements in the K8S cluster are set correctly, when pods are killed by the cluster their corresponding Payara Micro instances should leave the DataGrid/Hazelcast cluster in tandem.
I suggest you review the instructions given in this user guide: How to Use Payara Micro with Kubernetes – Payara Services Ltd
According to everything, the issue was that I hadn’t configured graceful shutdown properly. Payara and Hazelcast obviously need a bit of time to exchange information and allow the pod to terminate correctly.
One mystery remains: why does CPU usage gradually increase over time due to Hazelcast? It happens very slowly and doesn’t cause anything critical, but still…
Greetings Pedro,
Glad to hear you tracked down the cause of the stuck-member issue. Regarding CPU usage, if your application dynamically creates distributed objects, ensure they are explicitly destroyed when no longer needed. Periodically checking the cluster health and running diagnostics can help identify any potential leaks. I also recommend using the latest versions of Payara/Hazelcast, as some older versions might still have issues.
I hope this helps.
Thanks & regards
Ramya