Payara5 systemd service going to timeout on system reboot

ERRORS:

Aug 3 07:52:49 systemd: Failed to start Payara 5 Server.
Aug 3 07:52:49 systemd: Unit payara5_supplytool.service entered failed state.
Aug 3 07:52:49 systemd: payara5_test.service failed.

Aug 2 03:29:02 ip-10-140-126-132 asadmin: Waiting for test to start …
Aug 2 03:29:02 ip-10-140-126-132 asadmin: No response from the Domain Administration Server (supplytool) after 600 seconds.
Aug 2 03:29:02 ip-10-140-126-132 asadmin: The command is either taking too long to complete or the server has failed.
Aug 2 03:29:02 ip-10-140-126-132 asadmin: Please see the server log files for command status.
Aug 2 03:29:02 ip-10-140-126-132 asadmin: Please start with the --verbose option in order to see early messages.
Aug 2 03:29:02 ip-10-140-126-132 asadmin: Command start-domain failed.

checked the logs and did not find anything wrong with domain however same ssystemd can start domain when Ec2 os up and running…

Please suggest what else i can check further

using this systemd file to start domain1 on boot

[Unit]
Description = Payara 5 Server
After = network.target syslog.target

[Service]
User = ec2-user
ExecStart = /payara5/p5_current_version/bin/asadmin start-domain domain1
ExecReload = /payara5/p5_current_version/bin/asadmin restart-domain domian1
ExecStop = /payara5/p5_current_version/bin/asadmin stop-domain domain1
TimeoutStartSec = 900
TimeoutStopSec = 30
Type=forking

[Install]
WantedBy = multi-user.target

Hi @marif,

The server.log should contain some information about what the server was doing before there was a timeout. You stated this works as expected when the EC2 instance is up and running, if your DAS is on that instance, then it would be required to be running for remote nodes and instances to start successfully.

Are you trying to start a remote node or instance without the DAS running and available for nodes to connect to?

Thanks,
James

It seems payara5 process are taking too much CPU while on boot

3331 ec2-user 20 0 11.9g 1.6g 18188 S 79.4 1.3 2:06.11 java
5482 ec2-user 20 0 11.9g 782568 27096 S 71.2 0.6 1:30.18 java
4024 ec2-user 20 0 11.9g 1.5g 18932 S 67.6 1.3 2:15.79 java
5026 ec2-user 20 0 8066616 631644 30428 S 62.7 0.5 1:28.52 java
3039 ec2-user 20 0 11.9g 1.8g 18192 S 61.8 1.5 2:09.35 java
4992 ec2-user 20 0 10.3g 600420 27440 S 61.8 0.5 1:28.26 java
3309 ec2-user 20 0 11.9g 1.7g 18876 S 59.2 1.4 2:03.54 java
5064 ec2-user 20 0 7741948 851756 32324 S 57.5 0.7 1:28.23 java
3457 ec2-user 20 0 12.0g 2.1g 18500 S 53.6 1.8 2:02.10 java
4588 ec2-user 20 0 12.0g 2.1g 18792 S 52.6 1.7 2:10.64 java
5623 ec2-user 20 0 9708508 758432 30564 S 52.3 0.6 1:28.63 java
3229 ec2-user 20 0 12.0g 2.1g 18396 S 49.7 1.8 2:09.73 java
3230 ec2-user 20 0 12.0g 2.1g 18348 S 48.0 1.7 2:09.94 java
5303 ec2-user 20 0 7608848 664340 30432 S 47.1 0.5 1:23.38 java
5298 ec2-user 20 0 10.3g 767380 30720 S 45.1 0.6 1:16.71 java
4573 ec2-user 20 0 11.9g 2.0g 18368 S 44.1 1.7 2:04.12 java
3652 ec2-user 20 0 11.9g 1.6g 18416 S 38.9 1.4 1:53.46 java
3398 ec2-user 20 0 11.9g 2.0g 18144 S 36.9 1.7 2:18.16 java
5533 ec2-user 20 0 10.3g 582620 26492 S 35.6 0.5 1:25.58 java
5924 ec2-user 20 0 12.3g 2.4g 31736 S 35.3 2.0 1:13.29 java
5505 ec2-user 20 0 13.4g 1.3g 32664 S 34.6 1.1 1:23.85 java
5920 ec2-user 20 0 6319984 546004 25712 S 32.7 0.4 1:23.19 java
3278 ec2-user 20 0 12.0g 2.2g 18264 S 28.4 1.8 2:07.99 java
5526 ec2-user 20 0 9819.7m 963620 27688 S 28.4 0.8 1:21.47 java
3324 ec2-user 20 0 12.0g 2.1g 18388 S 24.2 1.8 2:03.14 java
5785 ec2-user 20 0 9168960 512692 27588 S 22.2 0.4 1:19.75 java
5075 ec2-user 20 0 9909256 628936 26572 S 21.6 0.5 1:19.20 java
5166 ec2-user 20 0 10.3g 560924 27168 S 21.2 0.4 1:27.51 java

ec2-user@ip-10-140-126-132 ~]$ uptime
16:06:02 up 8 min, 1 user, load average: 141.98, 134.78, 71.76

[ec2-user@ip-10-140-126-132 ~]$ uptime
16:01:25 up 3 min, 1 user, load average: 211.51, 107.97, 42.73

My EC2 has 16 vCPU.

Server.log is not showing much. asadmin try to start application on boot via systemd and after 600s goes to timeout and kill the process.