I am running Payara payara/server-full:5.2021.7 docker image in Kubernetes.
I want to capture Payara Microprofile metrics using Prometheus to later visualize them in Grafana.
I followed the steps in the available guides and videos but I can’t get it to work.
The first thing that is different is that the metrics are exposed on port 4848 (admin) and not on 8080 as in the examples. The Enable Secure Access option in Server-config → Microprofile → Metrics is disabled. Is it correct that the metrics are exposed on port 4848?
In Prometheus I get the following error: “INVALID” is not a valid start token
I think it is a problem with authentication. I configured in Prometheus the authentication with the Payara admin user data but it does not work. I tested with the curl command to the metrics endpoint but I can’t get it to accept the credentials, it always returns the login html from the Payara console.
I do not know how to continue, is there any guide to configure Payara Microprofile with Prometheus in Kubernetes?
Hello @Marcelo. In theory, you should be able to make Payara Server expose MicroProfile metrics in the default 8080 port, so without a clear detail of your entire setup, there’s no way to be sure what is the cause of the problems you are encountering.
However, for a modern Kubernetes setup, it is recommended to use Payara Micro instead, so I recommend you giving this detailed guide a lecture: How to use Payara Micro with Kubernetes.
Using the default options, you should be able to configure Prometheus to scrape metrics from all instances running in the cluster.
You should be able to access the /metrics endpoint at port 8080. But there’s a bug which I discovered yesterday when I tried it. It doesn’t work if Admin Console isn’t loaded yet. If you first access the port 4848, then /metrics becomes available also on port 8080.
I recommend raising this as a bug on Payara github project.
I tested with Payara Micro but I couldn’t get the .war deploy to work so I’m using the Payara server-full image, although my idea is to use Payara Micro in the future.
As I had mentioned I could not get /metrics on port 4848 to accept the credentials using the curl command but it does work on the RESTer browser extension. What I forgot to mention is that for port 4848 a situation similar to that mentioned by @ondromih occurs. If you do not first enter the console on port 4848 in a web browser the /metrics does not work on that port. Is it correct that the console has to be loaded manually?
The detail of the Docker image I am using is as follows:
if [ -z $POSTBOOT_COMMANDS ]; then echo "Variable POSTBOOT_COMMANDS is not set."; exit 1; fi
if [ -z $POSTDEPLOY_COMMANDS ]; then echo "Variable POSTDEPLOY_COMMANDS is not set."; exit 1; fi
# Create post boot and post deploy command files if they don't exist
touch $POSTBOOT_COMMANDS_FINAL
touch $POSTDEPLOY_COMMANDS
cat $POSTDEPLOY_COMMANDS >> $POSTBOOT_COMMANDS_FINAL
postboot.commands:
asadmin disable-phome-home
set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.server-header=false
set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.xpowered-by=false
add-library ${ENV=PAYARA_DIR}/mysql-connector.jar
create-jdbc-connection-pool --datasourceclassname com.mysql.cj.jdbc.MysqlDataSource --restype javax.sql.DataSource mysql_app_rootPool
create-jdbc-resource --connectionpoolid mysql_app_rootPool app
set resources.jdbc-connection-pool.mysql_app_rootPool.property.password=${ENV=POOL_PASSWORD}
set resources.jdbc-connection-pool.mysql_app_rootPool.property.databaseName=${ENV=POOL_DATABASE_NAME}
set resources.jdbc-connection-pool.mysql_app_rootPool.property.serverName=${ENV=POOL_SERVER_NAME}
set resources.jdbc-connection-pool.mysql_app_rootPool.property.user=${ENV=POOL_USER}
set resources.jdbc-connection-pool.mysql_app_rootPool.property.portNumber=${ENV=POOL_PORT_NUMBER}
set resources.jdbc-connection-pool.mysql_app_rootPool.property.url=${ENV=POOL_URL}
set configs.config.server-config.monitoring-service.monitoring-enabled=true
set configs.config.server-config.monitoring-service.mbean-enabled=true
set configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.transaction-service=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.connector-service=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.jms-service=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.security=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.jersey=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.web-services-container=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.jpa=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.thread-pool=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.ejb-container=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.orb=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.connector-connection-pool=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.deployment=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
postdeploy.commands:
set configs.config.server-config.http-service.virtual-server.server.default-web-module=app
When I finished editing this answer I realized that the default-web-module option is responsible for /metrics not working on port 8080. Is it possible that this endpoint works using the default-web-module option? Although I would like it to work on 4848 or with some authentication mechanism to not expose the metrics to the public.