Memory leak in JavaEE Transaction Manager Simplified

Technical Discussion
1

We have an app that runs on Payara and has a bunch of scheduled quartz jobs running. After 25 days of continous running we’ve noticed that the app consumes 95% of heap so we took a heap dump and after analyzing it it shows that its queue in SheduledThreadPoolExecutor that is retaining bilions (116 bilions in our case) RunnabeScheduledFutures.
After further search we’ve discovered that said executor is scheduledTransactionManagerExecutor from JavaEETransactionManagerSimplified.

Executor does not have the property to remove on cancel set, instead there is a code in the constructor to set future to remove canceled tasks after certain ammount of cancelation:

    public JavaEETransactionManagerSimplified() {
        transactions = new ThreadLocal<>();
        localCallCounter = new ThreadLocal<>();
        delegates = new ThreadLocal<>();
        scheduledTransactionManagerExecutor = new ScheduledThreadPoolExecutor(
            Math.min(Runtime.getRuntime().availableProcessors(), 3)
        );
        if (getPurgeCancelledTtransactionsAfter() > 0) {
            purgeScheduledTransactionsFuture = scheduledTransactionManagerExecutor.scheduleAtFixedRate(
                    this::purgeCancelledTransactionTimeouts, 5, 5, TimeUnit.SECONDS
            );
        }
    }

We have this property set in our domain.xml like so

      <transaction-service automatic-recovery="true" tx-log-dir="${com.sun.aas.instanceRoot}/logs" timeout-in-seconds="300">
        <property name="xaresource-txn-timeout" value="300"></property>
        <property name="purge-cancelled-transactions-after" value="500"></property>
      </transaction-service>

hovewer when the constructor is called this property is equal to 0, which makes sense for it is not set anywhere before, so the future is not created. The property is read properly with our value 500 in initProperties() method with

                String v = txnService.getPropertyValue("purge-cancelled-transactions-after");
                if (v != null && v.length() > 0) {
                    purgeCancelledTtransactions = Integer.parseInt(v);
                }

however the future is not created there after reading that, so the cleanup is non existant. Cancelled task pille up over time.

I’m not sure if it is something on our end with our config, however looking at the class i dont see how it is ever supposed to work. The property read in constructor is not set before and only chance for it to work is with a custom manager that overrides it.
Should we do that and write our own manager or is there something we should configure in domain.xml to use different manager?

2

In case that is not fixable on Payara’s end how would anyone go about forcing custom TransactionManager?
I created my own version and declared it in META-INF services however ServiceLocator refuses to pick up my class as TransactionManager.

3

For now i did a thing i’m not a fan of but works.
I used reflection to start and set the necessary future to clean the queue.
I wasn’t able to force ServiceLocator to work with my implementation so that was the most straightforward solution, and we kinda need it as the only alternative is to restart the service after certain number of days.
I’m also very surprised that it was not noticed before and also checked if this implementation maybe changed in new jakarta (checked what was in payara embedded 6 alpha 3) and the code is the same.
I’m still not sure if it isn’t something on my side with config or i don’t know what, because it looks like it will always leak and somebody should have found it before, hovewer i don’t see any info about this and cannot really investigate further due to time constraint. So atm im sticking to my workaround-solution.