Indefinite WriteLockManager.acquireLocksForClone() waits after upgrading to version 6_6.2025.10 (or later) when submitting transactional workloads to a managed task executor

Technical Discussion
1

I am getting WriteLockManager.acquireLocksForClone() waits after upgrading to version 6_6.2025.10 (or later) when using submitting transactional workloads to a managed task executor.

From the release notes I can see that there is a change in the EclipseLink version : 4.0.7

  1. When I run the same logic without the managed task executor everything is fine, even with high concurrency from the HTTP threads (REST calls).

  2. Without high concurrency it runs fine, even when using the managed task executor.

  3. As soon as I hit start submitting tasks in quick succession (from REST) and I am using the managed task executor. The system start locking up.

  4. This is the kind of stack trace we get:

“http-thread-pool::http-listener-1(4)” #115 [71171] daemon prio=5 os_prio=31 cpu=28,57ms elapsed=552,00s allocated=2324K defined_classes=34 tid=0x000000012082a800 nid=71171 in O
bject.wait() [0x0000000315c10000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait0(java.base@21.0.9/Native Method)

  • waiting on
    at java.lang.Object.wait(java.base@21.0.9/Object.java:366)
    at org.eclipse.persistence.internal.helper.WriteLockManager.acquireLocksForClone(WriteLockManager.java:185)
  • locked <0x0000000797afa9d0> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1068)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1026)
    …
  1. A bit more testing shows that this is mostly likely due do a write in the managed task which does not get cleaned up properly in the L2 cache concurrency manager.
  2. See below same code for task submission and sample code of the task

Sample code of task submission:

...
   @Resource
   private ManagedExecutorService managedService;
   @Resource
   private ContextService contextService;      
   @Inject
   private Instance<EmailWithTemplateTask> taskProvider;
...     

     EmailWithTemplateTask task = taskProvider.get();
     task.setOrganisationId(orgId);
     task.setTo(to);
     // more task init code here

     managedService.submit(contextService.createContextualProxy(task, Runnable.class));
...

EmailWithTemplateTask sample code:


@Dependent
public class EmailWithTemplateTask implements Runnable, ManagedTask {
   @EJB
   private OrganisationBean organisationBean;
...
   @Override
   public void run() {
      try {
...
         int orgId = 1;

         Organisation org = organisationBean.get(orgId);
         if (org == null) {
            log.error("Not running managed task with jobId [" + jobId + "]. Failed to find org [" + orgId + "] in the system.");
            return;
         }

         OrganisationManagedTask omt = new OrganisationManagedTask();
         omt.setOrganisationId(org);
         // more init code
         omt = organisationBean.create(omt);
...

```

And the OrganisationBean sample code:


   public OrganisationManagedTask create(OrganisationManagedTask omt) throws BeanValidationException {
      // Validate
      BeanValidationUtils.validate(omt);

      // Get fresh org
      Organisation o = em.find(Organisation.class, omt.getOrganisationId().getId());

      // Wire both sides
      omt.setOrganisationId(o);
      o.getOrganisationManagedTaskList().add(omt);

      em.persist(omt);
      em.flush();
      log.debug("Created new organisation managed task with id [" + omt.getId() + "]");

      return omt;
   }

This code has has been working fine for over 10 years. I think there might be an issue with write cache keys (Organisation: 1) not being released properly from tasks run from a managed executor under load.

Nothing else in the stack trace show threads stuck in the task code. So the write keys (Organisation: 1) should in theory always get released at the end the task execution.

Has anyone got any ideas?

Regards

Rainer