"cannot enlist more than one non-XA resource" with Tomcat

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

"cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Hej Ludovic,

we are using an LRC datasource in Tomcat 6.0. with Oracle. The datasource
factory is implemented like it described in
http://docs.codehaus.org/display/BTM/LastResourceCommit13.

Transactions are handled with a proxy object, which enwraps all service
calls as a dummy ejb environment. Calls through this proxy may be nested
and DataSource.getConnection() is might be called more then once during
a transaction.

proxy code snippet:
final TransactionManager transactionManager = (TransactionManager) context.lookup("java:comp/env/TransactionManager");
  Transaction transaction = transactionManager.getTransaction();
  if (!EJBJarLoader.EJBConfig.TransactionAttribute.NOT_SUPPORTED.equals(transactionAttribute)) {
    // transaction necessary
    if (transaction == null) {
      newTransaction = true;
      transactionManager.begin();
      transaction = transactionManager.getTransaction();
    } else if (EJBJarLoader.EJBConfig.TransactionAttribute.REQUIRES_NEW.equals(transactionAttribute)) {
      Trace.notice("Transaction [{0}]: Requested transaction type not supported. Invoking within current transaction \"{1}\"",
                     new Object[]{transactionAttribute, transaction}, this);
    }
  }
  // handling service call
  ....
  catch (ReflectionException e) {
    if (newTransaction) {
      rollbackTransaction (transactionManager, transaction, transactionAttribute);
    }
    throw convertedError (e.getCause ());
  }
  if (newTransaction) {
    ...
    if (shouldRollback) {
      rollbackTransaction(transactionManager, transaction, transactionAttribute);
    } else {
      commitTransaction (transactionManager, transaction, transactionAttribute);
    }
  }

Everything works fine as long as the application is not working to
capacity, but during load tests the following exception occurs:
 bitronix.tm.internal.BitronixSystemException: cannot enlist more than one non-XA resource,
  tried enlisting an XAResourceHolderState with uniqueName=jdbc/FxFDataSource
  XAResource=a LrcXAResource in state NO_TX with XID null, already enlisted:
  an XAResourceHolderState with uniqueName=jdbc/FxFDataSource XAResource=a
  LrcXAResource in state STARTED (started) with XID a Bitronix XID
  [4678465F4170706C69636174696F6E00000123CD0AB6FE000122AC : 4678465F4170706C69636174696F6E00000123CD0AB700000122AE]

Actually we use only one datasource and so we dont need a real XA
transaction manager but on the other hand we need a transaction manager
and can't find any suitable single resource manager.

Any idea what we're doing wrong or is there perhaps a bug in BTM?

Many thanks,
  Tino Strauss

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Ludovic Orban
Administrator
Which version of BTM are you using? There is a well known race condition in 1.3.2 that you might have triggered, it happens when you close your JDBC connection after you commit the transaction. The easiest thing to do is to try out 1.3.3-RC2 and see if it helps.

If not, please collect debug logs while reproducing the problem and post them here.
Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Hej,

Ludovic Orban wrote:
> Which version of BTM are you using? There is a well known race condition in

we are using BTM 1.3.2.

> 1.3.2 that you might have triggered, it happens when you close your JDBC
> connection after you commit the transaction. The easiest thing to do is to
> try out 1.3.3-RC2 and see if it helps.

Actually im sure, that we always close the connection and commit the
transaction in the correct order. But we will try out the newest release
and post the results.

Thanks a lot,
  Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Simon-2745
In reply to this post by tino_usenet@ebene42.de
[hidden email] wrote:
> Actually we use only one datasource and so we dont need a real XA
> transaction manager but on the other hand we need a transaction manager
> and can't find any suitable single resource manager.
>  
Tino,

There is a "hidden gem" within the XA specification on which JTA is
based on which says that if only one source is written to in any
transaction then only single phase commit needs to be undertaken. This
means that there is no need to use LRC just to get single phase commit.
Any mature transaction manager will automatically use single phase
commit if only one resource is written to within a transaction order to
run faster as all transaction managers are keen to have good performance.

Simon


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
In reply to this post by tino_usenet@ebene42.de
Good evening!

> Actually im sure, that we always close the connection and commit the
> transaction in the correct order. But we will try out the newest release
> and post the results.

Unfortunately the problem continues with 1.3.3-RC2. It seems that the
BitronixSystemException occurs, where Datasource.getConnection() is called a
2nd time before the first connection is closed.

pseudo code:
 getConnection()
  -> call other code e.g. third party lib
      getConnection()
      close()
  <-
 close()

But i have to check this again.

Debug log is attached. The load test produce unmanageable tons of log entries,
therefore i greped for a affected GTRID. I hope no necessary information is
lost.

Thanks and good night,
   Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

filtered_tx.log (30K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Ludovic Orban
Administrator
Thanks for the logs, I think I have a better idea of what you're doing now.

Let's go back to your pseudo code where I think the root of the problem lies:

 getConnection()
 -> call other code e.g. third party lib
     getConnection()
     close()
 <-
 close()

It looks like that the 'call other code' bit is not always closing the connection it acquires before it returns. What points me to believe this is that you have this line logged:

XAPool - current transaction GTRID is [4678465F4170706C69636174696F6E00000123DCB1B8F4000114B0]

immediately followed by that one:

XAPool - NOT_ACCESSIBLE xa resource GTRID: 4678465F4170706C69636174696F6E00000123DCB1B8F4000114B0

which is fine. The problem is that in one case you have that log line:

XAPool - current transaction GTRID is [4678465F4170706C69636174696F6E00000123DCB1B8F4000114B0]

not followed by the one indicating that a NOT_ACCESSIBLE connection is available. A few lines below you then have this:

XAResourceManager - creating new branch with a Bitronix XID [4678465F4170706C69636174696F6E00000123DCB1B8F4000114B0 : 4678465F4170706C69636174696F6E00000123DCB1B99A000114C5]

which means that the connection pool could not find a closed connection to be reused it tried creating a new branch on another connection. Unfortunately LRC resources cannot support more than one branch at a time and this is why the TM rejected this request with BitronixSystemException: cannot enlist more than one non-XA resource.


As far as I can say there's no problem with BTM as it's just preventing your code from doing something wrong. I'd suggest you to carefully review your 'call other code' piece of code and look for some connection leak or misuse.
Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Hej hej!

Ludovic Orban wrote:
[..]
> It looks like that the 'call other code' bit is not always closing the
> connection it acquires before it returns. What points me to believe this is
> that you have this line logged:

Thanks for your advice, we have checked our code and the involved third party
lib. As far as I see, every connection is properly closed in a finally-
statement.
Meanwhile we made a ugly hack, which helps for the moment - caching the first
opened connection during a transaction, reusing and closing it immediately
before 'commit'.
The bug only occurs while the application is under load, so it has the smell
of a concurrency problem.
Any hints yet?

Bye,
 Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Ludovic Orban
Administrator
Tino,

Indeed, this smells like a race condition. With the narrow view I have over your application I'm more inclined to say that it probably is in your part of the code but you never know, I could be wrong.

Have you tried looking at the pooled objects in the JMX registry after the problem occurs? This might give you hints about where the problem is happening.

To fully troubleshoot a connection leak I would need a complete debug log file with a reproduction of the problem but as you pointed out, this isn't always practical. If you have any suggestion about what BTM could do more to help you out I'll consider you request to implement it.

Ludovic


2009/9/24 Tino Strauss <[hidden email]>
Hej hej!

Ludovic Orban wrote:
[..]
> It looks like that the 'call other code' bit is not always closing the
> connection it acquires before it returns. What points me to believe this is
> that you have this line logged:

Thanks for your advice, we have checked our code and the involved third party
lib. As far as I see, every connection is properly closed in a finally-
statement.
Meanwhile we made a ugly hack, which helps for the moment - caching the first
opened connection during a transaction, reusing and closing it immediately
before 'commit'.
The bug only occurs while the application is under load, so it has the smell
of a concurrency problem.
Any hints yet?

Bye,
 Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Hej, hej!

Ludovic Orban wrote:
[..]
> Indeed, this smells like a race condition. With the narrow view I have over
> your application I'm more inclined to say that it probably is in your part
> of the code but you never know, I could be wrong.

Today I was looking again for code, which is able to prevent that connections
will be properly closed. In the end, there was only the little piece of code
in the finally-statement:

if (connection != null) {
  try {
    if (!connection.isClosed()) {
      connection.close();
    }
  } catch (SQLException e) {
     ....
  }
}

After removing the isClosed() check, the application survived the load test.
Unfortunally I haven't had really much time for testing, so I have to check
this again tomorow. But it seems, there is a race condition in isClosed().

 Bye,
  Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Ludovic Orban
Administrator
I'm not sure I would call that a race condition as a JDBC connection object cannot be shared by threads.

The BTM JDBC pool's code surrounding this is quite light, in JdbcConnectionHandle.java:

    public boolean isClosed() throws SQLException {
        if (jdbcPooledConnection == null)
            return true;
        return getDelegate().isClosed();
    }

so I highly suspect a bug in your database or JDBC driver. What database and driver are you using and what are their version number?


Please also note that checking for isClosed() is completely useless in this case as the java.sql.Connection.close() javadoc explicity states that closing an already closed connection has no effect (see: http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Connection.html#close%28%29) so I would just leave it out and not bother about it.


2009/9/24 Tino Strauss <[hidden email]>
Hej, hej!

Ludovic Orban wrote:
[..]
> Indeed, this smells like a race condition. With the narrow view I have over
> your application I'm more inclined to say that it probably is in your part
> of the code but you never know, I could be wrong.

Today I was looking again for code, which is able to prevent that connections
will be properly closed. In the end, there was only the little piece of code
in the finally-statement:

if (connection != null) {
 try {
   if (!connection.isClosed()) {
     connection.close();
   }
 } catch (SQLException e) {
    ....
 }
}

After removing the isClosed() check, the application survived the load test.
Unfortunally I haven't had really much time for testing, so I have to check
this again tomorow. But it seems, there is a race condition in isClosed().

 Bye,
 Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Ludovic Orban wrote:
> I'm not sure I would call that a race condition as a JDBC connection object
> cannot be shared by threads.

Hmm, you are right. But unfortunately, I was wrong with my hasty conclusion:
the bug is still reproducible :(
I've puted a compressed debug log into my webspace but I'm not sure if you want
to inspect it - uncompressed size is 465MB! If so, the affected GTRID is
4678465F4170706C69636174696F6E00000123F07B8067000164BC
http://www.ebene42.de/tx.log.bz2

[..]
> so I highly suspect a bug in your database or JDBC driver. What database and
> driver are you using and what are their version number?

It's Oracle 10.2.0.3.0. with JDBC driver in the same version.

> Please also note that checking for isClosed() is completely useless in this
> case as the java.sql.Connection.close() javadoc explicity states that
> closing an already closed connection has no effect (see:
> http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Connection.html#close%28%29)
> so I would just leave it out and not bother about it.

Again, you're right. However, previousely we had sometimes trouble with double
close and some old connection pools (such as early releases from dbcp).
Therefore the connection was checked before closing it, but its obsolete now.

Bye,
  Tino


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

Ludovic Orban
Administrator
Tino,

I've analyzed the log file you posted and unfortunately it confirms my theory: somewhere your code does not close the connection for some reason.

If you look at line 1886880 for instance you can see that BTM found your previously closed connection and is about to recycle it. At this time, it is the 5th time the connection is recycled for the transaction 4678465F4170706C69636174696F6E00000123F07B8067000164BC.

At line 1886894, a 6th recycling is tried but if you look between lines 1886880 and 1886894 you'll find no trace of the connection being closed (line 1886869 shows what the pool logs when a connection is closed). This basically means a fresh connection has to be acquired from the pool and must join the global transaction which is impossible for a LRC resource.

Here's the flow of your transaction using pseudo-code:

tm.begin()

c = pool.getConnection // 1
c.prepareStatement()
c.close()

c = pool.getConnection // 2
c.prepareStatement()
c.close()

c = pool.getConnection // 3
c.prepareStatement()
c.close()

c = pool.getConnection // 4
c.prepareStatement()
c.close()

c = pool.getConnection // 5
c.prepareStatement()

c = pool.getConnection // 6
c.prepareStatement() // "cannot enlist more than one non-XA resource" exception happens here
...


It could very well be that the connection acquired from the 5th getConnection() call is closed in your code somewhere after the 6th getConnection() which would mean there actually is no connection leak.

Does that make sens to you?

There are basically two potential solutions to your problem: either make sure you only use one connection at a time from the pool or stop using the LRCXADataSource and switch to the OracleXADataSource.

There is an issue open in JIRA to improve BTM's pool to pin connections to transactions (http://jira.codehaus.org/browse/BTM-35) but I'm seriously lacking time (or funding) to implement it.

Ludovic



2009/9/25 [hidden email] <[hidden email]>
Ludovic Orban wrote:
> I'm not sure I would call that a race condition as a JDBC connection object
> cannot be shared by threads.

Hmm, you are right. But unfortunately, I was wrong with my hasty conclusion:
the bug is still reproducible :(
I've puted a compressed debug log into my webspace but I'm not sure if you want
to inspect it - uncompressed size is 465MB! If so, the affected GTRID is
4678465F4170706C69636174696F6E00000123F07B8067000164BC
http://www.ebene42.de/tx.log.bz2

[..]
> so I highly suspect a bug in your database or JDBC driver. What database and
> driver are you using and what are their version number?

It's Oracle 10.2.0.3.0. with JDBC driver in the same version.

> Please also note that checking for isClosed() is completely useless in this
> case as the java.sql.Connection.close() javadoc explicity states that
> closing an already closed connection has no effect (see:
> http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Connection.html#close%28%29)
> so I would just leave it out and not bother about it.

Again, you're right. However, previousely we had sometimes trouble with double
close and some old connection pools (such as early releases from dbcp).
Therefore the connection was checked before closing it, but its obsolete now.

Bye,
  Tino


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: "cannot enlist more than one non-XA resource" with Tomcat

tino_usenet@ebene42.de
Hej Ludovic,

sorry for the late response, meanwhile I had to switch to another project.

Ludovic wrote:
[..]
> It could very well be that the connection acquired from the 5th
> getConnection() call is closed in your code somewhere after the 6th
> getConnection() which would mean there actually is no connection leak.
>
> Does that make sens to you?

Not really, there is just only one method with getConnection followed by
closing the connection in a finally-statement. This method is used everywhere,
without shared states or something other suspicious stuff. It's really
strange.

> There are basically two potential solutions to your problem: either make
> sure you only use one connection at a time from the pool or stop using the
> LRCXADataSource and switch to the OracleXADataSource.

That is what our workaround does, caching the first opened connection during a
transaction and reusing it. That's not my favourite solution but it works.

Thanks a lot for your help (especially analyzing the huge log)!

Bye,
  Tino

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email