Seeing occasional XAER_PROTO/XAER_RMFAIL errors

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Seeing occasional XAER_PROTO/XAER_RMFAIL errors

Christopher Wong
I wonder if someone can shed light on these XA exceptions I'm seeing on
an app server cluster connecting to an Oracle RAC cluster. Thanks in
advance for any insight. This does not happen all the time, but can
happen early on cluster startup or just during normal operations under
load. One node in the cluster would see an XAER_PROTO error:


twopc.Committer: resource 'myDb' reported XAER_PROTO when asked to commit transaction branch. Transaction is prepared and will commit via recovery service when resource availability allows.
oracle.jdbc.xa.OracleXAException
        at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1109)
        at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:561)
        at bitronix.tm.twopc.Committer$CommitJob.commitResource(Committer.java:194)
        at bitronix.tm.twopc.Committer$CommitJob.execute(Committer.java:183)
        at bitronix.tm.twopc.executor.Job.run(Job.java:70)
        at bitronix.tm.twopc.executor.SyncExecutor.submit(SyncExecutor.java:31)
        at bitronix.tm.twopc.AbstractPhaseEngine.runJobsForPosition(AbstractPhaseEngine.java:129)
        at bitronix.tm.twopc.AbstractPhaseEngine.executePhase(AbstractPhaseEngine.java:90)
        at bitronix.tm.twopc.Committer.commit(Committer.java:82)
        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:239)
        at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
        at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)

(...)


Another node in the cluster would see an XAER_RMFAIL at about the same
time, followed by a "unable to get a connection" error.

twopc.Committer: resource 'myDb' reported XAER_RMFAIL when asked to commit transaction branch. Transaction is prepared and will commit via recovery service when resource availability allows.
oracle.jdbc.xa.OracleXAException
        at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1109)
        at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:561)
        at bitronix.tm.twopc.Committer$CommitJob.commitResource(Committer.java:194)
        at bitronix.tm.twopc.Committer$CommitJob.execute(Committer.java:183)
        at bitronix.tm.twopc.executor.Job.run(Job.java:70)
        at bitronix.tm.twopc.executor.SyncExecutor.submit(SyncExecutor.java:31)
        at bitronix.tm.twopc.AbstractPhaseEngine.runJobsForPosition(AbstractPhaseEngine.java:129)
        at bitronix.tm.twopc.AbstractPhaseEngine.executePhase(AbstractPhaseEngine.java:90)
        at bitronix.tm.twopc.Committer.commit(Committer.java:82)
        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:239)
        at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
        at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)

(...)


org.hibernate.exception.GenericJDBCException: Cannot open connection
        at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
(...)`
Caused by: java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource myDb with 0 connection(s) (0 still available)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:211)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
        ... 67 more
Caused by: java.sql.SQLException: Closed Connection: OraclePooledConnection.getConnection() - SQLException Ocurred:Invalid or Stale Connection found in the Connection Cache
        at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
        at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:271)
        at oracle.jdbc.pool.OraclePooledConnection.getConnection(OraclePooledConnection.java:311)
        at oracle.jdbc.xa.OracleXAConnection.getConnection(OracleXAConnection.java:91)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory.invoke(JDBCConnectionProxyFactory.java:264)
        at $Proxy47.getConnection(Unknown Source)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.<init>(JdbcPooledConnection.java:87)
        at bitronix.tm.resource.jdbc.PoolingDataSource.createPooledConnection(PoolingDataSource.java:288)
        at bitronix.tm.resource.common.XAPool.createPooledObject(XAPool.java:291)
        at bitronix.tm.resource.common.XAPool.grow(XAPool.java:408)
        at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:387)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:129)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:97)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:207)
        ... 69 more




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Seeing occasional XAER_PROTO/XAER_RMFAIL errors

Ludovic Orban-2
Hi,

There are some serious limitations and things to know when using XA with Oracle RAC, especially w.r.t failover, load balancing and XA suspend/resume.

I would advise you to start with the following links, and pass that information to the DBAs. If you can, I would also advise you to contact Oracle support to get some help about the best things to do in your own environment.

http://www.oracle.com/technetwork/database/enterprise-edition/bestpracticesforxaandrac-128676.pdf
http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_xa.htm#BGBBCJJI
http://docs.oracle.com/cd/E11882_01/rac.112/e16795/hafeats.htm#BABBBCFG


To give you more help, I would need to know the _exact_ versions of both the Oracle server and your JDBC driver. Please also include all config info related to BTM (connection pools and TM config).

With a bit of luck, solving this problem will just be a matter of turning a knob or another that you may have forgotten but I cannot make any promise as Oracle RAC is quite a beast and RAC's XA support has been suboptimal in the past. Once again, I would advise you to involve Oracle support in this troubleshooting process.

Good luck!
Ludovic


2012/1/23 Christopher Wong <[hidden email]>
I wonder if someone can shed light on these XA exceptions I'm seeing on an app server cluster connecting to an Oracle RAC cluster. Thanks in advance for any insight. This does not happen all the time, but can happen early on cluster startup or just during normal operations under load. One node in the cluster would see an XAER_PROTO error:


twopc.Committer: resource 'myDb' reported XAER_PROTO when asked to commit transaction branch. Transaction is prepared and will commit via recovery service when resource availability allows.
oracle.jdbc.xa.OracleXAException
       at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1109)
       at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:561)
       at bitronix.tm.twopc.Committer$CommitJob.commitResource(Committer.java:194)
       at bitronix.tm.twopc.Committer$CommitJob.execute(Committer.java:183)
       at bitronix.tm.twopc.executor.Job.run(Job.java:70)
       at bitronix.tm.twopc.executor.SyncExecutor.submit(SyncExecutor.java:31)
       at bitronix.tm.twopc.AbstractPhaseEngine.runJobsForPosition(AbstractPhaseEngine.java:129)
       at bitronix.tm.twopc.AbstractPhaseEngine.executePhase(AbstractPhaseEngine.java:90)
       at bitronix.tm.twopc.Committer.commit(Committer.java:82)
       at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:239)
       at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)

(...)


Another node in the cluster would see an XAER_RMFAIL at about the same time, followed by a "unable to get a connection" error.

twopc.Committer: resource 'myDb' reported XAER_RMFAIL when asked to commit transaction branch. Transaction is prepared and will commit via recovery service when resource availability allows.
oracle.jdbc.xa.OracleXAException
       at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1109)
       at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:561)
       at bitronix.tm.twopc.Committer$CommitJob.commitResource(Committer.java:194)
       at bitronix.tm.twopc.Committer$CommitJob.execute(Committer.java:183)
       at bitronix.tm.twopc.executor.Job.run(Job.java:70)
       at bitronix.tm.twopc.executor.SyncExecutor.submit(SyncExecutor.java:31)
       at bitronix.tm.twopc.AbstractPhaseEngine.runJobsForPosition(AbstractPhaseEngine.java:129)
       at bitronix.tm.twopc.AbstractPhaseEngine.executePhase(AbstractPhaseEngine.java:90)
       at bitronix.tm.twopc.Committer.commit(Committer.java:82)
       at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:239)
       at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:120)
       at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)

(...)


org.hibernate.exception.GenericJDBCException: Cannot open connection
       at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
(...)`
Caused by: java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource myDb with 0 connection(s) (0 still available)
       at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:211)
       at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
       at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
       ... 67 more
Caused by: java.sql.SQLException: Closed Connection: OraclePooledConnection.getConnection() - SQLException Ocurred:Invalid or Stale Connection found in the Connection Cache
       at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
       at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
       at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
       at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
       at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:271)
       at oracle.jdbc.pool.OraclePooledConnection.getConnection(OraclePooledConnection.java:311)
       at oracle.jdbc.xa.OracleXAConnection.getConnection(OracleXAConnection.java:91)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory.invoke(JDBCConnectionProxyFactory.java:264)
       at $Proxy47.getConnection(Unknown Source)
       at bitronix.tm.resource.jdbc.JdbcPooledConnection.<init>(JdbcPooledConnection.java:87)
       at bitronix.tm.resource.jdbc.PoolingDataSource.createPooledConnection(PoolingDataSource.java:288)
       at bitronix.tm.resource.common.XAPool.createPooledObject(XAPool.java:291)
       at bitronix.tm.resource.common.XAPool.grow(XAPool.java:408)
       at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:387)
       at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:129)
       at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:97)
       at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:207)
       ... 69 more




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email