Exception during calling Connection close()

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Exception during calling Connection close()

HoiKin
I am using BTM with spring + hibernate and using the database SQL Server and Oracle.

Sometime, the exception thrown from JdbcPooledConnection#release()

And this exception cause the connection leak in the connection pool.

How can i fixed this problem?

Here is the stack trace of the Exception.

java.util.NoSuchElementException: iterator bounds reached
        at bitronix.tm.utils.Scheduler$SchedulerIterator.next(Scheduler.java:146)
        at bitronix.tm.internal.XAResourceManager.findXAResourceHolderState(XAResourceManager.java:167)
        at bitronix.tm.resource.common.TransactionContextHelper.isEnlistedInSomeTransaction(TransactionContextHelper.java:194)
        at bitronix.tm.resource.common.TransactionContextHelper.isEnlistedInSomeTransaction(TransactionContextHelper.java:212)
        at bitronix.tm.resource.common.TransactionContextHelper.requeue(TransactionContextHelper.java:113)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.release(JdbcPooledConnection.java:146)
        at bitronix.tm.resource.jdbc.JdbcConnectionHandle.close(JdbcConnectionHandle.java:77)
        at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:313)
        at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:274)
        at org.springframework.jdbc.datasource.DataSourceUtils$ConnectionSynchronization.afterCompletion(DataSourceUtils.java:443)
        at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCompletion(TransactionSynchronizationUtils.java:157)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.invokeAfterCompletion(AbstractPlatformTransactionManager.java:951)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCompletion(AbstractPlatformTransactionManager.java:926
)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy66.getStagingImageUploadQueue(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor1202.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:126)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
....
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
:
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

Ludovic Orban
Administrator
Hi,

What version of BTM are you using ? Can you please also collect debug logs, reproduce the issue and post the logs here ?

Thanks,
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

HoiKin
Hi,

  I am using BTM 1.3.2

  As the exception thrown from production environment randomly, I don't know how to reproduce the error.

Thanks,
HoiKin


Ludovic Orban wrote
Hi,

What version of BTM are you using ? Can you please also collect debug logs, reproduce the issue and post the logs here ?

Thanks,
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

Ludovic Orban
Administrator
I've tried to understand what could go wrong with the little amount of info you posted and I'm pretty certain you are facing a race condition.

I've checked the BTM code in depth and I could not find any obvious cause for that race condition, I can just say that TransactionContextHelper.isEnlistedInSomeTransaction() could (and seems to) contain one.

Fixing this race condition won't be an easy task as I would first need to find out how to reproduce it, or a the very least how it can happen.

In the short term, I see two options:

 - I could add extra synchronizing code in sensible places. This is more of a guess game than exact science as I currently have no way to make sure this will fix the problem.

 - We could try to figure out why the connection pool has to check all pending transactions when a connection is closed. This can only happen when you (or maybe Spring in this case) close the connection you got from the pool after the transaction has been committed. If you manage to get the connection closed before the transaction is committed I'm pretty sure this problem would disappear.

I think the second option is your best bet for now. Could you please describe in details the parts of your application that are doing data access and transaction management ? Are you using Hibernate, direct JDBC or whatever else ? How are you getting connections from the pool ? How are you closing them ? How did you configure Spring regarding transaction management ?

Please try to be as precise as possible and include code snippets / Spring config bits.

Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

HoiKin
First, thank a lot your great about my question.

All my hibernate and jdbc access (by JdbcTemplate) is wrapped by the spring.  I've checked the spring's code and the AbstractPlatformTransactionManager will close the connection after the underlying transaction manager commit.
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

HoiKin

HoiKin wrote
First, thank a lot your great about my question.

All my hibernate and jdbc access (by JdbcTemplate) is wrapped by the spring.  I've checked the spring's code and the AbstractPlatformTransactionManager will close the connection after the underlying transaction manager commit.
Spring will close the connection before commit if every connection is released. (org.springframework.jdbc.datasource.DataSourceUtils#ConnectionSynchronization)
Maybe my application make some connection didn't release before commit. I will try to make sure about that!!
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

Ludovic Orban
Administrator
Making sure connections are closed before the transaction terminates surely is better design and will definitely fix this problem as the code triggering the bug won't be executed anymore.

Please keep me posted about your progress and if you eventually managed to fix your problem.

In the meantime, this is nevertheless a bug in BTM that needs to be fixed. I've opened a JIRA issue to track it: http://jira.codehaus.org/browse/BTM-33

Thanks for reporting this.
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

HoiKin
I have change the property hibernate.connection.release_mode of hibernate to after_statement. After the setting changed, the connection will be closed before commit or rollback the transaction. Afterwards, the problem seems disappear now.

Again, thanks a lot for your great help!
Reply | Threaded
Open this post in threaded view
|

Re: Exception during calling Connection close()

Ludovic Orban
Administrator
Thanks for your feedback, I've added a note to the JIRA issue to make sure your discovery can also be found by other users facing this issue.

Ludovic