bitronix-scheduler blocks XAPool

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

bitronix-scheduler blocks XAPool

lesage
This post was updated on .
Hi all,

Is there someone who can help me ?
I remark that the problem occurs very often at application startup, when bitronix-scheduler starts at the same time another transaction starts


Thanks in advance,




Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

snicoll
Hi,

Are you using btm 2.0.1?

S.

On Thu, Sep 23, 2010 at 8:51 AM, lesage <[hidden email]> wrote:

Hi ,

I am facing with new issue :
Application is developped using tomcat , spring, hibernate, activemq ,
bitronix etc.
time to time application hangs. I capture with jconsole the thread dump

It seems that issue occurs when bitronix scheduler starts .

After analysing , It seems that bitronix-scheduler tries to access XAPool

with jconsole


Name: bitronix-scheduler
State: BLOCKED on bitronix.tm.resource.common.XAPool@5319f5 owned by:
ajp-10000-17
Total blocked: 1 Total waited: 12,351

Stack trace:
bitronix.tm.resource.common.XAPool.shrink(XAPool.java:211)
bitronix.tm.timer.PoolShrinkingTask.execute(PoolShrinkingTask.java:28)
bitronix.tm.timer.TaskScheduler.executeElapsedTasks(TaskScheduler.java:198)
bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:174)





and on another thread

Name: ajp-10000-7
State: BLOCKED on bitronix.tm.resource.common.XAPool@5319f5 owned by:
ajp-10000-17
Total blocked: 25 Total waited: 23

Stack trace:
bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68)
bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:178)
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:558)
org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
my.company.webservices.config.ConfigurationServiceImpl$$EnhancerByCGLIB$$66933f06.createCache(<generated>)
my.company.ui.server.services.GwtUserServiceImpl.getCurrentUser(GwtUserServiceImpl.java:41)


Could you please advice what is going on ?



--
View this message in context: http://old.nabble.com/bitronix-scheduler-blocks-XAPool-tp29786534p29786534.html
Sent from the Bitronix Transaction Manager mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage

Hi,

Yes

I am using 2.0.1

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

snicoll
Okay. I am asking because I had a similar issue which has been fixed in 2.0.1. Can you create a full threads dump when this happens? (kill -3 pid on *nix). Having a full view on your application will help.

Please post your datasouce/btm config as well.

Thanks,
Stéphane

On Thu, Sep 23, 2010 at 9:07 AM, lesage <[hidden email]> wrote:


Hi,

Yes

I am using 2.0.1

Thanks
--
View this message in context: http://old.nabble.com/bitronix-scheduler-blocks-XAPool-tp29786534p29786622.html
Sent from the Bitronix Transaction Manager mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage
datasource config

<bean id="dataSource" class="bitronix.tm.resource.jdbc.PoolingDataSource"
                init-method="init" destroy-method="close">               
               
               
               
               
                <property name="uniqueName" value="#{datasource.uniqueName}" />
               
                <property name="className" value="#{datasource.jdbc.driverClassName}" />
                <property name="minPoolSize" value="#{datasource.minPoolSize}" />
                <property name="maxPoolSize" value="#{datasource.maxPoolSize}" />
                <property name="acquireIncrement" value="#{datasource.acquireIncrement}" />
                <property name="testQuery" value="#{datasource.testQuery}" />
                <property name="useTmJoin" value="#{datasource.useTmJoin}" />
                <property name="deferConnectionRelease" value="#{datasource.deferConnectionRelease}" />
                <property name="automaticEnlistingEnabled" value="#{datasource.automaticEnlistingEnabled}" />
                <property name="acquisitionTimeout" value="#{datasource.acquisitionTimeout}" />
                <property name="acquisitionInterval" value="#{datasource.acquisitionInterval}" />
                <property name="preparedStatementCacheSize" value="#{datasource.preparedStatementCacheSize}" />
                <property name="twoPcOrderingPosition" value="#{datasource.twoPcOrderingPosition}" />
                <property name="applyTransactionTimeout" value="#{datasource.applyTransactionTimeout}" />
                <property name="allowLocalTransactions" value="#{datasource.allowLocalTransactions}" />
                <property name="driverProperties">
                        <props>
                                <prop key="user">#{datasource.jdbc.username}</prop>
                                <prop key="password">#{datasource.jdbc.password}</prop>
                                <prop key="URL">#{datasource.jdbc.url}</prop>
                        </props>
                </property>
        </bean>


--- btm config

<bean class="org.springframework.web.context.support.ServletContextPropertyPlaceholderConfigurer">
            <property name="searchContextAttributes" value="false"/> 
            <property name="placeholderPrefix" value="#{"/> 
            <property name="ignoreResourceNotFound" value="true"/>
        </bean>
       
       
        <bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices">
                <property name="serverId" value="#{bitronix.serverId}" />
                <property name="logPart1Filename" value="${catalina.base}/logs/btm1.tlog" />
                <property name="logPart2Filename" value="${catalina.base}/logs/btm2.tlog" />
                <property name="defaultTransactionTimeout" value="60" />
                <property name="gracefulShutdownInterval" value="60" />
                <property name="backgroundRecoveryIntervalSeconds" value="#{bitronix.tm.timer.backgroundRecoveryIntervalSeconds}" />
        </bean>

       
        <bean id="BitronixTransactionManager" factory-method="getTransactionManager" class="bitronix.tm.TransactionManagerServices"
                depends-on="btmConfig,dataSource" destroy-method="shutdown" />


       
        <bean id="JtaTransactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
                <property name="transactionManager" ref="BitronixTransactionManager" />
                <property name="userTransaction" ref="BitronixTransactionManager" />
        </bean>

        <tx:annotation-driven transaction-manager="transactionManager" />

Just for your information we use programatic transaction when handle JMS+datasoure

for datasource only we use declarative transaction with spring.
we share the same pool for both, may be this is an issue.



Thanks for help,


Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

snicoll
Well, I guess having the values for your tokens might help here :)

On Thu, Sep 23, 2010 at 9:20 AM, lesage <[hidden email]> wrote:

datasource config

<bean id="dataSource" class="bitronix.tm.resource.jdbc.PoolingDataSource"
               init-method="init" destroy-method="close">
               <!--   <property name="className"
value="#{datasource.jdbc.driverClassName}" /> -->



               <property name="uniqueName" value="#{datasource.uniqueName}" />

               <property name="className" value="#{datasource.jdbc.driverClassName}" />
               <property name="minPoolSize" value="#{datasource.minPoolSize}" />
               <property name="maxPoolSize" value="#{datasource.maxPoolSize}" />
               <property name="acquireIncrement" value="#{datasource.acquireIncrement}"
/>
               <property name="testQuery" value="#{datasource.testQuery}" />
               <property name="useTmJoin" value="#{datasource.useTmJoin}" />
               <property name="deferConnectionRelease"
value="#{datasource.deferConnectionRelease}" />
               <property name="automaticEnlistingEnabled"
value="#{datasource.automaticEnlistingEnabled}" />
               <property name="acquisitionTimeout"
value="#{datasource.acquisitionTimeout}" />
               <property name="acquisitionInterval"
value="#{datasource.acquisitionInterval}" />
               <property name="preparedStatementCacheSize"
value="#{datasource.preparedStatementCacheSize}" />
               <property name="twoPcOrderingPosition"
value="#{datasource.twoPcOrderingPosition}" />
               <property name="applyTransactionTimeout"
value="#{datasource.applyTransactionTimeout}" />
               <property name="allowLocalTransactions"
value="#{datasource.allowLocalTransactions}" />
               <property name="driverProperties">
                       <props>
                               <prop key="user">#{datasource.jdbc.username}</prop>
                               <prop key="password">#{datasource.jdbc.password}</prop>
                               <prop key="URL">#{datasource.jdbc.url}</prop>
                       </props>
               </property>
       </bean>


--- btm config

<bean
class="org.springframework.web.context.support.ServletContextPropertyPlaceholderConfigurer">
           <property name="searchContextAttributes" value="false"/>
           <property name="placeholderPrefix" value="#{"/>
           <property name="ignoreResourceNotFound" value="true"/>
       </bean>

       <!--  Bitronix Transaction Manager embedded configuration -->
       <bean id="btmConfig" factory-method="getConfiguration"
class="bitronix.tm.TransactionManagerServices">
               <property name="serverId" value="#{bitronix.serverId}" />
               <property name="logPart1Filename" value="${catalina.base}/logs/btm1.tlog"
/>
               <property name="logPart2Filename" value="${catalina.base}/logs/btm2.tlog"
/>
               <property name="defaultTransactionTimeout" value="60" />
               <property name="gracefulShutdownInterval" value="60" />
               <property name="backgroundRecoveryIntervalSeconds"
value="#{bitronix.tm.timer.backgroundRecoveryIntervalSeconds}" />
       </bean>

       <!-- create BTM transaction manager -->
       <bean id="BitronixTransactionManager"
factory-method="getTransactionManager"
class="bitronix.tm.TransactionManagerServices"
               depends-on="btmConfig,dataSource" destroy-method="shutdown" />


       <!-- Spring JtaTransactionManager -->
       <bean id="JtaTransactionManager"
class="org.springframework.transaction.jta.JtaTransactionManager">
               <property name="transactionManager" ref="BitronixTransactionManager" />
               <property name="userTransaction" ref="BitronixTransactionManager" />
       </bean>

       <tx:annotation-driven transaction-manager="transactionManager" />

Just for your information we use programatic transaction when handle
JMS+datasoure

for datasource only we use declarative transaction with spring.
we share the same pool for both, may be this is an issue.



Thanks for help,



--
View this message in context: http://old.nabble.com/bitronix-scheduler-blocks-XAPool-tp29786534p29786685.html
Sent from the Bitronix Transaction Manager mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage
This is context.xml file

        <Parameter name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect" override="false" /> 
        <Parameter name="datasource.jdbc.driverClassName" value="oracle.jdbc.xa.client.OracleXADataSource" override="false" />
        <Parameter name="datasource.jdbc.url" value="jdbc:oracle:thin:@amau0298:1521:orcl" override="false" />
        <Parameter name="datasource.jdbc.username" value="portailman" override="false" />
        <Parameter name="datasource.jdbc.password" value="admin" override="false" />
       
       
        <Parameter name="datasource.uniqueName" value="oracle" override="false" />
        <Parameter name="datasource.minPoolSize" value="5" override="false" />
        <Parameter name="datasource.maxPoolSize" value="20" override="false" />
        <Parameter name="datasource.acquireIncrement" value="1" override="false" />
        <Parameter name="datasource.allowLocalTransactions" value="true" override="false" />
        <Parameter name="datasource.testQuery" value="SELECT 1 FROM DUAL" override="false" />
        <Parameter name="datasource.useTmJoin" value="true" override="false" />
        <Parameter name="datasource.deferConnectionRelease" value="true" override="false" />
        <Parameter name="datasource.automaticEnlistingEnabled" value="true" override="false" />
        <Parameter name="datasource.acquisitionTimeout" value="5" override="false" />
        <Parameter name="datasource.acquisitionInterval" value="1" override="false" />
        <Parameter name="datasource.preparedStatementCacheSize" value="5" override="false" />
        <Parameter name="datasource.twoPcOrderingPosition" value="5" override="0" />       
        <Parameter name="datasource.acquisitionInterval" value="1" override="false" />
        <Parameter name="datasource.applyTransactionTimeout" value="true" override="false" />
Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage
Hi,

Thread dump is sent in the original post message.

Best regards,



Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

snicoll
A *full* threadS dump, especially to see if the JVM detects a deadlock.

On Thu, Sep 23, 2010 at 11:12 AM, lesage <[hidden email]> wrote:

Hi,

Thread dump is sent in the original post message.

Best regards,




--
View this message in context: http://old.nabble.com/bitronix-scheduler-blocks-XAPool-tp29786534p29787357.html
Sent from the Bitronix Transaction Manager mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage
Hi Here it is

Full thread dump Java HotSpot(TM) Server VM (14.3-b01 mixed mode):

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@aaa6e3" daemon prio=10 tid=0x09302000 nid=0x7a91 waiting on condition [0x9995c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa74c62d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@200c2d" daemon prio=10 tid=0x094afc00 nid=0x7a90 waiting on condition [0x989ba000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa74c62d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:9999" prio=10 tid=0x09303c00 nid=0x7913 runnable [0x98a5c000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:9999" prio=10 tid=0x09303000 nid=0x790e runnable [0x99a69000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"bitronix-recovery-thread" daemon prio=10 tid=0x9a8bf000 nid=0x7908 waiting for monitor entry [0x9b1ad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:72)
        - waiting to lock <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:204)
        at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:234)
        at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:204)
        at bitronix.tm.recovery.Recoverer.run(Recoverer.java:119)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(113)-127.0.0.1" daemon prio=10 tid=0x09301000 nid=0x7420 runnable [0x98aad000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0xa9e0db90> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"MasterPricer search on office ID :NITAF0100" daemon prio=10 tid=0x9a23e000 nid=0x7289 waiting for monitor entry [0x99869000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:124)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:178)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
        at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:558)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
        at fr.amadeuspro.webservices.sessionman.WsSessionFactory$$EnhancerByCGLIB$$fc6b87eb.makeObject(<generated>)
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
        at fr.amadeuspro.webservices.sessionman.WsSessionPool.borrowObject(WsSessionPool.java:86)
        at fr.amadeuspro.webservices.sessionman.WsSessionPool.getSession(WsSessionPool.java:156)
        at fr.amadeuspro.webservices.services.air.search.A1MasterPricerSearchService.execute(A1MasterPricerSearchService.java:41)
        at fr.amadeuspro.webservices.services.air.search.A1AirSearchServiceImpl.performSequenceSearch(A1AirSearchServiceImpl.java:245)
        at fr.amadeuspro.webservices.services.air.search.A1AirSearchServiceImpl.process(A1AirSearchServiceImpl.java:87)
        at fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl$MasterPricerSearchByNotificationThread.run(IMAAirSearchServiceImpl.java:511)

"MasterPricer search on office ID :NITA12100" daemon prio=10 tid=0x9a8fa400 nid=0x7288 runnable [0x9afad000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at oracle.net.ns.Packet.receive(Unknown Source)
        at oracle.net.ns.DataPacket.receive(Unknown Source)
        at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:219)
        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:813)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1049)
        at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:854)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1154)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)
        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3415)
        - locked <0xa9dc8858> (a oracle.jdbc.driver.T4CPreparedStatement)
        - locked <0xa48f34c0> (a oracle.jdbc.driver.T4CConnection)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.testConnection(JdbcPooledConnection.java:176)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.getConnectionHandle(JdbcPooledConnection.java:262)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:107)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:178)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
        at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:558)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
        at fr.amadeuspro.webservices.sessionman.WsSessionFactory$$EnhancerByCGLIB$$fc6b87eb.activateObject(<generated>)
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1196)
        at fr.amadeuspro.webservices.sessionman.WsSessionPool.borrowObject(WsSessionPool.java:86)
        at fr.amadeuspro.webservices.sessionman.WsSessionPool.getSession(WsSessionPool.java:156)
        at fr.amadeuspro.webservices.services.air.search.A1MasterPricerSearchService.execute(A1MasterPricerSearchService.java:41)
        at fr.amadeuspro.webservices.services.air.search.A1AirSearchServiceImpl.performSequenceSearch(A1AirSearchServiceImpl.java:245)
        at fr.amadeuspro.webservices.services.air.search.A1AirSearchServiceImpl.process(A1AirSearchServiceImpl.java:87)
        at fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl$MasterPricerSearchByNotificationThread.run(IMAAirSearchServiceImpl.java:511)

"ajp-10000-5" daemon prio=10 tid=0x08b3c800 nid=0x70e8 in Object.wait() [0x9affe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9d55f40> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1483)
        - locked <0xa9d55f40> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-4" daemon prio=10 tid=0x08aea400 nid=0x6f54 in Object.wait() [0x99aba000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9d48260> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1483)
        - locked <0xa9d48260> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-3" daemon prio=10 tid=0x0949c000 nid=0x6e7c in Object.wait() [0x9b1fd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9dee060> (a fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl$MasterPricerSearchByNotificationThread)
        at java.lang.Thread.join(Thread.java:1143)
        - locked <0xa9dee060> (a fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl$MasterPricerSearchByNotificationThread)
        at java.lang.Thread.join(Thread.java:1196)
        at fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl.processSearchMultiOid(IMAAirSearchServiceImpl.java:202)
        at fr.ima.bta.gds.services.air.search.IMAAirSearchServiceImpl.processRefreshSearchMultiOid(IMAAirSearchServiceImpl.java:571)
        at fr.ima.bta.ui.server.services.GwtGDSServiceImpl.processAirRefreshSearchMultiOid(GwtGDSServiceImpl.java:318)
        at fr.ima.bta.ui.server.services.GwtGDSServiceImpl$$FastClassByCGLIB$$9685b436.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
        at fr.ima.bta.ui.server.services.GwtGDSServiceImpl$$EnhancerByCGLIB$$8dfc2cd0.processAirRefreshSearchMultiOid(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:527)
        at fr.ima.bta.ui.server.GwtRpcController.processCall(GwtRpcController.java:48)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:86)
        at fr.ima.bta.ui.server.GwtRpcController.handleRequest(GwtRpcController.java:34)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
        at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
        at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.SessionFixationProtectionFilter.doFilterHttp(SessionFixationProtectionFilter.java:67)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.rememberme.RememberMeProcessingFilter.doFilterHttp(RememberMeProcessingFilter.java:116)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:277)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:427)
        at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:9999" prio=10 tid=0x9aca8800 nid=0x5e12 runnable [0x9b15c000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-2" daemon prio=10 tid=0x08a20000 nid=0x471e waiting for monitor entry [0x98afe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at bitronix.tm.resource.common.XAPool.stateChanged(XAPool.java:180)
        - waiting to lock <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.AbstractXAStatefulHolder.fireStateChanged(AbstractXAStatefulHolder.java:70)
        at bitronix.tm.resource.common.AbstractXAStatefulHolder.setState(AbstractXAStatefulHolder.java:41)
        at bitronix.tm.resource.common.TransactionContextHelper.requeue(TransactionContextHelper.java:123)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.release(JdbcPooledConnection.java:201)
        at bitronix.tm.resource.jdbc.JdbcConnectionHandle.close(JdbcConnectionHandle.java:83)
        at sun.reflect.GeneratedMethodAccessor332.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at bitronix.tm.resource.jdbc.BaseProxyHandlerClass.invoke(BaseProxyHandlerClass.java:44)
        at $Proxy11.close(Unknown Source)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:96)
        at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:474)
        at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:408)
        at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:347)
        at org.hibernate.impl.SessionImpl.close(SessionImpl.java:325)
        at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
        at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:733)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:989)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:782)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
        at fr.ima.bta.security.PortailBtaUserDetailsService$$EnhancerByCGLIB$$a72ad434.loadUserByUsername(<generated>)
        at org.springframework.security.providers.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:83)
        at org.springframework.security.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:121)
        at org.springframework.security.providers.ProviderManager.doAuthentication(ProviderManager.java:188)
        at org.springframework.security.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:46)
        at org.springframework.security.ui.webapp.AuthenticationProcessingFilter.attemptAuthentication(AuthenticationProcessingFilter.java:82)
        at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:258)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
        at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
        at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
        at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:427)
        at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-1" daemon prio=10 tid=0x094a5400 nid=0x4717 in Object.wait() [0x9a5fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa978e1a0> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1483)
        - locked <0xa978e1a0> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(129)-127.0.0.1" daemon prio=10 tid=0x09187800 nid=0x4700 runnable [0x998ba000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0xb2c687c0> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(126)-127.0.0.1" daemon prio=10 tid=0x094e1c00 nid=0x46fd runnable [0x9a15c000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0xa9e0de78> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Connection(6)-127.0.0.1" daemon prio=10 tid=0x08afac00 nid=0x46fc in Object.wait() [0x9a9f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9762240> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
        - locked <0xa9762240> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
        at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
        at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
        at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
        at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

"JMX server connection timeout 47" daemon prio=10 tid=0x08afa400 nid=0x46f3 in Object.wait() [0x9925c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9765ab0> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
        - locked <0xa9765ab0> (a [I)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-Acceptor-0" daemon prio=10 tid=0x9a2a2800 nid=0x468d runnable [0x992ad000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.tomcat.jni.Socket.accept(Native Method)
        at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1128)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-CometPoller-0" daemon prio=10 tid=0x9a38dc00 nid=0x468c in Object.wait() [0x992fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9532380> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1285)
        - locked <0xa9532380> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Thread.run(Thread.java:619)

"ajp-10000-Poller-0" daemon prio=10 tid=0x9b452400 nid=0x468b runnable [0x9945c000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.tomcat.jni.Poll.poll(Native Method)
        at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1316)
        at java.lang.Thread.run(Thread.java:619)

"http-10002-Acceptor-0" daemon prio=10 tid=0x9a2f0800 nid=0x468a runnable [0x994ad000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.tomcat.jni.Socket.accept(Native Method)
        at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1128)
        at java.lang.Thread.run(Thread.java:619)

"http-10002-Sendfile-0" daemon prio=10 tid=0x9b454c00 nid=0x4689 in Object.wait() [0x994fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa94b63a0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1743)
        - locked <0xa94b63a0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
        at java.lang.Thread.run(Thread.java:619)

"http-10002-CometPoller-0" daemon prio=10 tid=0x9b46f000 nid=0x4688 in Object.wait() [0x9b3ad000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa94b6360> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1285)
        - locked <0xa94b6360> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Thread.run(Thread.java:619)

"http-10002-Poller-0" daemon prio=10 tid=0x9accc400 nid=0x4687 in Object.wait() [0x9b3fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa94b6320> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1285)
        - locked <0xa94b6320> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
        at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x9b4fb400 nid=0x4686 waiting on condition [0x99b0b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
        at java.lang.Thread.run(Thread.java:619)

"jmsContainer-1" prio=10 tid=0x9a39d800 nid=0x4630 in Object.wait() [0x99b5c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa7b4bda0> (a java.lang.Object)
        at org.apache.activemq.MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77)
        - locked <0xa7b4bda0> (a java.lang.Object)
        at org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:412)
        at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:531)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:982)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:974)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
        at java.lang.Thread.run(Thread.java:619)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:9999" prio=10 tid=0x08b43000 nid=0x462f runnable [0x99bad000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
        at java.lang.Thread.run(Thread.java:619)

"Thread-8" daemon prio=10 tid=0x09231c00 nid=0x4625 waiting on condition [0x99bfe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
        at java.lang.Thread.run(Thread.java:619)

"JMX server connection timeout 27" daemon prio=10 tid=0x09230c00 nid=0x4624 in Object.wait() [0x9990b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa76d89d8> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
        - locked <0xa76d89d8> (a [I)
        at java.lang.Thread.run(Thread.java:619)

"RMI Scheduler(0)" daemon prio=10 tid=0x094bcc00 nid=0x4622 waiting on condition [0x999ad000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa43b3fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"RMI RenewClean-[127.0.0.1:44084]" daemon prio=10 tid=0x0946b000 nid=0x4621 in Object.wait() [0x999fe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa774a658> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xa774a658> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:619)

"RMI Reaper" prio=10 tid=0x08d4e400 nid=0x461f in Object.wait() [0x9a1ad000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4336b48> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xa4336b48> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x08a80400 nid=0x461e runnable [0x9a1fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xa76dea50> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-1099" daemon prio=10 tid=0x08a6a000 nid=0x461d runnable [0x9a55c000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xa75c68e0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"CommunicatorServer" prio=10 tid=0x08b01400 nid=0x461c runnable [0x9a5ad000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xa75c8358> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at com.sun.jdmk.comm.HtmlAdaptorServer.doReceive(HtmlAdaptorServer.java:1038)
        at com.sun.jdmk.comm.CommunicatorServer.run(CommunicatorServer.java:610)
        at java.lang.Thread.run(Thread.java:619)

"InactivityMonitor WriteCheck" daemon prio=10 tid=0x08a81c00 nid=0x4611 in Object.wait() [0x9a955000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa74c6768> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xa74c6768> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"InactivityMonitor ReadCheck" daemon prio=10 tid=0x08a80c00 nid=0x4610 in Object.wait() [0x9a9a6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa74c63d8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xa74c63d8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"bitronix-disk-force-batcher" daemon prio=10 tid=0x9a362000 nid=0x45f6 in Object.wait() [0x9ad54000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa9d88ce0> (a bitronix.tm.journal.DiskForceWaitQueue)
        at java.lang.Object.wait(Object.java:485)
        at bitronix.tm.journal.DiskForceWaitQueue.waitUntilNotEmpty(DiskForceWaitQueue.java:51)
        - locked <0xa9d88ce0> (a bitronix.tm.journal.DiskForceWaitQueue)
        at bitronix.tm.journal.DiskForceBatcherThread.runForceBatch(DiskForceBatcherThread.java:68)
        at bitronix.tm.journal.DiskForceBatcherThread.run(DiskForceBatcherThread.java:91)

"Timer-0" daemon prio=10 tid=0x9a2ff400 nid=0x45ee in Object.wait() [0x9adad000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa690c660> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xa690c660> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"bitronix-scheduler" daemon prio=10 tid=0x9a284000 nid=0x455a waiting for monitor entry [0x9adfe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at bitronix.tm.resource.common.XAPool.shrink(XAPool.java:211)
        - waiting to lock <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.timer.PoolShrinkingTask.execute(PoolShrinkingTask.java:28)
        at bitronix.tm.timer.TaskScheduler.executeElapsedTasks(TaskScheduler.java:198)
        at bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:174)

"GC Daemon" daemon prio=10 tid=0x9ac4d800 nid=0x453b in Object.wait() [0x9af50000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4575930> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0xa4575930> (a sun.misc.GC$LatencyLock)

"RMI TCP Accept-0" daemon prio=10 tid=0x9b855c00 nid=0x451a runnable [0x9b9be000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xa4456d88> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x089b9000 nid=0x4519 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x089b7000 nid=0x4518 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x089b4000 nid=0x4517 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x089b2800 nid=0x4516 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0899fc00 nid=0x4515 in Object.wait() [0x9bdce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa43a8388> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0xa43a8388> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0899e400 nid=0x4514 in Object.wait() [0x9be1f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa43a8368> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0xa43a8368> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0891e400 nid=0x450f runnable [0xb744d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xa96661e8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:647)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:607)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=10 tid=0x0899a400 nid=0x4513 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x08924c00 nid=0x4510 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x08926400 nid=0x4511 runnable

"VM Periodic Task Thread" prio=10 tid=0x9b85a800 nid=0x451b waiting on condition

JNI global references: 879

Heap
 PSYoungGen      total 23104K, used 15399K [0xb2670000, 0xb4050000, 0xb42e0000)
  eden space 19904K, 77% used [0xb2670000,0xb3579f68,0xb39e0000)
  from space 3200K, 0% used [0xb3d10000,0xb3d10000,0xb4030000)
  to   space 3264K, 0% used [0xb39e0000,0xb39e0000,0xb3d10000)
 PSOldGen        total 132480K, used 93460K [0xa42e0000, 0xac440000, 0xb2670000)
  object space 132480K, 70% used [0xa42e0000,0xa9e25160,0xac440000)
 PSPermGen       total 109056K, used 92377K [0x9c2e0000, 0xa2d60000, 0xa42e0000)
  object space 109056K, 84% used [0x9c2e0000,0xa1d16720,0xa2d60000)
Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

Ludovic Orban
Administrator
Hi,

Sorry for the long delay. Anyway I've had a closer look at your thread dump and apparently the XA pool is blocked by the 'MasterPricer search on office ID :NITA12100' thread while its trying to test a connection.

I don't know how long this thread has been stuck trying to test the connection when you took the stack trace -and if it really was at all- but what I can tell you is that only a problem on the network, in Oracle's JDBC driver or in the database itself can result in such behavior.

You could try adding a timeout yourself by patching BTM's connection testing code (that's in JdbcPooledConnection.java in method testConnection() around line 173) and change this:

        // Throws a SQLException if the connection is dead
        if (log.isDebugEnabled()) log.debug("testing with query '" + query + "' connection of " + this);
        PreparedStatement stmt = connection.prepareStatement(query);
        ResultSet rs = stmt.executeQuery();
        rs.close();
        stmt.close();
        if (log.isDebugEnabled()) log.debug("testQuery successfully tested connection of " + this);

into that:

        // Throws a SQLException if the connection is dead
        if (log.isDebugEnabled()) log.debug("testing with query '" + query + "' connection of " + this);
        PreparedStatement stmt = connection.prepareStatement(query);
        stmt.setQueryTimeout(1);
        ResultSet rs = stmt.executeQuery();
        rs.close();
        stmt.close();
        if (log.isDebugEnabled()) log.debug("testQuery successfully tested connection of " + this);


Another alternative would be to try out Oracle's JDBC 4 driver as the new standard JDBC4 isValid() testing mechanism is used instead.

It would be interesting to find out what exactly is causing this problem and if any of the two alternatives can solve it. I also wonder if a new connection testing timeout property shouldn't make its way in the connection pools properties.

If your thread dump does not lie the problem is around the connection testing facility as for some reason some connections appears to be dead and unresponsive. This seems to be the area to investigate in depth.

Keep us posted.
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
In reply to this post by lesage
Hi Ludovic

You rescue us

I don't know what exactly the issue

but application hangs very often

I capture all thread dump for each hang et I remark that Oracle driver hangs when socket exception occurs


 - locked <0xa9dc8858> (a oracle.jdbc.driver.T4CPreparedStatement)
        - locked <0xa48f34c0> (a oracle.jdbc.driver.T4CConnection)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.testConnection(JdbcPooledConnection.java:176)
        at bitronix.tm.resource.jdbc.JdbcPooledConnection.getConnectionHandle(JdbcPooledConnection.java:262)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:107)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)
        at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:68)
        - locked <0xa48f3198> (a bitronix.tm.resource.common.XAPool)



I have tried the -Doracle.jdbc.ReadTimeout=10000 as well but it does not solve my issue.
I have made google search on this oracle error and there is a lot a post on it.

Other idea on this ?

Thanks in advance ,

Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

Ludovic Orban-2
What exact driver version are you using? A different version may help as well as trying to patch BTM yourself as I suggested.

If the issue is urgent you may consider buying some support hours.
Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage
Hi Ludovic,

I am using thin jdbc driver XA datasource
version 10.2.0.4 the latest for Oracle 10.2g

I will check with my manager.

Thanks in advance,


Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

Ludovic Orban-2
I would also try out the latest 11.x.x.x driver as well, it should be compatible with the 10g server.

It's probably wise to involve Oracle in the loop as I'm quite sure the problem lies somewhere in between the JDBC driver and the DB. I might be able to code a workaround but the real problem would still lie somewhere.

Reply | Threaded
Open this post in threaded view
|

Re: bitronix-scheduler blocks XAPool

lesage

Hi Ludovic

You are right,
After changering Oracle JDBC drivers to version 11.2 for JAVA6, deadlock seems to disappear.

We survey this.

Thanks in advance