Bitronix Spring Startup When JMS Server Offline

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Bitronix Spring Startup When JMS Server Offline

greenbean
Bitronix Spring Startup When JMS Server Offline

Hello, is there a way to have the PoolingConnectionFactory not error out if JMS factory (ActiveMQ) cannot create a connection when the Spring context is created?  We create connections to remote ActiveMQ brokers that may be down at certain times.  We would like to have the software retry connections until successful.  We have a timeout of 5000 on the brokerURL to prevent the thread creating the Spring context from hanging until the connection is created.  Alternatively when the timeout occurs, Bitronix throws an exception, and the Spring context is not created correctly.

Attached is output showing the exception that occurs after a 5000 timeout is reached.  Is there a way to configure the PoolingConnectionFactory to start up correctly even if connections are not available to fill the pool at init time?  I tried a minPoolSize of zero, but that did not seem to make a difference.

    <bean id="jmsFactory" class="bitronix.tm.resource.jms.PoolingConnectionFactory" init-method="init"

        destroy-method="close">

        <property name="className" value="org.apache.activemq.ActiveMQXAConnectionFactory" />

        <property name="uniqueName" value="activeMQJmsFactory" />

        <property name="maxPoolSize" value="250" />

        <property name="twoPcOrderingPosition" value="-1" />

        <property name="allowLocalTransactions" value="false" />

        <property name="driverProperties">

            <props>

                <prop key="brokerURL">failover:(ssl://xxx.xxx.xxx.xxx:61618)?timeout=5000</prop>

            </props>

        </property>

    </bean>

<<exception.txt>>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

exception.txt (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bitronix Spring Startup When JMS Server Offline

Ludovic Orban-2
Have you tried removing init-method="init"?


2011/1/12 Urciolo, Kevin J (IS) <[hidden email]>

Hello, is there a way to have the PoolingConnectionFactory not error out if JMS factory (ActiveMQ) cannot create a connection when the Spring context is created?  We create connections to remote ActiveMQ brokers that may be down at certain times.  We would like to have the software retry connections until successful.  We have a timeout of 5000 on the brokerURL to prevent the thread creating the Spring context from hanging until the connection is created.  Alternatively when the timeout occurs, Bitronix throws an exception, and the Spring context is not created correctly.

Attached is output showing the exception that occurs after a 5000 timeout is reached.  Is there a way to configure the PoolingConnectionFactory to start up correctly even if connections are not available to fill the pool at init time?  I tried a minPoolSize of zero, but that did not seem to make a difference.

    <bean id="jmsFactory" class="bitronix.tm.resource.jms.PoolingConnectionFactory" init-method="init"

        destroy-method="close">

        <property name="className" value="org.apache.activemq.ActiveMQXAConnectionFactory" />

        <property name="uniqueName" value="activeMQJmsFactory" />

        <property name="maxPoolSize" value="250" />

        <property name="twoPcOrderingPosition" value="-1" />

        <property name="allowLocalTransactions" value="false" />

        <property name="driverProperties">

            <props>

                <prop key="brokerURL">failover:(ssl://xxx.xxx.xxx.xxx:61618)?timeout=5000</prop>

            </props>

        </property>

    </bean>

<<exception.txt>>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

greenbean
I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.





From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Wednesday, January 12, 2011 5:51 PM
To: [hidden email]
Subject: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

Have you tried removing init-method="init"?

2011/1/12 Urciolo, Kevin J (IS) <[hidden email]>
Hello, is there a way to have the PoolingConnectionFactory not error out if JMS factory (ActiveMQ) cannot create a connection when the Spring context is created?  We create connections to remote ActiveMQ brokers that may be down at certain times.  We would like to have the software retry connections until successful.  We have a timeout of 5000 on the brokerURL to prevent the thread creating the Spring context from hanging until the connection is created.  Alternatively when the timeout occurs, Bitronix throws an exception, and the Spring context is not created correctly.
Attached is output showing the exception that occurs after a 5000 timeout is reached.  Is there a way to configure the PoolingConnectionFactory to start up correctly even if connections are not available to fill the pool at init time?  I tried a minPoolSize of zero, but that did not seem to make a difference.
    <bean id="jmsFactory" class="bitronix.tm.resource.jms.PoolingConnectionFactory" init-method="init"
        destroy-method="close">
        <property name="className" value="org.apache.activemq.ActiveMQXAConnectionFactory" />
        <property name="uniqueName" value="activeMQJmsFactory" />
        <property name="maxPoolSize" value="250" />
        <property name="twoPcOrderingPosition" value="-1" />
        <property name="allowLocalTransactions" value="false" />
        <property name="driverProperties">
            <props>
                <prop key="brokerURL">failover:(ssl://xxx.xxx.xxx.xxx:61618)?timeout=5000</prop>
            </props>
        </property>
    </bean>
<<exception.txt>>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

error-after-activemq-restart.txt (3K) Download Attachment
error-during-activemq-down.txt (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

Ludovic Orban-2
The error you get while DMLC is retrying the connection is plain normal and expectable: the very first thing BTM does when it sees a resource for the very first time or after each failure is run incremental recovery on it to make sure it is sane. As long as the incremental recovery fails, the resource is considered unsafe / invalid / down and BTM doesn't even try get a connection from it. That's plain normal in your case as ActiveMQ is down, just catch the JMS exception and deal with it.

The other error is more worrying. Either BTM or ActiveMQ are messing up their internals. What happens is that when DMLC calls receive() the JMS connection's XAResource gets enlisted by the BTM pool into the current transaction which triggers the transaction to look back for the connection that created that XAResource... which it cannot find.

Can you send me a simple app which I could use to reproduce the problem?


2011/2/14 Urciolo, Kevin J (IS) <[hidden email]>
I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.
Reply | Threaded
Open this post in threaded view
|

RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

greenbean

I placed a tar gzip containing code to reproduce the issue here:  http://www.2shared.com/file/p5Ww0h08/testertar.html

 

 

I apologize for the size of gzip  (approximately 27 MBs).  I did not have enough time to eliminate all the superfluous JARS from my project.

 

Steps:

1.  gunzip tester.tar.gz

2.  tar xvf tester.tar

3.  java –jar tester.jar

 

Example output is attached to this email.

 

Thanks,

 

Kevin

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Tuesday, February 15, 2011 5:57 AM
To: [hidden email]
Subject: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

The error you get while DMLC is retrying the connection is plain normal and expectable: the very first thing BTM does when it sees a resource for the very first time or after each failure is run incremental recovery on it to make sure it is sane. As long as the incremental recovery fails, the resource is considered unsafe / invalid / down and BTM doesn't even try get a connection from it. That's plain normal in your case as ActiveMQ is down, just catch the JMS exception and deal with it.

The other error is more worrying. Either BTM or ActiveMQ are messing up their internals. What happens is that when DMLC calls receive() the JMS connection's XAResource gets enlisted by the BTM pool into the current transaction which triggers the transaction to look back for the connection that created that XAResource... which it cannot find.

Can you send me a simple app which I could use to reproduce the problem?

2011/2/14 Urciolo, Kevin J (IS) <[hidden email]>

I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

output.txt (82K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

greenbean
In reply to this post by Ludovic Orban-2

An updated file is located here:  http://www.sendspace.com/file/tgydby

 

I included the wrong spring context xml previously.  It was missing the DMLC bean.

 

Kevin Urciolo
Northrop Grumman Information Systems
703-803-4931

571-249-8761 (cell)

 

From: Urciolo, Kevin J (IS)
Sent: Tuesday, February 15, 2011 4:44 PM
To: [hidden email]
Subject: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

I placed a tar gzip containing code to reproduce the issue here:  http://www.2shared.com/file/p5Ww0h08/testertar.html

 

 

I apologize for the size of gzip  (approximately 27 MBs).  I did not have enough time to eliminate all the superfluous JARS from my project.

 

Steps:

1.  gunzip tester.tar.gz

2.  tar xvf tester.tar

3.  java –jar tester.jar

 

Example output is attached to this email.

 

Thanks,

 

Kevin

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Tuesday, February 15, 2011 5:57 AM
To: [hidden email]
Subject: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

The error you get while DMLC is retrying the connection is plain normal and expectable: the very first thing BTM does when it sees a resource for the very first time or after each failure is run incremental recovery on it to make sure it is sane. As long as the incremental recovery fails, the resource is considered unsafe / invalid / down and BTM doesn't even try get a connection from it. That's plain normal in your case as ActiveMQ is down, just catch the JMS exception and deal with it.

The other error is more worrying. Either BTM or ActiveMQ are messing up their internals. What happens is that when DMLC calls receive() the JMS connection's XAResource gets enlisted by the BTM pool into the current transaction which triggers the transaction to look back for the connection that created that XAResource... which it cannot find.

Can you send me a simple app which I could use to reproduce the problem?

2011/2/14 Urciolo, Kevin J (IS) <[hidden email]>

I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.

Reply | Threaded
Open this post in threaded view
|

Re: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

Ludovic Orban-2
That is due to a bug in BTM's pools: https://jira.codehaus.org/browse/BTM-101

I've deployed a fresh snapshot build containing all bug fixed until now at this location:
https://nexus.codehaus.org/content/repositories/snapshots/org/codehaus/btm/btm/2.1.1-SNAPSHOT/btm-2.1.1-20110216.014317-3.jar

Please try it and let me know if that helps.


2011/2/16 Urciolo, Kevin J (IS) <[hidden email]>

An updated file is located here:  http://www.sendspace.com/file/tgydby

 

I included the wrong spring context xml previously.  It was missing the DMLC bean.

 

Kevin Urciolo
Northrop Grumman Information Systems
703-803-4931

571-249-8761 (cell)

 

From: Urciolo, Kevin J (IS)
Sent: Tuesday, February 15, 2011 4:44 PM

Subject: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

I placed a tar gzip containing code to reproduce the issue here:  http://www.2shared.com/file/p5Ww0h08/testertar.html

 

 

I apologize for the size of gzip  (approximately 27 MBs).  I did not have enough time to eliminate all the superfluous JARS from my project.

 

Steps:

1.  gunzip tester.tar.gz

2.  tar xvf tester.tar

3.  java –jar tester.jar

 

Example output is attached to this email.

 

Thanks,

 

Kevin

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Tuesday, February 15, 2011 5:57 AM
To: [hidden email]
Subject: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

The error you get while DMLC is retrying the connection is plain normal and expectable: the very first thing BTM does when it sees a resource for the very first time or after each failure is run incremental recovery on it to make sure it is sane. As long as the incremental recovery fails, the resource is considered unsafe / invalid / down and BTM doesn't even try get a connection from it. That's plain normal in your case as ActiveMQ is down, just catch the JMS exception and deal with it.

The other error is more worrying. Either BTM or ActiveMQ are messing up their internals. What happens is that when DMLC calls receive() the JMS connection's XAResource gets enlisted by the BTM pool into the current transaction which triggers the transaction to look back for the connection that created that XAResource... which it cannot find.

Can you send me a simple app which I could use to reproduce the problem?

2011/2/14 Urciolo, Kevin J (IS) <[hidden email]>

I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.


Reply | Threaded
Open this post in threaded view
|

RE: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

greenbean

The reconnection logic works correctly now.

 

However, I am running in to a situation where my ActiveMQ connections are established to the wrong URL.  For instance, my JMS connection factory URL list is [ssl://W.X.Y.Z:61618] and after reconnection attempts, the list becomes [tcp://localhost:61616,ssl://W.X.Y.Z:61618].  Therefore, I get a connection to localhost when it was not in my original configuration.

 

My Spring context has several  different ActiveMQ connection factories, each surrounded by a bitronix.tm.resource.jms.PoolingConnectionFactory.  Is it possible something in Bitronix is mixing up the different PoolingConnectionFactory objects?  I see the incorrectly URL set at the point shown in the stack trace below.  I have not been able to trace down where the mix up is happening yet.  It is either in Bitronix or ActiveMQ.

 

Daemon Thread [bitronix-recovery-thread] (Suspended (modification of field array in CopyOnWriteArrayList)) 

                CopyOnWriteArrayList<E>.setArray(Object[]) line: 72    

                CopyOnWriteArrayList<E>.add(int, E) line: 422  

                OrderedFailoverTransport.add(boolean, URI[]) line: 613              

                OrderedFailoverTransportFactory.createTransport(URISupport$CompositeData) line: 66             

                OrderedFailoverTransportFactory.doConnect(URI) line: 37         

                OrderedFailoverTransportFactory(TransportFactory).doConnect(URI, Executor) line: 51               

                TransportFactory.connect(URI, Executor) line: 80            

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createTransport() line: 243          

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createActiveMQConnection(String, String) line: 258               

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createActiveMQConnection() line: 230  

                ActiveMQXAConnectionFactory.createXAConnection() line: 59 

                PoolingConnectionFactory.createPooledConnection(Object, ResourceBean) line: 229   

                XAPool.createPooledObject(Object) line: 291   

                XAPool.grow() line: 408

                XAPool.getInPool() line: 387      

                XAPool.getConnectionHandle(boolean) line: 129             

                PoolingConnectionFactory.startRecovery() line: 160       

                Recoverer.recover(XAResourceProducer) line: 253         

                Recoverer.recoverAllResources() line: 223          

                Recoverer.run() line: 138             

                Thread.run() line: 662   

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Tuesday, February 15, 2011 8:48 PM
To: [hidden email]
Subject: Re: [btm-user] RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

That is due to a bug in BTM's pools: https://jira.codehaus.org/browse/BTM-101

I've deployed a fresh snapshot build containing all bug fixed until now at this location:
https://nexus.codehaus.org/content/repositories/snapshots/org/codehaus/btm/btm/2.1.1-SNAPSHOT/btm-2.1.1-20110216.014317-3.jar

Please try it and let me know if that helps.

2011/2/16 Urciolo, Kevin J (IS) <[hidden email]>

An updated file is located here:  http://www.sendspace.com/file/tgydby

 

I included the wrong spring context xml previously.  It was missing the DMLC bean.

 

Kevin Urciolo
Northrop Grumman Information Systems
703-803-4931

571-249-8761 (cell)

 

From: Urciolo, Kevin J (IS)
Sent: Tuesday, February 15, 2011 4:44 PM

Subject: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

I placed a tar gzip containing code to reproduce the issue here:  http://www.2shared.com/file/p5Ww0h08/testertar.html

 

 

I apologize for the size of gzip  (approximately 27 MBs).  I did not have enough time to eliminate all the superfluous JARS from my project.

 

Steps:

1.  gunzip tester.tar.gz

2.  tar xvf tester.tar

3.  java –jar tester.jar

 

Example output is attached to this email.

 

Thanks,

 

Kevin

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Ludovic Orban
Sent: Tuesday, February 15, 2011 5:57 AM
To: [hidden email]
Subject: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

 

The error you get while DMLC is retrying the connection is plain normal and expectable: the very first thing BTM does when it sees a resource for the very first time or after each failure is run incremental recovery on it to make sure it is sane. As long as the incremental recovery fails, the resource is considered unsafe / invalid / down and BTM doesn't even try get a connection from it. That's plain normal in your case as ActiveMQ is down, just catch the JMS exception and deal with it.

The other error is more worrying. Either BTM or ActiveMQ are messing up their internals. What happens is that when DMLC calls receive() the JMS connection's XAResource gets enlisted by the BTM pool into the current transaction which triggers the transaction to look back for the connection that created that XAResource... which it cannot find.

Can you send me a simple app which I could use to reproduce the problem?

2011/2/14 Urciolo, Kevin J (IS) <[hidden email]>

I tried removing the init method.  This allowed the spring context to be created correctly without errors.  Also, the DMLC retried the connection as expected.  A connection was established after restarting the ActiveMQ message broker.  However, I get the error from Bitronix in the attached txt file named "error-after-activemq-restart.txt" stating the resource is unknown.  The other attached file shows the error that occurs while the DMLC is retrying the connection.

 

Reply | Threaded
Open this post in threaded view
|

Re: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

Ludovic Orban-2
This time I really doubt this problem could be caused by BTM but I can't tell with so little information.

If you can debug through this code I suggest you to have a look at the value returned by the ActiveMQXAConnectionFactory's instance getBrokerURL() method call and compare that to the value of the PoolingConnectionFactory's driverProperties Properties object under the key 'brokerURL'.

This should give you an idea of what's going on. If neither are wrong there's a 99% chance the bug is in ActiveMQ.


2011/2/18 Urciolo, Kevin J (IS) <[hidden email]>

The reconnection logic works correctly now.

 

However, I am running in to a situation where my ActiveMQ connections are established to the wrong URL.  For instance, my JMS connection factory URL list is [ssl://W.X.Y.Z:61618] and after reconnection attempts, the list becomes [tcp://localhost:61616,ssl://W.X.Y.Z:61618].  Therefore, I get a connection to localhost when it was not in my original configuration.

 

My Spring context has several  different ActiveMQ connection factories, each surrounded by a bitronix.tm.resource.jms.PoolingConnectionFactory.  Is it possible something in Bitronix is mixing up the different PoolingConnectionFactory objects?  I see the incorrectly URL set at the point shown in the stack trace below.  I have not been able to trace down where the mix up is happening yet.  It is either in Bitronix or ActiveMQ.

 

Daemon Thread [bitronix-recovery-thread] (Suspended (modification of field array in CopyOnWriteArrayList)) 

                CopyOnWriteArrayList<E>.setArray(Object[]) line: 72    

                CopyOnWriteArrayList<E>.add(int, E) line: 422  

                OrderedFailoverTransport.add(boolean, URI[]) line: 613              

                OrderedFailoverTransportFactory.createTransport(URISupport$CompositeData) line: 66             

                OrderedFailoverTransportFactory.doConnect(URI) line: 37         

                OrderedFailoverTransportFactory(TransportFactory).doConnect(URI, Executor) line: 51               

                TransportFactory.connect(URI, Executor) line: 80            

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createTransport() line: 243          

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createActiveMQConnection(String, String) line: 258               

                ActiveMQXAConnectionFactory(ActiveMQConnectionFactory).createActiveMQConnection() line: 230  

                ActiveMQXAConnectionFactory.createXAConnection() line: 59 

                PoolingConnectionFactory.createPooledConnection(Object, ResourceBean) line: 229   

                XAPool.createPooledObject(Object) line: 291   

                XAPool.grow() line: 408

                XAPool.getInPool() line: 387      

                XAPool.getConnectionHandle(boolean) line: 129             

                PoolingConnectionFactory.startRecovery() line: 160       

                Recoverer.recover(XAResourceProducer) line: 253         

                Recoverer.recoverAllResources() line: 223          

                Recoverer.run() line: 138             

                Thread.run() line: 662   

 

Reply | Threaded
Open this post in threaded view
|

Re: RE: EXT :Re: [btm-user] RE: EXTERNAL:Re: [btm-user] Bitronix Spring Startup When JMS Server Offline

Dennis Brakhane-2
On Fri, Feb 18, 2011 at 10:17 PM, Ludovic Orban <[hidden email]> wrote:
> This should give you an idea of what's going on. If neither are wrong
> there's a 99% chance the bug is in ActiveMQ.

From what I've (over)heard from people who used ActiveMQ, it seems to be
rather buggy, so it's likely to be the case.

Just my 2 cents

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email