SonicMQ connections closed

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

SonicMQ connections closed

Gérald Quintana
Hello,

Once upon a time on a production server (BTM 2.0.1+SonicMQ 7.6):
- 10:55:40 progress.message.client.ENetworkFailure: Unexpected
exception during connect during message sending
- 10:55:42 javax.jms.IllegalStateException: The connection is closed
- 10:55:51 javax.jms.IllegalStateException: The connection is closed
- 10:55:*: javax.jms.IllegalStateException: The connection is closed +
No message can be sent nor received
- 10:56:*: javax.jms.IllegalStateException: The connection is closed +
No message can be sent nor received
- 10:56:18 bitronix.tm.recovery.RecoveryException: error starting recovery
- 10:56:25 Successful message sending and receiving: Houra!
- 11:34:00 Message are received at a very slow rate (1 message/minute)
and queue is filling
- 11:42 Server (BTM+Application) is restarted
- 11:43 All pending messages have been consumed and processed

BTM Recovery trial seems to have solved a problem at 10:55:51: What happened?
Can I do something to avoid this problem? Do you have an explanation
for the JMS message consumption slow down?

Gérald

Here are the complete stack traces:

10:55:40,666 [http-8081-1] ERROR
org.springframework.jms.UncategorizedJmsException: Uncategorized
exception occured during JMS processing; nested exception is
javax.jms.JMSException: Unexpected exception during connect
[...]
Caused by: javax.jms.JMSException: Unexpected exception during connect
        at progress.message.jimpl.JMSExceptionUtil.createJMSException(JMSExceptionUtil.java:42)
        at progress.message.jimpl.Session.<init>(Session.java:556)
        at progress.message.jimpl.Session.<init>(Session.java:631)
        at progress.message.jimpl.xa.XASession.<init>(XASession.java:74)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:99)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:79)
        at bitronix.tm.resource.jms.DualSessionWrapper.createXASession(DualSessionWrapper.java:96)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:74)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:59)
        at bitronix.tm.resource.jms.DualSessionWrapper.createQueue(DualSessionWrapper.java:443)
        at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:105)
        at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:66)
        at org.springframework.jms.support.destination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:100)
        at org.springframework.jms.core.JmsTemplate.access$200(JmsTemplate.java:90)
        at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:550)
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:471)
        ... 72 more
Caused by: [154] progress.message.client.ENetworkFailure: Unexpected
exception during connect
        at progress.message.zclient.Connection.connectWithRecoveryOpt(Connection.java:1902)
        at progress.message.zclient.Connection.connect(Connection.java:1689)
        at progress.message.zclient.Connection.connect(Connection.java:2000)
        at progress.message.jimpl.Session.<init>(Session.java:488)
        ... 86 more
       
10:55:51,292 [http-8081-10] ERROR
[...]
Caused by: javax.jms.IllegalStateException: The connection is closed.
        at progress.message.jimpl.Connection.getJMSObjectClosedException(Connection.java:2094)
        at progress.message.jimpl.Connection.checkClosed(Connection.java:2080)
        at progress.message.jimpl.Connection.getClientID(Connection.java:971)
        at progress.message.jimpl.Session.<init>(Session.java:414)
        at progress.message.jimpl.Session.<init>(Session.java:631)
        at progress.message.jimpl.xa.XASession.<init>(XASession.java:74)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:99)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:79)
        at bitronix.tm.resource.jms.DualSessionWrapper.createXASession(DualSessionWrapper.java:96)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:74)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:59)
        at bitronix.tm.resource.jms.DualSessionWrapper.createQueue(DualSessionWrapper.java:443)
        at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:105)
        at org.springframework.jms.support.destination.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:66)
        at org.springframework.jms.support.destination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:100)
        at org.springframework.jms.core.JmsTemplate.access$200(JmsTemplate.java:90)
        at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:550)
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:471)
        ... 73 more

10:56:18 bitronix.tm.recovery.Recoverer recoverAllResources
ATTENTION: error running recovery on resource
'jms/etdXAConnectionFactory', resource marked as failed (background
recoverer will retry recovery)
bitronix.tm.recovery.RecoveryException: error starting recovery
        at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:137)
        at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:234)
        at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:204)
        at bitronix.tm.recovery.Recoverer.run(Recoverer.java:119)
        at java.lang.Thread.run(Thread.java:619)
Caused by: javax.jms.IllegalStateException: The connection is closed.
        at progress.message.jimpl.Connection.getJMSObjectClosedException(Connection.java:2094)
        at progress.message.jimpl.Connection.checkClosed(Connection.java:2080)
        at progress.message.jimpl.Connection.getClientID(Connection.java:971)
        at progress.message.jimpl.Session.<init>(Session.java:414)
        at progress.message.jimpl.Session.<init>(Session.java:631)
        at progress.message.jimpl.xa.XASession.<init>(XASession.java:74)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:99)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:79)
        at bitronix.tm.resource.jms.DualSessionWrapper.createXASession(DualSessionWrapper.java:96)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:68)
        at bitronix.tm.resource.jms.JmsPooledConnection.createRecoveryXAResourceHolder(JmsPooledConnection.java:63)
        at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:134)
        ... 4 more
30 sept. 2010 10:56:22 bitronix.tm.recovery.IncrementalRecoverer recover
INFO: incremental recovery committed 0 dangling transaction(s) and
rolled back 0 aborted transaction(s) on resource
[jms/etdXAConnectionFactory] (restricted to serverId
'tomcat-btm-nodeETD-A')

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Ludovic Orban-2
This looks to me like a SonicMQ problem, my guess is that the MQ server closed all connections which makes the client's pooled connections invalid. I suggest you to investigate if something went wrong on the SonicMQ server or on the network between your client and your MQ server.

The fact that BTM's incremental recovery solved the problem tends to indicate that dropping the pooled connections and creating fresh ones solves the problem. The current algorithm used in version 2.0.1 can be a bit slow refreshing the pool in certain situations, see http://jira.codehaus.org/browse/BTM-89. This improvement will make its way in the coming 2.0.2 release.
Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Gérald Quintana
Thanks for your answer.

We contacted Progress support for the problem.

Extra question: why, during a few minutes, BTM gave closed connections
to the application. Would it be possible to do JMS connection checking
before serving them?

Gérald

2010/9/30 Ludovic Orban <[hidden email]>:

> This looks to me like a SonicMQ problem, my guess is that the MQ server
> closed all connections which makes the client's pooled connections invalid.
> I suggest you to investigate if something went wrong on the SonicMQ server
> or on the network between your client and your MQ server.
>
> The fact that BTM's incremental recovery solved the problem tends to
> indicate that dropping the pooled connections and creating fresh ones solves
> the problem. The current algorithm used in version 2.0.1 can be a bit slow
> refreshing the pool in certain situations, see
> http://jira.codehaus.org/browse/BTM-89. This improvement will make its way
> in the coming 2.0.2 release.
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Ludovic Orban-2
That's a good question. I tried very many different things to test JMS connections in the past and I could not find any reliable way to do that: this should be the job of the javax.jms.ExceptionListener but when I tried it some years ago many implementations were just bogus and useless. Maybe the situation changed in the meantime and I should reevaluate that. I've added a JIRA for this task: BTM-93.

Fortunately most JMS implementations offer some kind of automated connection checking  like for instance:

 - Transparent reconnect (SwiftMQ - http://www.swiftmq.com/products/router/swiftlets/sys_jms/clients/remote/reconnect/index.html)
 - Connection Failover (ActiveMQ - http://activemq.apache.org/failover-transport-reference.html)
 - Dead connection detection (HornetQ - http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/connection-ttl.html#d0e3194)

I don't know if there is any such mechanism in SonicMQ but you could ask Progress support for a definitive answer.


2010/9/30 Gérald Quintana <[hidden email]>
Thanks for your answer.

We contacted Progress support for the problem.

Extra question: why, during a few minutes, BTM gave closed connections
to the application. Would it be possible to do JMS connection checking
before serving them?

Gérald


Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Gérald Quintana
Thanks for you help,

Gérald

2010/9/30 Ludovic Orban <[hidden email]>:

> That's a good question. I tried very many different things to test JMS
> connections in the past and I could not find any reliable way to do that:
> this should be the job of the javax.jms.ExceptionListener but when I tried
> it some years ago many implementations were just bogus and useless. Maybe
> the situation changed in the meantime and I should reevaluate that. I've
> added a JIRA for this task: BTM-93.
>
> Fortunately most JMS implementations offer some kind of automated connection
> checking  like for instance:
>
>  - Transparent reconnect (SwiftMQ -
> http://www.swiftmq.com/products/router/swiftlets/sys_jms/clients/remote/reconnect/index.html)
>  - Connection Failover (ActiveMQ -
> http://activemq.apache.org/failover-transport-reference.html)
>  - Dead connection detection (HornetQ -
> http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/connection-ttl.html#d0e3194)
>
> I don't know if there is any such mechanism in SonicMQ but you could ask
> Progress support for a definitive answer.
>
>
> 2010/9/30 Gérald Quintana <[hidden email]>
>>
>> Thanks for your answer.
>>
>> We contacted Progress support for the problem.
>>
>> Extra question: why, during a few minutes, BTM gave closed connections
>> to the application. Would it be possible to do JMS connection checking
>> before serving them?
>>
>> Gérald
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Gérald Quintana
Good morning,

Problem occurred again during the night, there were no activity at
that moment. The connection was lost immediately in the Recoverer this
time. After that we got reconnected, but messages were delivered at a
very slow rate

Should I apply the patch described in BTM-89: is there a chance it
improves things while I am waiting for Progress help?

Gérald

04:18:56 bitronix.tm.recovery.Recoverer recoverAllResources
ATTENTION: error running recovery on resource
'jms/etdXAConnectionFactory', resource marked as failed (background
recoverer will retry recovery)
bitronix.tm.recovery.RecoveryException: error starting recovery
        at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:137)
        at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:234)
        at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:204)
        at bitronix.tm.recovery.Recoverer.run(Recoverer.java:119)
        at java.lang.Thread.run(Thread.java:619)
Caused by: javax.jms.JMSException: Unexpected exception during connect
        at progress.message.jimpl.JMSExceptionUtil.createJMSException(JMSExceptionUtil.java:42)
        at progress.message.jimpl.Session.<init>(Session.java:556)
        at progress.message.jimpl.Session.<init>(Session.java:631)
        at progress.message.jimpl.xa.XASession.<init>(XASession.java:74)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:99)
        at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:79)
        at bitronix.tm.resource.jms.DualSessionWrapper.createXASession(DualSessionWrapper.java:96)
        at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:68)
        at bitronix.tm.resource.jms.JmsPooledConnection.createRecoveryXAResourceHolder(JmsPooledConnection.java:63)
        at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:134)
        ... 4 more
Caused by: [154] progress.message.client.ENetworkFailure: Unexpected
exception during connect
        at progress.message.zclient.Connection.connectWithRecoveryOpt(Connection.java:1902)
        at progress.message.zclient.Connection.connect(Connection.java:1689)
        at progress.message.zclient.Connection.connect(Connection.java:2000)
        at progress.message.jimpl.Session.<init>(Session.java:488)
        ... 12 more
04:19:38 bitronix.tm.recovery.IncrementalRecoverer recover
INFO: incremental recovery committed 0 dangling transaction(s) and
rolled back 0 aborted transaction(s) on resource
[jms/etdXAConnectionFactory] (restricted to serverId
'tomcat-btm-nodeETD-A')

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: SonicMQ connections closed

Ludovic Orban-2
It may help detecting dead connections faster but that very much depends on the SonicMQ client implementation and the way you configured your JMS pool.

I suspect that either SonicMQ or some network equipment (firewall?) killed the connections because of a lack of activity. Have you set the pingInterval property in your JMS pool's driverProperties? I have no experienc ewith SonicMQ but my guts are telling me this settings exactly is meant to circumvent the problem I'm describing.

My suggestion would be to enable this pingInterval setting with a value of 60s or so to prevent TCP connections drop. I would also suggest you to set your minimal pool size to 0 to make sure if you have no activity during the night (for instance) the connections get closed by BTM's pool and get re-created again when load starts appearing over again. This kind of setup is very much used with databases and works fine without drawbacks most of the time. I believe the same would be true for a JMS server.


I also have no idea why messages are delivered at a very low pace after you got reconnected. That must have something to do with SonicMQ again.


2010/10/1 Gérald Quintana <[hidden email]>
Good morning,

Problem occurred again during the night, there were no activity at
that moment. The connection was lost immediately in the Recoverer this
time. After that we got reconnected, but messages were delivered at a
very slow rate

Should I apply the patch described in BTM-89: is there a chance it
improves things while I am waiting for Progress help?

Gérald

04:18:56 bitronix.tm.recovery.Recoverer recoverAllResources
ATTENTION: error running recovery on resource
'jms/etdXAConnectionFactory', resource marked as failed (background
recoverer will retry recovery)
bitronix.tm.recovery.RecoveryException: error starting recovery
       at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:137)
       at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:234)
       at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:204)
       at bitronix.tm.recovery.Recoverer.run(Recoverer.java:119)
       at java.lang.Thread.run(Thread.java:619)
Caused by: javax.jms.JMSException: Unexpected exception during connect
       at progress.message.jimpl.JMSExceptionUtil.createJMSException(JMSExceptionUtil.java:42)
       at progress.message.jimpl.Session.<init>(Session.java:556)
       at progress.message.jimpl.Session.<init>(Session.java:631)
       at progress.message.jimpl.xa.XASession.<init>(XASession.java:74)
       at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:99)
       at progress.message.jimpl.xa.XAConnection.createXASession(XAConnection.java:79)
       at bitronix.tm.resource.jms.DualSessionWrapper.createXASession(DualSessionWrapper.java:96)
       at bitronix.tm.resource.jms.DualSessionWrapper.getSession(DualSessionWrapper.java:68)
       at bitronix.tm.resource.jms.JmsPooledConnection.createRecoveryXAResourceHolder(JmsPooledConnection.java:63)
       at bitronix.tm.resource.jms.PoolingConnectionFactory.startRecovery(PoolingConnectionFactory.java:134)
       ... 4 more
Caused by: [154] progress.message.client.ENetworkFailure: Unexpected
exception during connect
       at progress.message.zclient.Connection.connectWithRecoveryOpt(Connection.java:1902)
       at progress.message.zclient.Connection.connect(Connection.java:1689)
       at progress.message.zclient.Connection.connect(Connection.java:2000)
       at progress.message.jimpl.Session.<init>(Session.java:488)
       ... 12 more
04:19:38 bitronix.tm.recovery.IncrementalRecoverer recover
INFO: incremental recovery committed 0 dangling transaction(s) and
rolled back 0 aborted transaction(s) on resource
[jms/etdXAConnectionFactory] (restricted to serverId
'tomcat-btm-nodeETD-A')

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email