XAPool lock problem

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

XAPool lock problem

Lars Arvidson

Hi,

This is my first post to this list. I’m not a native English speaker so you will have to excuse my bad language.

I want to thank everyone involved for a really nice JTA implementation!

I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five postgresql servers). At the moment I have maxPoolSize=150 and minPoolSize=50.

I have started to notice periods (a few seconds up to half a minute) during which the tomcat CPU usage is close to zero and number of threads are spiking. When doing a full thread dump I can see that more or less all threads are blocking in the XAPool and there is a one thread running in XAPool.shrink().

Looking at the implementation if XAPool (2.1.3) I can see that most operations are synchronized on the XAPool object. In particular locks are held when connections to the database are established.

Shrink is also synchronized. It will close all connections that have been idle more than MaxIdleTime seconds, even if it takes the total number of pooled objects below MinPoolSize. It will then bring the number of connections back to MinPoolSize.

With my settings shrink may close up to 150 connections and then establish up to 50 new connections while keeping XAPool locked.

Is it intended to close all idle connections even if it brings the number of connections below MinPoolSize? On an idle server this will close and reestablish all connection every MaxIdleTime (default 60s?).

Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the problems I have are already fixed. Is the source for 2.2 available somewhere?

Is it possible to estimate when 2.2 will be released? Is there any plans to backport some of the improvements into 2.1.X?

I’m currently trying out a quick (and dirty) fix to alleviate the problems I have. In short I have changed shrink to only synchronize when manipulating “objects” list, not when closing connection or creating connections.

 

Regards

Lars Arvidson

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

Brett Wooldridge-2
Lars,

Unfortunately, it does not appear that all of the work on the 2.2 branch will be integrated into the Bitronix mainline -- XAPool being one of them.  My company and I developed many (most?) of the changes on the 2.2 branch and have been in production with them since last November.

If you would like to try it, we are maintaining a fork on Github which can be found here:


Brett


On Thu, Sep 6, 2012 at 6:35 PM, Lars Arvidson <[hidden email]> wrote:

Hi,

This is my first post to this list. I’m not a native English speaker so you will have to excuse my bad language.

I want to thank everyone involved for a really nice JTA implementation!

I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five postgresql servers). At the moment I have maxPoolSize=150 and minPoolSize=50.

I have started to notice periods (a few seconds up to half a minute) during which the tomcat CPU usage is close to zero and number of threads are spiking. When doing a full thread dump I can see that more or less all threads are blocking in the XAPool and there is a one thread running in XAPool.shrink().

Looking at the implementation if XAPool (2.1.3) I can see that most operations are synchronized on the XAPool object. In particular locks are held when connections to the database are established.

Shrink is also synchronized. It will close all connections that have been idle more than MaxIdleTime seconds, even if it takes the total number of pooled objects below MinPoolSize. It will then bring the number of connections back to MinPoolSize.

With my settings shrink may close up to 150 connections and then establish up to 50 new connections while keeping XAPool locked.

Is it intended to close all idle connections even if it brings the number of connections below MinPoolSize? On an idle server this will close and reestablish all connection every MaxIdleTime (default 60s?).

Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the problems I have are already fixed. Is the source for 2.2 available somewhere?

Is it possible to estimate when 2.2 will be released? Is there any plans to backport some of the improvements into 2.1.X?

I’m currently trying out a quick (and dirty) fix to alleviate the problems I have. In short I have changed shrink to only synchronize when manipulating “objects” list, not when closing connection or creating connections.

 

Regards

Lars Arvidson

 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

Ludovic Orban-2
Lars,

The 2.2 branch is an experimental one maintained by Brett with lots of
misc improvements like in the XAPool concurrency. As Brett said, it's
quite stable but it's not officially supported by Bitronix Software
and probably doesn't have as many deployments (and thus, as much
testing) as the mainline.

The changes that went in 2.2 will eventually be backported to the
mainline someday, but it's impossible to say when unless some sponsor
would drop in.

--
Ludovic


On Thu, Sep 6, 2012 at 3:55 PM, Brett Wooldridge
<[hidden email]> wrote:

> Lars,
>
> Unfortunately, it does not appear that all of the work on the 2.2 branch
> will be integrated into the Bitronix mainline -- XAPool being one of them.
> My company and I developed many (most?) of the changes on the 2.2 branch and
> have been in production with them since last November.
>
> If you would like to try it, we are maintaining a fork on Github which can
> be found here:
>
> https://github.com/brettwooldridge/bitronix-hp
>
> Brett
>
>
> On Thu, Sep 6, 2012 at 6:35 PM, Lars Arvidson <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> This is my first post to this list. I’m not a native English speaker so
>> you will have to excuse my bad language.
>>
>> I want to thank everyone involved for a really nice JTA implementation!
>>
>> I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five
>> postgresql servers). At the moment I have maxPoolSize=150 and
>> minPoolSize=50.
>>
>> I have started to notice periods (a few seconds up to half a minute)
>> during which the tomcat CPU usage is close to zero and number of threads are
>> spiking. When doing a full thread dump I can see that more or less all
>> threads are blocking in the XAPool and there is a one thread running in
>> XAPool.shrink().
>>
>> Looking at the implementation if XAPool (2.1.3) I can see that most
>> operations are synchronized on the XAPool object. In particular locks are
>> held when connections to the database are established.
>>
>> Shrink is also synchronized. It will close all connections that have been
>> idle more than MaxIdleTime seconds, even if it takes the total number of
>> pooled objects below MinPoolSize. It will then bring the number of
>> connections back to MinPoolSize.
>>
>> With my settings shrink may close up to 150 connections and then establish
>> up to 50 new connections while keeping XAPool locked.
>>
>> Is it intended to close all idle connections even if it brings the number
>> of connections below MinPoolSize? On an idle server this will close and
>> reestablish all connection every MaxIdleTime (default 60s?).
>>
>> Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the
>> problems I have are already fixed. Is the source for 2.2 available
>> somewhere?
>>
>> Is it possible to estimate when 2.2 will be released? Is there any plans
>> to backport some of the improvements into 2.1.X?
>>
>> I’m currently trying out a quick (and dirty) fix to alleviate the problems
>> I have. In short I have changed shrink to only synchronize when manipulating
>> “objects” list, not when closing connection or creating connections.
>>
>>
>>
>> Regards
>>
>> Lars Arvidson
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SV: XAPool lock problem

Lars Arvidson
Hi,

Thanks for fast replies!

I would prefer to stay with the 2.1 mainline version as if feels a bit safer. Unfortunately the current XAPool 2.1 implementation causes too much problems in my application. Looks like my quick and dirty fix is working (application have not stalled yet) but it's not a proper fix.
I could probably allocate some time to backport Bretts version of XAPool. Would such a patch be accepted into the 2.1?

/Lars

> Lars,
>
> The 2.2 branch is an experimental one maintained by Brett with lots of
> misc improvements like in the XAPool concurrency. As Brett said, it's
> quite stable but it's not officially supported by Bitronix Software
> and probably doesn't have as many deployments (and thus, as much
> testing) as the mainline.
>
> The changes that went in 2.2 will eventually be backported to the
> mainline someday, but it's impossible to say when unless some sponsor
> would drop in.
>
> --
> Ludovic
>
>
> On Thu, Sep 6, 2012 at 3:55 PM, Brett Wooldridge
> <[hidden email]> wrote:
>> Lars,
>>
>> Unfortunately, it does not appear that all of the work on the 2.2 branch
>> will be integrated into the Bitronix mainline -- XAPool being one of them.
>> My company and I developed many (most?) of the changes on the 2.2 branch and
>> have been in production with them since last November.
>>
>> If you would like to try it, we are maintaining a fork on Github which can
>> be found here:
>>
>> https://github.com/brettwooldridge/bitronix-hp
>>
>> Brett
>>
>>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

Ludovic Orban-2
Lars,

Your patch would definitely be accepted but I think we should discuss
it first. I have to admit that I haven't looked in-depth at what Brett
did so an explanation would help me get up to speed.

Thanks,
Ludovic

On Fri, Sep 7, 2012 at 8:39 AM, Lars Arvidson <[hidden email]> wrote:

> Hi,
>
> Thanks for fast replies!
>
> I would prefer to stay with the 2.1 mainline version as if feels a bit safer. Unfortunately the current XAPool 2.1 implementation causes too much problems in my application. Looks like my quick and dirty fix is working (application have not stalled yet) but it's not a proper fix.
> I could probably allocate some time to backport Bretts version of XAPool. Would such a patch be accepted into the 2.1?
>
> /Lars
>
>> Lars,
>>
>> The 2.2 branch is an experimental one maintained by Brett with lots of
>> misc improvements like in the XAPool concurrency. As Brett said, it's
>> quite stable but it's not officially supported by Bitronix Software
>> and probably doesn't have as many deployments (and thus, as much
>> testing) as the mainline.
>>
>> The changes that went in 2.2 will eventually be backported to the
>> mainline someday, but it's impossible to say when unless some sponsor
>> would drop in.
>>
>> --
>> Ludovic
>>
>>
>> On Thu, Sep 6, 2012 at 3:55 PM, Brett Wooldridge
>> <[hidden email]> wrote:
>>> Lars,
>>>
>>> Unfortunately, it does not appear that all of the work on the 2.2 branch
>>> will be integrated into the Bitronix mainline -- XAPool being one of them.
>>> My company and I developed many (most?) of the changes on the 2.2 branch and
>>> have been in production with them since last November.
>>>
>>> If you would like to try it, we are maintaining a fork on Github which can
>>> be found here:
>>>
>>> https://github.com/brettwooldridge/bitronix-hp
>>>
>>> Brett
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

Ludovic Orban-2
In reply to this post by Brett Wooldridge-2
Brett,

I'd be more than happy to integrate those changes into the mainline.

I've been thinking a bit about the project's future for the long term
and what slowed until now. Some of the reasons are technical, like for
instance the support for JDBC 3 and JDK 5 which could be dropped, as
well as the .zip distribution which probably is useless nowadays. I
also believe the choice of the LGPL license was a mistake, and I
should consider moving to the Apache2 licence.

I want to keep control of the code base unless someone else jumps in
and takes this position, I wouldn't mind at all quite the contrary
since my time has been too limited for too long and I certainly am the
most important cause of BTM's development stall. The only condition
would be to bump the version to 3.x.x as I would really like to make
it obvious that major changes went in.

In short, I'm open to any proposition that could make BTM more active
even if that means I have to step back now that I don't make a living
of it anymore.

Thanks,
Ludovic

On Thu, Sep 6, 2012 at 3:55 PM, Brett Wooldridge
<[hidden email]> wrote:

> Lars,
>
> Unfortunately, it does not appear that all of the work on the 2.2 branch
> will be integrated into the Bitronix mainline -- XAPool being one of them.
> My company and I developed many (most?) of the changes on the 2.2 branch and
> have been in production with them since last November.
>
> If you would like to try it, we are maintaining a fork on Github which can
> be found here:
>
> https://github.com/brettwooldridge/bitronix-hp
>
> Brett
>
>
> On Thu, Sep 6, 2012 at 6:35 PM, Lars Arvidson <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> This is my first post to this list. I’m not a native English speaker so
>> you will have to excuse my bad language.
>>
>> I want to thank everyone involved for a really nice JTA implementation!
>>
>> I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five
>> postgresql servers). At the moment I have maxPoolSize=150 and
>> minPoolSize=50.
>>
>> I have started to notice periods (a few seconds up to half a minute)
>> during which the tomcat CPU usage is close to zero and number of threads are
>> spiking. When doing a full thread dump I can see that more or less all
>> threads are blocking in the XAPool and there is a one thread running in
>> XAPool.shrink().
>>
>> Looking at the implementation if XAPool (2.1.3) I can see that most
>> operations are synchronized on the XAPool object. In particular locks are
>> held when connections to the database are established.
>>
>> Shrink is also synchronized. It will close all connections that have been
>> idle more than MaxIdleTime seconds, even if it takes the total number of
>> pooled objects below MinPoolSize. It will then bring the number of
>> connections back to MinPoolSize.
>>
>> With my settings shrink may close up to 150 connections and then establish
>> up to 50 new connections while keeping XAPool locked.
>>
>> Is it intended to close all idle connections even if it brings the number
>> of connections below MinPoolSize? On an idle server this will close and
>> reestablish all connection every MaxIdleTime (default 60s?).
>>
>> Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the
>> problems I have are already fixed. Is the source for 2.2 available
>> somewhere?
>>
>> Is it possible to estimate when 2.2 will be released? Is there any plans
>> to backport some of the improvements into 2.1.X?
>>
>> I’m currently trying out a quick (and dirty) fix to alleviate the problems
>> I have. In short I have changed shrink to only synchronize when manipulating
>> “objects” list, not when closing connection or creating connections.
>>
>>
>>
>> Regards
>>
>> Lars Arvidson
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SV: XAPool lock problem

Lars Arvidson
In reply to this post by Ludovic Orban-2
Hi,

There's a nice write-up about XAPool changes in http://docs.codehaus.org/display/BTM/BTM-2.2, scroll down to "Monitors (lock contention)" -> "BTM 2.2".

At a quick glance at the 2.2 implementation I noticed that a "MaxLifeTime" configuration parameter has been added (probably after the linked document was written) to control how long a connection survives.

/Lars

> Lars,
>
> Your patch would definitely be accepted but I think we should discuss
> it first. I have to admit that I haven't looked in-depth at what Brett
> did so an explanation would help me get up to speed.
>
> Thanks,
> Ludovic
>
> On Fri, Sep 7, 2012 at 8:39 AM, Lars Arvidson <[hidden email]> wrote:
>> Hi,
>>
>> Thanks for fast replies!
>>
>> I would prefer to stay with the 2.1 mainline version as if feels a bit safer. Unfortunately the current XAPool 2.1 implementation causes too much problems in my application. Looks like my quick and dirty fix is working (application have not stalled yet) but it's not a proper fix.
>> I could probably allocate some time to backport Bretts version of XAPool. Would such a patch be accepted into the 2.1?
>>
>> /Lars

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SV: XAPool lock problem

Lars Arvidson
In reply to this post by Brett Wooldridge-2

I ended up deploying the 2.2 version and it has worked well this far.

 

The only modification I made was to add a “toString()” to “JdbcUncachedPreparedStatementHandle”.

 

@Override

public String toString() {

return delegate.toString();

}

 

I use statement.toString() for some logging and it worked with BTM 2.1.X but when I switch to 2.2 it stopped deligating it to the wrapped statement.

Maybe you could consider adding it to 2.2 or was the change intentional?

 

Regards

Lars

 

Från: Brett Wooldridge [mailto:[hidden email]]
Skickat: den 6 september 2012 15:56
Till: [hidden email]
Ämne: Re: [btm-user] XAPool lock problem

 

Lars,

 

Unfortunately, it does not appear that all of the work on the 2.2 branch will be integrated into the Bitronix mainline -- XAPool being one of them.  My company and I developed many (most?) of the changes on the 2.2 branch and have been in production with them since last November.

 

If you would like to try it, we are maintaining a fork on Github which can be found here:

 

 

Brett

 

On Thu, Sep 6, 2012 at 6:35 PM, Lars Arvidson <[hidden email]> wrote:

Hi,

This is my first post to this list. I’m not a native English speaker so you will have to excuse my bad language.

I want to thank everyone involved for a really nice JTA implementation!

I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five postgresql servers). At the moment I have maxPoolSize=150 and minPoolSize=50.

I have started to notice periods (a few seconds up to half a minute) during which the tomcat CPU usage is close to zero and number of threads are spiking. When doing a full thread dump I can see that more or less all threads are blocking in the XAPool and there is a one thread running in XAPool.shrink().

Looking at the implementation if XAPool (2.1.3) I can see that most operations are synchronized on the XAPool object. In particular locks are held when connections to the database are established.

Shrink is also synchronized. It will close all connections that have been idle more than MaxIdleTime seconds, even if it takes the total number of pooled objects below MinPoolSize. It will then bring the number of connections back to MinPoolSize.

With my settings shrink may close up to 150 connections and then establish up to 50 new connections while keeping XAPool locked.

Is it intended to close all idle connections even if it brings the number of connections below MinPoolSize? On an idle server this will close and reestablish all connection every MaxIdleTime (default 60s?).

Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the problems I have are already fixed. Is the source for 2.2 available somewhere?

Is it possible to estimate when 2.2 will be released? Is there any plans to backport some of the improvements into 2.1.X?

I’m currently trying out a quick (and dirty) fix to alleviate the problems I have. In short I have changed shrink to only synchronize when manipulating “objects” list, not when closing connection or creating connections.

 

Regards

Lars Arvidson

 

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

Brett Wooldridge-2
Lars,

The removal of the toString() was unintentional.  The old code wrapped a dynamic proxy around everything, so calls like that were transparently passed through.  The 2.2 code uses explicit delegation (1:1 mapping) in the wrapper, and I forgot to add toString() because it was not part of the interface (Statement or PreparedStatement).  I'll add it to the 2.2 branch code.

Regards,
Brett

On Tue, Oct 2, 2012 at 11:46 PM, Lars Arvidson <[hidden email]> wrote:

I ended up deploying the 2.2 version and it has worked well this far.

 

The only modification I made was to add a “toString()” to “JdbcUncachedPreparedStatementHandle”.

 

@Override

public String toString() {

return delegate.toString();

}

 

I use statement.toString() for some logging and it worked with BTM 2.1.X but when I switch to 2.2 it stopped deligating it to the wrapped statement.

Maybe you could consider adding it to 2.2 or was the change intentional?

 

Regards

Lars

 

Från: Brett Wooldridge [mailto:[hidden email]]
Skickat: den 6 september 2012 15:56
Till: [hidden email]
Ämne: Re: [btm-user] XAPool lock problem

 

Lars,

 

Unfortunately, it does not appear that all of the work on the 2.2 branch will be integrated into the Bitronix mainline -- XAPool being one of them.  My company and I developed many (most?) of the changes on the 2.2 branch and have been in production with them since last November.

 

If you would like to try it, we are maintaining a fork on Github which can be found here:

 

 

Brett

 

On Thu, Sep 6, 2012 at 6:35 PM, Lars Arvidson <[hidden email]> wrote:

Hi,

This is my first post to this list. I’m not a native English speaker so you will have to excuse my bad language.

I want to thank everyone involved for a really nice JTA implementation!

I use BTM with Tomcat and Postgresql 9.1 (two tomcat servers and five postgresql servers). At the moment I have maxPoolSize=150 and minPoolSize=50.

I have started to notice periods (a few seconds up to half a minute) during which the tomcat CPU usage is close to zero and number of threads are spiking. When doing a full thread dump I can see that more or less all threads are blocking in the XAPool and there is a one thread running in XAPool.shrink().

Looking at the implementation if XAPool (2.1.3) I can see that most operations are synchronized on the XAPool object. In particular locks are held when connections to the database are established.

Shrink is also synchronized. It will close all connections that have been idle more than MaxIdleTime seconds, even if it takes the total number of pooled objects below MinPoolSize. It will then bring the number of connections back to MinPoolSize.

With my settings shrink may close up to 150 connections and then establish up to 50 new connections while keeping XAPool locked.

Is it intended to close all idle connections even if it brings the number of connections below MinPoolSize? On an idle server this will close and reestablish all connection every MaxIdleTime (default 60s?).

Looking at http://docs.codehaus.org/display/BTM/BTM-2.2, it looks like the problems I have are already fixed. Is the source for 2.2 available somewhere?

Is it possible to estimate when 2.2 will be released? Is there any plans to backport some of the improvements into 2.1.X?

I’m currently trying out a quick (and dirty) fix to alleviate the problems I have. In short I have changed shrink to only synchronize when manipulating “objects” list, not when closing connection or creating connections.

 

Regards

Lars Arvidson

 

 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SV: XAPool lock problem

Lars Arvidson

> I'll add it to the 2.2 branch code.

 

Perfect, thanks!

 

Regards

Lars

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: XAPool lock problem

jayantvaish5
This post has NOT been accepted by the mailing list yet.
In reply to this post by Ludovic Orban-2
Hi Ludovic,

Are there any plans releasing bitronix with the fix "It will close all connections that have been idle more than MaxIdleTime seconds, even if it takes the total number of pooled objects below MinPoolSize." ?

Also, what are the drawbacks if we increase MaxIdleTime?

Thanks,
Jayant
Loading...