BTM and clustering

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

BTM and clustering

Arnaud Cogoluegnes
Hi all,

has anyone already run BTM on a cluster or on a load balanced configuration? what are the impacts on the datasource(s) provided by BTM? can the datasource(s) be configured separately (in each node) as well as the differents BTM instances?

any hint is welcome!

thanks

Arnaud
Reply | Threaded
Open this post in threaded view
|

Re: BTM and clustering

Ludovic Orban
Administrator
Hi,

The BTM datasources are completely cluster-safe. Unfortunately, the recovery engine isn't so in case of a crash, the cluster node that crashed must be the one performing recovery.

This isn't guaranteed for now so you have to be very careful that if a node crash, it must be restarted before you can restart any other node. If you don't, you might get heuristic errors during recovery !

It would be a great addition to fully support clustering and shouldn't be hard to do so I'd suggest you to open a new issue in JIRA requesting this enhancement and schedule it for version 1.3.1.

Thanks,
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: BTM and clustering

Arnaud Cogoluegnes
thanks for your reply. Do you mean that if a node crashes during a transaction, it must be manually restarted without other nodes being started between the crash and the reboot? could you please a provide an example?

Arnaud

PS : JIRA created
Reply | Threaded
Open this post in threaded view
|

Re: BTM and clustering

Ludovic Orban
Administrator
The problem is that only the cluster node that started a transaction knows if it should ultimately be committed or rolled back as this information is stored in the disk journal which is local per node and non-shareable.

Say you have two nodes in your cluster: N1 and N2. Let's pretend N1 crashes in the middle of a transaction, leaving TX1 in-doubt in the databases. And let's say the transaction has to be committed.

Case 1: leave N2 running and restart N1
Upon restart of N1, BTM will start the recovery process: it will look at all pending transactions in the databases and look for TX1's ID in its disk journal. The journal will contain an entry stating if the transactions should be committed or rolled back which will be done by the recovery engine. N1 is back in business and TX1 will be committed, the databases are kept consistent.

Case 1: restart N2 before you restart N1
Now say that N1 crashes again. Before you restart it, you restart N2 for any reason (maybe it also crashed). During BTM restart on N2, it will find TX1 in the databases but not in its disk journal so it will consider it has to rollback TX1. Then N1 restarts, runs recovery and finds that a transaction in its disk journal is not present in the databases. It will report this as a heuristic outcome but the databases aren't consistent anymore !

I've commented the JIRA entry with the solution to this problem. It's not that hard to implement but you'll have to wait for 1.3 to be released before I can start working on it.
Reply | Threaded
Open this post in threaded view
|

Re: BTM and clustering

Arnaud Cogoluegnes
thanks for this clear example. Does that mean any node shutdown (ex.: for software upgrade) should be done without any transaction running? (before the recovery engine fully supports clustering!)
Reply | Threaded
Open this post in threaded view
|

Re: BTM and clustering

Ludovic Orban
Administrator
In short: yes.

BTM will help you a bit here: when you call BitronixTransactionManager.shutdown(), it will wait some time to allow the running transactions to finish their work.

See: http://btm.codehaus.org/api/current/bitronix/tm/BitronixTransactionManager.html#shutdown()