Oracle JDBC driver is leaking T4CPreparedStatement objects
I've noticed a slow memory leak in our application since we migrated from UCP to BTM; Oracle's T4CConnection objects accumulate a linked list of T4CPreparedStatement objects.
The application has two pools of database connections, typically with 5 active connections in each. BTM's own prepared statement cache is configured with 10, and so I'm not expecting there to be more than 100 PreparedStatement objects in use within the JVM. However, a heap dump from my JVM shows over 1000 T4CPreparedStatements.
I am suspecting that the JDBC driver is building these lists internally to provide its own simple "statement caching" functionality. However, I haven't enabled any such thing explicitly, and all the default configuration values that I have tested show that the driver's caches should all be off anyway. My studies of the Oracle driver suggest that these lists should be closed when the underlying "physical" connection objects are closed, and left open when a "logical" connection is closed and returned to a connection pool. (Although I haven't configured any pool of Oracle connections other than BTM either!)
So it occurred to me that maybe BTM's JdbcPooledConnection.close() function could be leaking? For example, suppose connection.close() threw an SQLException - The xaConnection would then never be closed and the pool's "destroy" even would never be fired! In fact, this "destroy" event wouldn't be fired if xaConnection.close() were to throw an SQLException either.
I've been digging deeper into my JVM's heap dump and have discovered the following:
- The 1000 T4CPreparedStatement objects are attached to only 10 T4CConnection objects.
- There are 10 of BTM's LruStatementCache objects, each containing between 6 and 10 T4CPreparedStatement.
- The T4CPreparedStatements are considered to be "open".
- There are 10 bitronix.tm.resource.jdbc.JdbcPooledConnection objects, each wrapping an oracle.jdbc.driver.LogicalConnection object.
- There are 74 LruStatementCache$CacheKey objects, and every single one has [sql = null, resultSetType = TYPE_FORWARD_ONLY, resultSetConcurrency = CONCUR_READ_ONLY]. (HOW?!?)
I am still digging, but am becoming increasingly worried...
I've tried to test this theory, but am currently fighting a losing battle against a solid wall of Java proxies. But anyway...
It occurs to me that when LruStatementCache invokes a LruEvictionListener.onEviction(PreparedStatement) handler, what it actually wants to do is close the *real* PreparedStatement object, because this object has just been evicted from JdbcPooledConnection.statementsCache.
What is actually happening (in my case) is that onEviction() is invoking PreparedStatementJavaProxy.close() - via some unspecified $Proxy object. Not only that, but since PreparedStatementJavaProxy.pretendClose == true, this close() method then exits without doing anything. The end result is that an open PreparedStatement object is leaked, which is quite serious with Oracle because T4CPreparedStatements are rather "heavy" things.
I'm not sure what the proper fix is here. I haven't been able to drill through to the delegate PreparedStatement object yet, but maybe the eviction handler should be called with the delegate object in the first place?