BTM-2.1.1 Hangs

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

BTM-2.1.1 Hangs

greenbean
Does anyone know why BTM would hang at the following locations?  I have two threads hanging at the locations shown below.
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.findXAResourceHolder(ResourceRegistrar.java:105)
bitronix.tm.BitronixTransaction.enlistResource(BitronixTransaction.java:83)
bitronix.tm.resource.common.TransactionContextHelper.enlistInCurrentTransaction(TransactionContextHelper.java:70)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.enlistResource(JdbcConnectionHandle.java:84)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.prepareStatement(JdbcConnectionHandle.java:243)
sun.reflect.GeneratedMethodAccessor540.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
bitronix.tm.resource.jdbc.BaseProxyHandlerClass.invoke(BaseProxyHandlerClass.java:63)
$Proxy163.prepareStatement(Unknown Source)
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.register(ResourceRegistrar.java:70)
bitronix.tm.resource.jms.PoolingConnectionFactory.buildXAPool(PoolingConnectionFactory.java:121)
bitronix.tm.resource.jms.PoolingConnectionFactory.init(PoolingConnectionFactory.java:72)
   - locked bitronix.tm.resource.jms.PoolingConnectionFactory@5475745d
bitronix.tm.resource.jms.PoolingConnectionFactory.createConnection(PoolingConnectionFactory.java:135)
org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
org.springframework.jms.core.JmsTemplate.access$0(JmsTemplate.java:1)
org.springframework.jms.core.JmsTemplate$JmsTemplateResourceFactory.createConnection(JmsTemplate.java:1031)
org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:297)
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:453)
org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:612
 
 
Reply | Threaded
Open this post in threaded view
|

Re: BTM-2.1.1 Hangs

snicoll
Can you give a bit more details about your environment. Are you running with the latest version?

S.

On Tue, May 31, 2011 at 7:13 PM, Urciolo, Kevin J (IS) <[hidden email]> wrote:
Does anyone know why BTM would hang at the following locations?  I have two threads hanging at the locations shown below.
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.findXAResourceHolder(ResourceRegistrar.java:105)
bitronix.tm.BitronixTransaction.enlistResource(BitronixTransaction.java:83)
bitronix.tm.resource.common.TransactionContextHelper.enlistInCurrentTransaction(TransactionContextHelper.java:70)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.enlistResource(JdbcConnectionHandle.java:84)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.prepareStatement(JdbcConnectionHandle.java:243)
sun.reflect.GeneratedMethodAccessor540.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
bitronix.tm.resource.jdbc.BaseProxyHandlerClass.invoke(BaseProxyHandlerClass.java:63)
$Proxy163.prepareStatement(Unknown Source)
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.register(ResourceRegistrar.java:70)
bitronix.tm.resource.jms.PoolingConnectionFactory.buildXAPool(PoolingConnectionFactory.java:121)
bitronix.tm.resource.jms.PoolingConnectionFactory.init(PoolingConnectionFactory.java:72)
   - locked bitronix.tm.resource.jms.PoolingConnectionFactory@5475745d
bitronix.tm.resource.jms.PoolingConnectionFactory.createConnection(PoolingConnectionFactory.java:135)
org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
org.springframework.jms.core.JmsTemplate.access$0(JmsTemplate.java:1)
org.springframework.jms.core.JmsTemplate$JmsTemplateResourceFactory.createConnection(JmsTemplate.java:1031)
org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:297)
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:453)
org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:612
 
 

Reply | Threaded
Open this post in threaded view
|

Re: BTM-2.1.1 Hangs

Ludovic Orban-2
Can you send a full thread dump showing the deadlock?

2011/5/31 Stephane Nicoll <[hidden email]>
Can you give a bit more details about your environment. Are you running with the latest version?

S.


On Tue, May 31, 2011 at 7:13 PM, Urciolo, Kevin J (IS) <[hidden email]> wrote:
Does anyone know why BTM would hang at the following locations?  I have two threads hanging at the locations shown below.
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.findXAResourceHolder(ResourceRegistrar.java:105)
bitronix.tm.BitronixTransaction.enlistResource(BitronixTransaction.java:83)
bitronix.tm.resource.common.TransactionContextHelper.enlistInCurrentTransaction(TransactionContextHelper.java:70)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.enlistResource(JdbcConnectionHandle.java:84)
bitronix.tm.resource.jdbc.JdbcConnectionHandle.prepareStatement(JdbcConnectionHandle.java:243)
sun.reflect.GeneratedMethodAccessor540.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
bitronix.tm.resource.jdbc.BaseProxyHandlerClass.invoke(BaseProxyHandlerClass.java:63)
$Proxy163.prepareStatement(Unknown Source)
org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
------------------------------------------------------------------------------------------------------------------------------------------------------------
bitronix.tm.resource.ResourceRegistrar.register(ResourceRegistrar.java:70)
bitronix.tm.resource.jms.PoolingConnectionFactory.buildXAPool(PoolingConnectionFactory.java:121)
bitronix.tm.resource.jms.PoolingConnectionFactory.init(PoolingConnectionFactory.java:72)
   - locked bitronix.tm.resource.jms.PoolingConnectionFactory@5475745d
bitronix.tm.resource.jms.PoolingConnectionFactory.createConnection(PoolingConnectionFactory.java:135)
org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
org.springframework.jms.core.JmsTemplate.access$0(JmsTemplate.java:1)
org.springframework.jms.core.JmsTemplate$JmsTemplateResourceFactory.createConnection(JmsTemplate.java:1031)
org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:297)
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:453)
org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:612
 
 


Mul
Reply | Threaded
Open this post in threaded view
|

Re: BTM-2.1.1 Hangs

Mul
I am having the same issue.

The problem manifested after upgrading from 1.3.3 to 2.1.2.

The thread processing the request hangs when calling line 250 of bitronix.tm.resource.jdbc.PoolingDataSource:

    return (Connection) Proxy.newProxyInstance(ClassLoaderUtils.getClassLoader(), new Class[] { Connection.class }, connectionHandle);

The above call eventually finds its way down to java.lang.reflect.Proxy and issues a wait() on the cache Map object, and is never notified, so hangs indefinitely.

        synchronized (cache) {
            do {
                Object value = cache.get(key);
                if (value instanceof Reference) {
                    proxyClass = (Class) ((Reference) value).get();
                }
                if (proxyClass != null) {
                    // proxy class already generated: return it
                    return proxyClass;
                } else if (value == pendingGenerationMarker) {
                    // proxy class being generated: wait for it
                    try {
                        cache.wait();
                    } catch (InterruptedException e) {
                    }
                    continue;
                } else {
                    cache.put(key, pendingGenerationMarker);
                    break;
                }
            } while (true);
        }

I need to upgrade to Bitronix 2+ because we're hitting some of the issues fixed in the later releases, eg BTM-55,57,68.

I'm running Java version "1.6.0_29" on Mac OS X 10.6.8.

I see the same problem on Linux:

    Linux version 2.6.37.6-0.7-xen (geeko@buildhost) (gcc version 4.5.1 20101208 [gcc-4_5-branch revision 167585] (SUSE Linux) ) #1 SMP 2011-07-21 02:17:24 +0200
    Welcome to openSUSE 11.4 "Celadon" - Kernel \r (\l).

Stack trace at point of hang:

Thread [1981297331@qtp-1373806697-10]
        Proxy.getProxyClass(ClassLoader, Class<?>...) line: 436
        Proxy.newProxyInstance(ClassLoader, Class<?>[], InvocationHandler) line: 581
        PoolingDataSource.getConnection() line: 250
        LocalJtaDataSourceConnectionProvider(LocalDataSourceConnectionProvider).getConnection() line: 81
        ConnectionManager.openConnection() line: 446
        ConnectionManager.getConnection() line: 167
        BatchingBatcher(AbstractBatcher).prepareQueryStatement(String, boolean, ScrollMode) line: 161
        OneToManyLoader(Loader).prepareQueryStatement(QueryParameters, boolean, SessionImplementor) line: 1577
        OneToManyLoader(Loader).doQuery(SessionImplementor, QueryParameters, boolean) line: 696
        OneToManyLoader(Loader).doQueryAndInitializeNonLazyCollections(SessionImplementor, QueryParameters, boolean) line: 259
        OneToManyLoader(Loader).loadCollection(SessionImplementor, Serializable, Type) line: 2019
        OneToManyLoader(CollectionLoader).initialize(Serializable, SessionImplementor) line: 59
        OneToManyPersister(AbstractCollectionPersister).initialize(Serializable, SessionImplementor) line: 587
        DefaultInitializeCollectionEventListener.onInitializeCollection(InitializeCollectionEvent) line: 83
        SessionImpl.initializeCollection(PersistentCollection, boolean) line: 1744
        PersistentMap(AbstractPersistentCollection).initialize(boolean) line: 366
        PersistentMap(AbstractPersistentCollection).read() line: 108
        PersistentMap(AbstractPersistentCollection).readElementByIndex(Object) line: 183
        PersistentMap.get(Object) line: 169
        ProcessDefinition.getDefinition(Class) line: 431
        ProcessDefinition.getFileDefinition() line: 441
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
        Method.invoke(Object, Object...) line: 597
        JavassistLazyInitializer.invoke(Object, Method, Method, Object[]) line: 197
        ProcessDefinition_$$_javassist_137.getFileDefinition() line: not available
        ProcessClassLoader.findResource(String) line: 47
        ProcessClassLoader(ClassLoader).getResource(String) line: 1002
        ClassLoaderRepository.toURL(String) line: 253
        ClassLoaderRepository.loadClass(String) line: 266
        LTWWorld(BcelWorld).lookupJavaClass(ClassPathManager, String) line: 224
        LTWWorld(BcelWorld).resolveDelegate(ReferenceType) line: 205
        LTWWorld.resolveDelegate(ReferenceType) line: 107
        LTWWorld(World).resolveToReferenceType(UnresolvedType, boolean) line: 363
        LTWWorld(World).resolve(UnresolvedType, boolean) line: 258
        ExactAnnotationTypePattern.resolveBindings(IScope, Bindings, boolean) line: 306
        AnnotationPointcut.resolveBindings(IScope, Bindings) line: 182
        AndPointcut.resolveBindings(IScope, Bindings) line: 76
        AndPointcut(Pointcut).resolve(IScope) line: 196
        AtAjAttributes$LazyResolvedPointcutDefinition.getPointcut() line: 1693
        ReferencePointcut.concretize1(ResolvedType, ResolvedType, IntMap) line: 357
        ReferencePointcut(Pointcut).concretize(ResolvedType, ResolvedType, IntMap) line: 233
        ReferencePointcut(Pointcut).concretize(ResolvedType, ResolvedType, int, ShadowMunger) line: 220
        BcelAdvice(Advice).concretize(ResolvedType, World, PerClause) line: 336
        BcelAdvice.concretize(ResolvedType, World, PerClause) line: 108
        CrosscuttingMembers.addShadowMunger(ShadowMunger) line: 92
        CrosscuttingMembers.addShadowMungers(Collection) line: 85
        ReferenceType(ResolvedType).collectCrosscuttingMembers(boolean) line: 561
        CrosscuttingMembersSet.addOrReplaceAspect(ResolvedType, boolean) line: 75
        CrosscuttingMembersSet.addOrReplaceAspect(ResolvedType) line: 62
        BcelWeaver.addLibraryAspect(String) line: 200
        ClassLoaderWeavingAdaptor.registerAspects(BcelWeaver, ClassLoader, List) line: 432
        ClassLoaderWeavingAdaptor.registerDefinitions(BcelWeaver, ClassLoader, List) line: 267
        ClassLoaderWeavingAdaptor.initialize(ClassLoader, IWeavingContext) line: 159
        Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.initialize(ClassLoader, IWeavingContext) line: 269
        Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.getWeavingAdaptor(ClassLoader, IWeavingContext) line: 274
        Aj$WeaverContainer.getWeaver(ClassLoader, IWeavingContext) line: 252
        Aj.preProcess(String, byte[], ClassLoader) line: 89
        ClassPreProcessorAgentAdapter.transform(ClassLoader, String, Class<?>, ProtectionDomain, byte[]) line: 52
        TransformerManager.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 169
        InstrumentationImpl.transform(ClassLoader, String, Class, ProtectionDomain, byte[], boolean) line: 365
        Proxy.defineClass0(ClassLoader, String, byte[], int, int) line: not available [native method]
        Proxy.getProxyClass(ClassLoader, Class<?>...) line: 504
        Proxy.newProxyInstance(ClassLoader, Class<?>[], InvocationHandler) line: 581
        PoolingDataSource.getConnection() line: 250
        LocalJtaDataSourceConnectionProvider(LocalDataSourceConnectionProvider).getConnection() line: 81
        ConnectionManager.openConnection() line: 446
        ConnectionManager.getConnection() line: 167
        BatchingBatcher(AbstractBatcher).prepareQueryStatement(String, boolean, ScrollMode) line: 161
        BasicCollectionLoader(Loader).prepareQueryStatement(QueryParameters, boolean, SessionImplementor) line: 1577
        BasicCollectionLoader(Loader).doQuery(SessionImplementor, QueryParameters, boolean) line: 696
        BasicCollectionLoader(Loader).doQueryAndInitializeNonLazyCollections(SessionImplementor, QueryParameters, boolean) line: 259
        BasicCollectionLoader(Loader).loadCollection(SessionImplementor, Serializable, Type) line: 2019
        BasicCollectionLoader(CollectionLoader).initialize(Serializable, SessionImplementor) line: 59
        BasicCollectionPersister(AbstractCollectionPersister).initialize(Serializable, SessionImplementor) line: 587
        DefaultInitializeCollectionEventListener.onInitializeCollection(InitializeCollectionEvent) line: 83
        SessionImpl.initializeCollection(PersistentCollection, boolean) line: 1744
        PersistentList(AbstractPersistentCollection).initialize(boolean) line: 366
        PersistentList(AbstractPersistentCollection).read() line: 108
        PersistentList(AbstractPersistentCollection).readSize() line: 131
        PersistentList.isEmpty() line: 121
        Decision.execute(ExecutionContext) line: 127
        Decision(Node).enter(ExecutionContext) line: 390
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
        Method.invoke(Object, Object...) line: 597
        JavassistLazyInitializer.invoke(Object, Method, Method, Object[]) line: 197
        Node_$$_javassist_251.enter(ExecutionContext) line: not available
        Transition.take(ExecutionContext) line: 167
        TaskNode(Node).leave(ExecutionContext, Transition) line: 479
        TaskNode.leave(ExecutionContext, Transition) line: 213
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
        Method.invoke(Object, Object...) line: 597
        JavassistLazyInitializer.invoke(Object, Method, Method, Object[]) line: 197
        Node_$$_javassist_251.leave(ExecutionContext, Transition) line: not available
        Token.signal(Transition, ExecutionContext) line: 223
        Token.signal(Transition) line: 188
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
        Method.invoke(Object, Object...) line: 597
        JavassistLazyInitializer.invoke(Object, Method, Method, Object[]) line: 197
        Token_$$_javassist_139.signal(Transition) line: not available
        TaskInstance.end(Transition) line: 495
        TaskInstance.end(String) line: 436
        JbpmWorkflowTask.complete(String, String) line: 208
        JbpmCompleteTaskWorkflowInvocationStrategy.doInvocation(JbpmContext) line: 71
        AbstractJbpmInvoker.invokeWorkflow_aroundBody0(AbstractJbpmInvoker, JbpmWorkflowManager$WorkflowInvocationStrategy, TaskReference, JoinPoint) line: 51
        AbstractJbpmInvoker$AjcClosure1.run(Object[]) line: 1
        JoinPointImpl.proceed() line: 101
        ValidationAspect.invoke(ProceedingJoinPoint) line: 75
        JbpmTaskManager(AbstractJbpmInvoker).invokeWorkflow(WorkflowInvocationStrategy, TaskReference<Long>) line: 41
        JbpmTaskManager.completeTask_aroundBody0(JbpmTaskManager, TaskOperationDTO, TaskReference, JoinPoint) line: 263
        JbpmTaskManager$AjcClosure1.run(Object[]) line: 1
        JoinPointImpl.proceed() line: 101
        ValidationAspect.invoke(ProceedingJoinPoint) line: 75
        JbpmTaskManager.completeTask(TaskOperationDTO, TaskReference<Long>) line: 254
        JbpmTaskManager.completeTask(Long, TaskOperationDTO) line: 248
        JbpmWorkflowManager.completeTask(Long, TaskOperationDTO) line: 238
        JbpmWorkflowManager.completeTask(Object, TaskOperationDTO) line: 1
        WorkflowFacade.completeTask(Long, TaskOperationDTO) line: 226
        TaskCompleteExecutor.execute(Long, TaskOperationDTO, WorkflowFacade) line: 22
        WorkflowFacade.apply(TaskOperationDTO, Long) line: 101
        WorkflowFacade$$FastClassByCGLIB$$5dea281a.invoke(int, Object, Object[]) line: not available
        MethodProxy.invoke(Object, Object[]) line: 149
        Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint() line: 688
        Cglib2AopProxy$CglibMethodInvocation(ReflectiveMethodInvocation).proceed() line: 150
        TransactionInterceptor.invoke(MethodInvocation) line: 110
        Cglib2AopProxy$CglibMethodInvocation(ReflectiveMethodInvocation).proceed() line: 172
        Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Object, Method, Object[], MethodProxy) line: 621
        WorkflowFacade$$EnhancerByCGLIB$$9cfa1ff7.apply(TaskOperationDTO, Long) line: not available
        TaskDispatchingResource.doStore(Representation) line: 94
        TaskDispatchingResource(BaseResource).put(Representation) line: 257
        TaskDispatchingResource(BaseResource).put(Representation, Variant) line: 272
        TaskDispatchingResource(ServerResource).doHandle(Variant) line: 505
        TaskDispatchingResource(ServerResource).doNegotiatedHandle() line: 576
        TaskDispatchingResource(ServerResource).doConditionalHandle() line: 254
        TaskDispatchingResource(ServerResource).handle() line: 923
        MyCompanyFinder(Finder).handle(Request, Response) line: 510
        Route(Filter).doHandle(Request, Response) line: 156
        Route(Filter).handle(Request, Response) line: 201
        MyCompanyRouter(Router).doHandle(Restlet, Request, Response) line: 321
        MyCompanyRouter(Router).handle(Request, Response) line: 520
        Route(Filter).doHandle(Request, Response) line: 156
        Route(Filter).handle(Request, Response) line: 201
        MyCompanyRouter(Router).doHandle(Restlet, Request, Response) line: 321
        MyCompanyRouter(Router).handle(Request, Response) line: 520
        AuditManagementFilter(Filter).doHandle(Request, Response) line: 156
        AuditManagementFilter(Filter).handle(Request, Response) line: 201
        NoCacheFilter(Filter).doHandle(Request, Response) line: 156
        NoCacheFilter(Filter).handle(Request, Response) line: 201
        Encoder(Filter).doHandle(Request, Response) line: 156
        Encoder(Filter).handle(Request, Response) line: 201
        RangeFilter(Filter).doHandle(Request, Response) line: 156
        RangeFilter(Filter).handle(Request, Response) line: 201
        Decoder(Filter).doHandle(Request, Response) line: 156
        Decoder(Filter).handle(Request, Response) line: 201
        StatusFilter(Filter).doHandle(Request, Response) line: 156
        StatusFilter.doHandle(Request, Response) line: 152
        StatusFilter(Filter).handle(Request, Response) line: 201
        TunnelFilter(Filter).doHandle(Request, Response) line: 156
        TunnelFilter(Filter).handle(Request, Response) line: 201
        ApplicationHelper(ChainHelper<T>).handle(Request, Response) line: 111
        ApplicationHelper.handle(Request, Response) line: 72
        Application.handle(Request, Response) line: 344
        VirtualHost$1(Filter).doHandle(Request, Response) line: 156
        VirtualHost$1(Filter).handle(Request, Response) line: 201
        VirtualHost(Router).doHandle(Restlet, Request, Response) line: 321
        VirtualHost(Router).handle(Request, Response) line: 520
        HostRoute(Filter).doHandle(Request, Response) line: 156
        HostRoute(Filter).handle(Request, Response) line: 201
        ServerRouter(Router).doHandle(Restlet, Request, Response) line: 321
        ServerRouter(Router).handle(Request, Response) line: 520
        StatusFilter(Filter).doHandle(Request, Response) line: 156
        StatusFilter.doHandle(Request, Response) line: 152
        StatusFilter(Filter).handle(Request, Response) line: 201
        LogFilter(Filter).doHandle(Request, Response) line: 156
        LogFilter(Filter).handle(Request, Response) line: 201
        ComponentHelper(ChainHelper<T>).handle(Request, Response) line: 111
        SpringComponent(Component).handle(Request, Response) line: 403
        RangeFilter(Filter).doHandle(Request, Response) line: 156
        RangeFilter(Filter).handle(Request, Response) line: 201
        Decoder(Filter).doHandle(Request, Response) line: 156
        Decoder(Filter).handle(Request, Response) line: 201
        StatusFilter(Filter).doHandle(Request, Response) line: 156
        StatusFilter.doHandle(Request, Response) line: 152
        StatusFilter(Filter).handle(Request, Response) line: 201
        TunnelFilter(Filter).doHandle(Request, Response) line: 156
        TunnelFilter(Filter).handle(Request, Response) line: 201
        ApplicationHelper(ChainHelper<T>).handle(Request, Response) line: 111
        ApplicationHelper.handle(Request, Response) line: 72
        Application.handle(Request, Response) line: 344
        ServletAdapter.service_aroundBody0(ServletAdapter, HttpServletRequest, HttpServletResponse, JoinPoint) line: 201
        ServletAdapter$AjcClosure1.run(Object[]) line: 1
        JoinPointImpl.proceed() line: 101
        OpenSessionInViewAspect.invoke(ProceedingJoinPoint) line: 45
        ServletAdapter.service(HttpServletRequest, HttpServletResponse) line: 183
        RestletFrameworkServlet.doService(HttpServletRequest, HttpServletResponse) line: 121
        RestletFrameworkServlet(FrameworkServlet).processRequest(HttpServletRequest, HttpServletResponse) line: 644
        RestletFrameworkServlet(FrameworkServlet).doPut(HttpServletRequest, HttpServletResponse) line: 571
        RestletFrameworkServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 730
        RestletFrameworkServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 820
        ServletHolder.handle(ServletRequest, ServletResponse) line: 511
        ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1166
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 328
        LogoutFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 105
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        FilterSecurityInterceptor.invoke(FilterInvocation) line: 116
        FilterSecurityInterceptor.doFilter(ServletRequest, ServletResponse, FilterChain) line: 83
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        ExceptionTranslationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 95
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        SessionManagementFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 100
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        AnonymousAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 79
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        RememberMeAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 126
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        SecurityContextHolderAwareRequestFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 54
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        RequestCacheAwareFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 35
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        BasicAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 119
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        SecurityContextPersistenceFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 80
        FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse) line: 340
        FilterChainProxy.doFilter(ServletRequest, ServletResponse, FilterChain) line: 175
        DelegatingFilterProxy.invokeDelegate(Filter, ServletRequest, ServletResponse, FilterChain) line: 237
        DelegatingFilterProxy.doFilter(ServletRequest, ServletResponse, FilterChain) line: 167
        ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157
        ServletHandler(ServletHandler).handle(String, HttpServletRequest, HttpServletResponse, int) line: 388
        SecurityHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 216
        SessionHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 182
        WebAppContext(ContextHandler).handle(String, HttpServletRequest, HttpServletResponse, int) line: 765
        WebAppContext.handle(String, HttpServletRequest, HttpServletResponse, int) line: 418
        HandlerCollection.handle(String, HttpServletRequest, HttpServletResponse, int) line: 114
        Server(HandlerWrapper).handle(String, HttpServletRequest, HttpServletResponse, int) line: 152
        Server.handle(HttpConnection) line: 326
        HttpConnection.handleRequest() line: 542
        HttpConnection$RequestHandler.content(Buffer) line: 938
        HttpParser.parseNext() line: 755
        HttpParser.parseAvailable() line: 218
        HttpConnection.handle() line: 404
        SelectChannelConnector$ConnectorEndPoint(SelectChannelEndPoint).run() line: 409
        QueuedThreadPool$PoolThread.run() line: 582

Command line:

    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
        -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:62258
        -Duser.region=AU
        -Duser.language=en
        -server
        -Xmx1024m
        -XX:MaxPermSize=312m
        -javaagent:/org/aspectj/aspectjweaver/1.6.2/aspectjweaver-1.6.2.jar
        -Dfile.encoding=UTF-8

Classpath:

    /javax/activation/activation/1.1/activation-1.1.jar
    /javax/jms/jms/1.1/jms-1.1.jar
    /javax/mail/mail/1.4/mail-1.4.jar
    /ant/ant/1.6.5/ant-1.6.5.jar
    /commons-codec/commons-codec/1.4/commons-codec-1.4.jar
    /commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
    /com/sun/grizzly/framework/1.6.1/framework-1.6.1.jar
    /geronimo-spec/geronimo-spec-jta/1.0.1B-rc4/geronimo-spec-jta-1.0.1B-rc4.jar
    /org/mortbay/jetty/jetty/6.1.22/jetty-6.1.22.jar
    /org/mortbay/jetty/jetty-grizzly/6.1.22/jetty-grizzly-6.1.22.jar
    /org/mortbay/jetty/jetty-naming/6.1.22/jetty-naming-6.1.22.jar
    /org/mortbay/jetty/jetty-plus/6.1.22/jetty-plus-6.1.22.jar
    /org/mortbay/jetty/jetty-spring/6.1.22/jetty-spring-6.1.22.jar
    /org/mortbay/jetty/jetty-util/6.1.22/jetty-util-6.1.22.jar
    /com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
    /com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
    /org/mortbay/jetty/jsp-2.1-glassfish/9.1.1.B60.25.p1/jsp-2.1-glassfish-9.1.1.B60.25.p1.jar
    /org/mortbay/jetty/jsp-api-2.1-glassfish/9.1.1.B60.25.p1/jsp-api-2.1-glassfish-9.1.1.B60.25.p1.jar
    /log4j/log4j/1.2.15/log4j-1.2.15.jar
    /net/padlocksoftware/padlock/2.0.2/padlock-2.0.2.jar
    /postgresql/postgresql/9.0-801.jdbc4/postgresql-9.0-801.jdbc4.jar
    /org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar
    /org/mortbay/jetty/start/6.1.22/start-6.1.22.jar
    /application-container/src/test/resources/eclipse/aus
    /org/apache/lucene/lucene-spellchecker/3.0.1/lucene-spellchecker-3.0.1.jar
    /org/easytesting/fest-assert/1.4/fest-assert-1.4.jar
    /org/easytesting/fest-util/1.1.6/fest-util-1.1.6.jar
    /application-container/src/test/resources/eclipse
    /au/gov/hic/psi-crypto/1.0/psi-crypto-1.0.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar
    /System/Library/Frameworks/JavaVM.framework/Frameworks/JavaRuntimeSupport.framework/Resources/Java/JavaRuntimeSupport.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/ui.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsse.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jce.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/charsets.jar
    /System/Library/Java/Extensions/AppleScriptEngine.jar
    /System/Library/Java/Extensions/dns_sd.jar
    /System/Library/Java/Extensions/j3daudio.jar
    /System/Library/Java/Extensions/j3dcore.jar
    /System/Library/Java/Extensions/j3dutils.jar
    /System/Library/Java/Extensions/jai_codec.jar
    /System/Library/Java/Extensions/jai_core.jar
    /System/Library/Java/Extensions/mlibwrapper_jai.jar
    /System/Library/Java/Extensions/MRJToolkit.jar
    /System/Library/Java/Extensions/QTJava.zip
    /System/Library/Java/Extensions/vecmath.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext/apple_provider.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext/dnsns.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext/localedata.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext/sunjce_provider.jar
    /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext/sunpkcs11.jar
    /ar/com/fdvs/DynamicJasper/3.2.1/DynamicJasper-3.2.1.jar
    /au/gov/hic/Hic-HolDVA/4-26/Hic-HolDVA-4-26.jar
    /au/gov/hic/Hic-HolClassic/1-305/Hic-HolClassic-1-305.jar
    /au/gov/hic/J2seLibrary/1-305/J2seLibrary-1-305.jar
    /org/springframework/security/spring-security-config/3.1.0.RC2/spring-security-config-3.1.0.RC2.jar
    /org/springframework/security/spring-security-core/3.1.0.RC2/spring-security-core-3.1.0.RC2.jar
    /org/springframework/security/spring-security-web/3.1.0.RC2/spring-security-web-3.1.0.RC2.jar
    /org/springframework/spring-aop/3.0.5.RELEASE/spring-aop-3.0.5.RELEASE.jar
    /org/springframework/spring-asm/3.0.5.RELEASE/spring-asm-3.0.5.RELEASE.jar
    /org/springframework/spring-aspects/3.0.5.RELEASE/spring-aspects-3.0.5.RELEASE.jar
    /org/springframework/spring-beans/3.0.5.RELEASE/spring-beans-3.0.5.RELEASE.jar
    /org/springframework/spring-context/3.0.5.RELEASE/spring-context-3.0.5.RELEASE.jar
    /org/springframework/spring-context-support/3.0.5.RELEASE/spring-context-support-3.0.5.RELEASE.jar
    /org/springframework/spring-core/3.0.5.RELEASE/spring-core-3.0.5.RELEASE.jar
    /org/springframework/spring-expression/3.0.5.RELEASE/spring-expression-3.0.5.RELEASE.jar
    /net/sf/jasperreports/jasperreports-fonts/4.0.2/jasperreports-fonts-4.0.2.jar
    /org/springframework/spring-jdbc/3.0.5.RELEASE/spring-jdbc-3.0.5.RELEASE.jar
    /org/springframework/spring-jms/3.0.5.RELEASE/spring-jms-3.0.5.RELEASE.jar
    /org/springframework/spring-orm/3.0.5.RELEASE/spring-orm-3.0.5.RELEASE.jar
    /org/springframework/spring-tx/3.0.5.RELEASE/spring-tx-3.0.5.RELEASE.jar
    /org/springframework/spring-web/3.0.5.RELEASE/spring-web-3.0.5.RELEASE.jar
    /org/springframework/spring-webmvc/3.0.5.RELEASE/spring-webmvc-3.0.5.RELEASE.jar
    /net/sourceforge/log4jdbc/log4jdbc4/1.2beta2/log4jdbc4-1.2beta2.jar
    /medicare-jaxb/jax-qname/july2010/jax-qname-july2010.jar
    /medicare-jaxb/jaxb-api/july2010/jaxb-api-july2010.jar
    /medicare-jaxb/jaxp-api/july2010/jaxp-api-july2010.jar
    /medicare-jaxb/jaxb-ri/july2010/jaxb-ri-july2010.jar
    /medicare-jaxb/jaxb-xjc/july2010/jaxb-xjc-july2010.jar
    /medicare-jaxb/jaxb-libs/july2010/jaxb-libs-july2010.jar
    /commons-collections/commons-collections/3.1/commons-collections-3.1.jar
    /asm/asm/1.5.3/asm-1.5.3.jar
    /pd4ml/pd4ml/371b4/pd4ml-371b4.jar
    /pd4ml/ss_css2/371b4/ss_css2-371b4.jar
    /org/apache/commons/commons-lang3/3.0.1/commons-lang3-3.0.1.jar
    /commons-lang/commons-lang/2.5/commons-lang-2.5.jar
    /commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar
    /commons-io/commons-io/1.4/commons-io-1.4.jar
    /org/restlet/jee/org.restlet/2.0-M4/org.restlet-2.0-M4.jar
    /org/osgi/osgi_R4_core/1.0/osgi_R4_core-1.0.jar
    /org/apache/activemq/activemq-core/5.5.0/activemq-core-5.5.0.jar
    /org/apache/activemq/activemq-pool/5.5.0/activemq-pool-5.5.0.jar
    /commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar
    /commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar
    /org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar
    /org/apache/activemq/kahadb/5.5.0/kahadb-5.5.0.jar
    /org/apache/activemq/protobuf/activemq-protobuf/1.0/activemq-protobuf-1.0.jar
    /org/osgi/org.osgi.core/4.1.0/org.osgi.core-4.1.0.jar
    /org/springframework/osgi/spring-osgi-core/1.2.1/spring-osgi-core-1.2.1.jar
    /org/springframework/osgi/spring-osgi-io/1.2.1/spring-osgi-io-1.2.1.jar
    /org/apache/geronimo/specs/geronimo-j2ee-management_1.0_spec/1.0/geronimo-j2ee-management_1.0_spec-1.0.jar
    /commons-net/commons-net/2.0/commons-net-2.0.jar
    /org/apache/activemq/activeio-core/3.1.2/activeio-core-3.1.2.jar
    /org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar
    /commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.jar
    /log4j/apache-log4j-extras/1.1/apache-log4j-extras-1.1.jar
    /log4j/log4j/1.2.14/log4j-1.2.14.jar
    /org/apache/activemq/activemq-ra/5.5.0/activemq-ra-5.5.0.jar
    /org/apache/xbean/xbean-spring/3.3/xbean-spring-3.3.jar
    /org/json/json/20070829/json-20070829.jar
    /org/tranql/tranql-connector/1.4/tranql-connector-1.4.jar
    /org/apache/geronimo/specs/geronimo-j2ee-connector_1.5_spec/1.1/geronimo-j2ee-connector_1.5_spec-1.1.jar
    /org/freemarker/freemarker/2.3.14/freemarker-2.3.14.jar
    /javatar/javatar/2.5/javatar-2.5.jar
    /aopalliance/aopalliance/1.0/aopalliance-1.0.jar
    /org/liquibase/liquibase-core/2.0.1/liquibase-core-2.0.1.jar
    /javax/mail/mail/1.4.2/mail-1.4.2.jar
    /jaybird/jaybird/2.1.3/jaybird-2.1.3.jar
    /jaybird/jaybird-pool/2.1.3/jaybird-pool-2.1.3.jar
    /org/hibernate/hibernate-core/3.3.2.GA/hibernate-core-3.3.2.GA.jar
    /antlr/antlr/2.7.6/antlr-2.7.6.jar
    /javax/transaction/jta/1.1/jta-1.1.jar
    /org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar
    /org/hibernate/hibernate-ehcache/3.3.2.GA/hibernate-ehcache-3.3.2.GA.jar
    /net/sf/ehcache/ehcache/1.2.3/ehcache-1.2.3.jar
    /org/apache/lucene/lucene-core/3.0.1/lucene-core-3.0.1.jar
    /commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
    /commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar
    /stax/stax-api/1.0.1/stax-api-1.0.1.jar
    /javassist/javassist/3.9.0.GA/javassist-3.9.0.GA.jar
    /cglib/cglib/2.1_3/cglib-2.1_3.jar
    /dom4j/dom4j/1.6.1/dom4j-1.6.1.jar
    /xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
    /nu/xom/xom/1.2.4/xom-1.2.4.jar
    /jaxen/jaxen/1.1/jaxen-1.1.jar
    /jdom/jdom/1.0/jdom-1.0.jar
    /xerces/xercesImpl/2.6.2/xercesImpl-2.6.2.jar
    /org/hibernate/hibernate-annotations/3.4.0.GA/hibernate-annotations-3.4.0.GA.jar
    /org/hibernate/ejb3-persistence/1.0.2.GA/ejb3-persistence-1.0.2.GA.jar
    /org/hibernate/hibernate-commons-annotations/3.1.0.GA/hibernate-commons-annotations-3.1.0.GA.jar
    /org/slf4j/slf4j-log4j12/1.5.6/slf4j-log4j12-1.5.6.jar
    /javax/persistence/persistence-api/1.0/persistence-api-1.0.jar
    /jasperreports/jasperreports/3.7.6/jasperreports-3.7.6.jar
    /org/mvel/mvel2/2.0.14/mvel2-2.0.14.jar
    /joda-time/joda-time/1.5.2/joda-time-1.5.2.jar
    /org/drools/drools-core/5.1.1/drools-core-5.1.1.jar
    /org/drools/drools-grid-core/5.1.1/drools-grid-core-5.1.1.jar
    /org/drools/drools-api/5.1.1/drools-api-5.1.1.jar
    /org/drools/drools-spring/5.1.1/drools-spring-5.1.1.jar
    /org/drools/drools-compiler/5.1.1/drools-compiler-5.1.1.jar
    /org/antlr/antlr-runtime/3.1.3/antlr-runtime-3.1.3.jar
    /org/eclipse/jdt/core/compiler/ecj/3.5.1/ecj-3.5.1.jar
    /com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar
    /xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar
    /org/apache/poi/poi/3.2-FINAL/poi-3.2-FINAL.jar
    /org/jbpm/jbpm3/jbpm-jpdl/3.3.1.GA/jbpm-jpdl-3.3.1.GA.jar
    /bsh/bsh/1.3.0/bsh-1.3.0.jar
    /junit/junit/4.8.1/junit-4.8.1.jar
    /org/apache/ant/ant/1.7.0/ant-1.7.0.jar
    /org/apache/ant/ant-launcher/1.7.0/ant-launcher-1.7.0.jar
    /org/apache/jackrabbit/jackrabbit-core/1.4.5/jackrabbit-core-1.4.5.jar
    /concurrent/concurrent/1.3.4/concurrent-1.3.4.jar
    /org/apache/jackrabbit/jackrabbit-api/1.4/jackrabbit-api-1.4.jar
    /org/apache/jackrabbit/jackrabbit-jcr-commons/1.4/jackrabbit-jcr-commons-1.4.jar
    /org/apache/jackrabbit/jackrabbit-spi-commons/1.4/jackrabbit-spi-commons-1.4.jar
    /org/apache/jackrabbit/jackrabbit-spi/1.4/jackrabbit-spi-1.4.jar
    /org/apache/jackrabbit/jackrabbit-text-extractors/1.4/jackrabbit-text-extractors-1.4.jar
    /pdfbox/pdfbox/0.6.4/pdfbox-0.6.4.jar
    /org/textmining/tm-extractors/0.4/tm-extractors-0.4.jar
    /nekohtml/nekohtml/0.9.4/nekohtml-0.9.4.jar
    /org/apache/derby/derby/10.2.1.6/derby-10.2.1.6.jar
    /org/jbpm/jbpm-identity/3.2.2/jbpm-identity-3.2.2.jar
    /org/beanshell/bsh/2.0b4/bsh-2.0b4.jar
    /org/aspectj/aspectjrt/1.6.2/aspectjrt-1.6.2.jar
    /org/aspectj/aspectjweaver/1.6.2/aspectjweaver-1.6.2.jar
    /org/codehaus/groovy/groovy-all/1.7.3/groovy-all-1.7.3.jar
    /net/sf/saxon/saxon-he/9.2.0.6/saxon-he-9.2.0.6.jar
    /org/milyn/milyn-smooks-core/1.3/milyn-smooks-core-1.3.jar
    /org/milyn/milyn-commons/1.3/milyn-commons-1.3.jar
    /dtdparser/dtdparser/1.21/dtdparser-1.21.jar
    /org/milyn/milyn-smooks-javabean/1.3/milyn-smooks-javabean-1.3.jar
    /commons-digester/commons-digester/1.7/commons-digester-1.7.jar
    /com/lowagie/itext/2.1.7/itext-2.1.7.jar
    /xalan/xalan/2.3.1/xalan-2.3.1.jar
    /cryptix/cryptix/3.2.0/cryptix-3.2.0.jar
    /cryptix/cryptix-pgp/3.2.0/cryptix-pgp-3.2.0.jar
    /org/bouncycastle/bcpg-jdk16/144/bcpg-jdk16-144.jar
    /org/bouncycastle/bcprov-ext-jdk16/144/bcprov-ext-jdk16-144.jar
    /net/sf/gnu-hylafax/gnu-hylafax-pool/1.0.3/gnu-hylafax-pool-1.0.3.jar
    /net/sf/gnu-hylafax/gnu-hylafax-core/1.0.3/gnu-hylafax-core-1.0.3.jar
    /net/sf/gnu-hylafax/gnu-hylafax-inet-ftp/1.0.3/gnu-hylafax-inet-ftp-1.0.3.jar
    /net/sf/gnu-hylafax/gnu-hylafax-utils/1.0.3/gnu-hylafax-utils-1.0.3.jar
    /commons-cli/commons-cli/1.0/commons-cli-1.0.jar
    /urbanophile/java-getopt/1.0.9/java-getopt-1.0.9.jar
    /org/codehaus/btm/btm/2.1.2/btm-2.1.2.jar
    /org/restlet/jee/org.restlet.ext.json/2.0-M4/org.restlet.ext.json-2.0-M4.jar
    /org/restlet/jee/org.restlet.lib.org.json/2.0/org.restlet.lib.org.json-2.0.jar
    /org/restlet/jee/org.restlet.ext.freemarker/2.0-M4/org.restlet.ext.freemarker-2.0-M4.jar
    /org/restlet/jee/org.restlet.ext.xml/2.0-M4/org.restlet.ext.xml-2.0-M4.jar
    /org/restlet/jee/org.restlet.ext.spring/2.0-M4/org.restlet.ext.spring-2.0-M4.jar
    /org/restlet/jee/org.restlet.ext.servlet/2.0-M4/org.restlet.ext.servlet-2.0-M4.jar
    /javax/servlet/servlet-api/2.5/servlet-api-2.5.jar
    /com/google/code/findbugs/findbugs/1.3.8/findbugs-1.3.8.jar
    /com/google/code/findbugs/bcel/1.3.8/bcel-1.3.8.jar
    /com/google/code/findbugs/jsr305/1.3.8/jsr305-1.3.8.jar
    /com/google/code/findbugs/jFormatString/1.3.8/jFormatString-1.3.8.jar
    /com/google/code/findbugs/annotations/1.3.8/annotations-1.3.8.jar
    /asm/asm-analysis/3.1/asm-analysis-3.1.jar
    /asm/asm-tree/3.1/asm-tree-3.1.jar
    /asm/asm-commons/3.1/asm-commons-3.1.jar
    /asm/asm-util/3.1/asm-util-3.1.jar
    /asm/asm-xml/3.1/asm-xml-3.1.jar
    /com/h2database/h2/1.1.115/h2-1.1.115.jar
    /commons-el/commons-el/1.0/commons-el-1.0.jar
    /cpsuite/cpsuite/1.1.0/cpsuite-1.1.0.jar
    /org/dbunit/dbunit/2.2/dbunit-2.2.jar
    /junit-addons/junit-addons/1.4/junit-addons-1.4.jar
    /xerces/xmlParserAPIs/2.6.2/xmlParserAPIs-2.6.2.jar
    /org/jbehave/jbehave-core/2.1/jbehave-core-2.1.jar
    /junit/junit-dep/4.4/junit-dep-4.4.jar
    /org/hamcrest/hamcrest-all/1.1/hamcrest-all-1.1.jar
    /org/mockito/mockito-all/1.8.4/mockito-all-1.8.4.jar
    /au/gov/hic/Hic-HolMedical/4-87/Hic-HolMedical-4-87.jar
    /au/gov/hic/hic-online/6.11-4/hic-online-6.11-4.jar
    /com/google/common/guava/r07/guava-r07.jar
    /commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar
    /net/sourceforge/barbecue/barbecue/1.5-beta1/barbecue-1.5-beta1.jar
    /medicare-online-aus/src/test/resources
    /org/jboss/resteasy/jaxrs-api/2.3.1.GA/jaxrs-api-2.3.1.GA.jar
    /org/jboss/resteasy/resteasy-jaxrs/2.3.1.GA/resteasy-jaxrs-2.3.1.GA.jar
    /org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar
    /org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar
    /org/hornetq/hornetq-bootstrap/2.2.5.Final/hornetq-bootstrap-2.2.5.Final.jar
    /org/hornetq/hornetq-core/2.2.5.Final/hornetq-core-2.2.5.Final.jar
    /org/hornetq/hornetq-core-client/2.2.5.Final/hornetq-core-client-2.2.5.Final.jar
    /org/hornetq/hornetq-jms/2.2.5.Final/hornetq-jms-2.2.5.Final.jar
    /org/hornetq/hornetq-jms-client/2.2.5.Final/hornetq-jms-client-2.2.5.Final.jar
    /org/hornetq/hornetq-logging/2.2.5.Final/hornetq-logging-2.2.5.Final.jar
    /org/hornetq/hornetq-spring-integration/2.2.5.Final/hornetq-spring-integration-2.2.5.Final.jar
    /io/netty/netty/3.3.1.Final/netty-3.3.1.Final.jar
Reply | Threaded
Open this post in threaded view
|

Re: BTM-2.1.1 Hangs

Ludovic Orban-2
Sorry for the delay getting back to you.


From the stack trace you provided, we can clearly conclude that Proxy.newProxyInstance is getting re-entered as I see it twice. This definitely explains why it's hanging looking at how its cache algorithm is crafted.

The reason why it's getting re-entered is less clear but it definitely involves a mix of at least things: AspectJ and Hibernate. It looks like the Proxy.newProxyInstance call is being intercepted by AspectJ (the InstrumentationImpl.transform call) which ends up accessing a Hibernate persisted collection (the PersistentMap.get call) which eventually convinces Hibernate to initialize the collection (the SessionImpl.initializeCollection call) which obviously forces Hibernate to acquire a connection which requires generating a proxy.

I understand why you consider this a bug as this method call should be thread-safe and reentrant but I fail to see how I could get this solved, short of switching to a 3rd party proxying lib. Wouldn't it be cleaner and more straightforward to figure out why AspectJ is accessing a Hbernate lib and preventing that, or to eagerly fetch it?

--
Ludovic
Reply | Threaded
Open this post in threaded view
|

Re: BTM-2.1.1 Hangs

Tommy Yim Li
I'm a colleague of Mul and we've been working together on this problem again recently because of other issues we've had with Bitronix 1.x and we really needed to upgrade Bitronix 2.1.3.

Your reply gave us a good insight and really helped us narrow the the issue so thanks for that.

It turns out this deadlock is due to JBPM 3 (a dead project) swapping out the current thread's classloader and (in the stacktrace above, done by Decision.execute(ExecutionContext)) replacing it with its own as outlined here https://issues.jboss.org/browse/JBPM-1448.  The problem described from their jira is not relevant to us anyway so we commented it out and it worked (hack using aop).

What we're a bit baffled by is why the first call of newProxyInstance() works but the second call didn't since the logic to swap out the classloader is done before the first call.  However, we're not too fussed since we know the current version of JBPM we're using is a baggage and we have plans to upgrade this anyway.

Tommy.

Ludovic Orban-2 wrote
Sorry for the delay getting back to you.


From the stack trace you provided, we can clearly conclude that
Proxy.newProxyInstance is getting re-entered as I see it twice. This
definitely explains why it's hanging looking at how its cache algorithm is
crafted.

The reason why it's getting re-entered is less clear but it definitely
involves a mix of at least things: AspectJ and Hibernate. It looks like the
Proxy.newProxyInstance call is being intercepted by AspectJ (the
InstrumentationImpl.transform call) which ends up accessing a Hibernate
persisted collection (the PersistentMap.get call) which eventually
convinces Hibernate to initialize the collection (the
SessionImpl.initializeCollection call) which obviously forces Hibernate to
acquire a connection which requires generating a proxy.

I understand why you consider this a bug as this method call should be
thread-safe and reentrant but I fail to see how I could get this solved,
short of switching to a 3rd party proxying lib. Wouldn't it be cleaner and
more straightforward to figure out why AspectJ is accessing a Hbernate lib
and preventing that, or to eagerly fetch it?

--
Ludovic