Ticket #245 (closed defect: invalid)

Opened 7 years ago

Last modified 6 years ago

Container shutdown (not all threads exit)

Reported by: ibaldin Owned by: ibaldin
Priority: major Milestone: Dungeness 4.0
Component: ORCA: Tomcat container Version: Dungeness 4.0
Keywords: Cc:

Description (last modified by ibaldin) (diff)

I think that orca container is not being shutdown properly. I have checked it on different machines with both orca-3.1 and two versions of orca-3.1-extended. When I run /opt/orca/tomcat/stop.sh , the last thing I see in catalina.out is the following:

Feb 26, 2012 9:57:05 AM org.apache.catalina.core.StandardServer? await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Feb 26, 2012 9:57:05 AM org.apache.coyote.AbstractProtocol? pause
INFO: Pausing ProtocolHandler? http-bio-11080?

We also don't see the following in catalina.out, which is printed by ContainerServletContextListener:contextDestroyed() . This is the only method that calls the container shutdown code. (Globals.getContainer().shutdown())

"
Orca context about to be destroyed
"

We no longer see the following outputs in orca.log, which are supposed to be printed by OrcaContainer:shutdown() :

"
Orca container shutting down...
Orca container is no longer active...
"

This means that lots of threads might still be running. An anecdotal evidence is the existence of orca**** temp files in /tmp even after orca container shuts down, which might mean that the OrcaThreadPool? hasn't been shutdown properly.

Change History

Changed 7 years ago by ibaldin

  • description modified (diff)

I have tested this with orca-3.1-extended branch.

I don't think orca is getting shutdown notification from tomcat. If I understand correctly, ContainerServletContextListener:contextDestroyed() is supposed to be invoked when tomcat is shutdown, since ContainerServletContextListener? implements ServletContextListener?. But, I don't see the system outs from contextDestroyed() in catalina.out. I have seen those system outs at some point of time in the past, most probably before migrating to tomcat 7. If orca container got shutdown notification as a result of invocation of contextDestroyed(), log outputs like "Orca container shutting down..." would have appeared in orca.log, which I don't see either.

Regards,
- Anirban

Changed 7 years ago by ibaldin

OK, so testing on my laptop, I can see that the shutdown call does get called in T7. So this may be an issue with Linux and Tomcat.

We do get a bunch of these (a few timer threads, some unnamed threads and all actor threads) :


Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [ben-service] but has failed to stop it. This is very likely to create a memory
leak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [ndl-broker] but has failed to stop it. This is very likely to create a memory
leak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-28] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-29] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-30] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-31] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-32] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-33] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-34] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-35] but has failed to stop it. This is very likely to create a memory l
eak.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-36] but has failed to stop it. This is very likely to create a memory l
eak.
Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-37] but has failed to stop it. This is very likely to create a memory l
eak.
Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [Thread-38] but has failed to stop it. This is very likely to create a memory l
eak.
Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesThreads
SEVERE: The web application /orca appears to have started a thread named [euca-vm-site] but has failed to stop it. This is very likely to create a memor
y leak.


as well as this


Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesJdbc
SEVERE: The web application /orca registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. T
o prevent a memory leak, the JDBC Driver has been forcibly unregistered.Feb 27, 2012 10:34:40 AM org.apache.catalina.loader.WebappClassLoader? clearReferencesJdbc
SEVERE: The web application /orca registered the JDBC driver [org.apache.commons.dbcp.PoolingDriver?] but failed to unregister it when the web applicatio
n was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.


from Tomcat (latest tomcat has some memory leak detection in it). Question is - is this harmful? I'm not sure. I presume most of those threads are daemon threads, so it doesn't matter. Not sure about the JDBC part. A little disconcerting that all actor threads do not exit apparently.

Changed 6 years ago by ibaldin

  • status changed from new to closed
  • resolution set to invalid

No longer relevant with the switch to embedded jetty.

Note: See TracTickets for help on using tickets.