Creating and analyzing Java heap dumps

Heap analysis is useful in finding memory leaks, which typically manifest themselves with this error:

Java heap space java.lang.OutOfMemoryError: Java heap space

Creating a heap dump

Creating the heap dump depends on the particular way the application is running. For webservice applications running inside the container, first determine the PID of the container, then execute

jmap -F -dump:format=b,file=heap-20241.bin 20241

The '-F' is optional and should only be used if the process is not responding to jmap at all. Without '-F' you can use '-dump:live,format=b,file=<filename>' to get only the live objects in the heap. Note that jmap tool comes standard as part of a JDK. Note - heap files can be very large (1+GB, depending on the JVM settings).

If it is running inside a wrapper, sent SIGSTOP to the wrapper process before starting and SIGCONT afterwards to make sure the wrapper does not kill the JVM due to lack of response during the heap dump. Look in /var/run/orca for the pid's. Also ps -ef will tell the parent.

Analyzing a heap dump

Many methods exist. The easiest is to use the tool called 'visualvm' that comes part of the jdk - it is a GUI-driven tool that makes analyzing the heap more intuitive.

Looking at CPU performance

Used “top” to find the thread inside the JVM that was eating CPU - it’s the actor thread.

How to do this:

  1. Start top
  2. Hit “H” to enable “show threads” mode
  3. Let it settle for a bit - it takes a second or two for the counts to get right
  4. Find the java thread eating the CPU
  5. Use a decimal to hex converter, to convert the PID to hex
  6. Send a “kill -3” to the parent process for the thread - what was eating CPU, before you did “show threads” in top
  7. Look in the orca-stdout.log/controller-stdout.log for the hex version of the PID; that’s the thread that’s busy.

Taking a thread dump

For analyzing deadlocks, thread dump is a good tool. Issuing 'kill -3' to the wrapper process or to the java process itself will result in thread dump going to stdout (wherever it is directed).