Posts Tagged BASH

Often, when working with Java machines behind firewalls, there is the requirement to debug or analyze Java servers or processes in general using tools like Java VisualVM, jconsole or others. To make it as simple as possible, JMX- and RMI-ports need to be configured on Java application server and authentication or certificate checking for security are disabled in this example.

Java application on remote server

When starting up the application, add these (sample) configuration parameters:

-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.rmi.port=9011
-Djava.rmi.server.hostname=localhost

Both ports need to be tunneled via SSH to be accessible on the local machine.

SSH-Tunnel on local machine

When working on *nix environments, the default SSH command is capable to establish the SSH-tunnel to the remote server.

Default command:

ssh -C $username@$server_ip -L $port_local_machine:localhost:$port_remote_machine

Example command to map both ports from example above:

ssh -C $username@$server_ip -L 9010:localhost:9010 -L 9011:localhost:9011

Another solution, especially on Windows machines, would be to use PuTTY.

In PuTTY Configuration window, select the category: “Connection” – “SSH” – “Tunnels” and put the necessary configuration in “Add new forwarded port”:

For this example:

Source port: 9010
Destination: localhost:9010
Choose “Local”- and “Auto”-radiobox and press the “Add”-Button.

Source port: 9011
Destination: localhost:9011
Choose “Local”- and “Auto”-radiobox and press the “Add”-Button.

Debug tool

Java VisualVM, jconsole or other tools can be used by starting them on the local machine and telling them to connect to a remote process: “localhost:9010”

If a running Java application on a machine generates (too) much CPU load it is often necessary to identify the Java thread which is responsible for this behaviour. On a Linux or Unix environment this can easily be done by using a shell:

Finding the problematic Java process

The “top”-command which shows tasks sorted by load intensity is available on nearly all installations:

$ top

Analysing the given table it is quite simple to find out the PID (process id) of the Java process that causes the problem.
Another way is to use the “ps” command to identify the process:

$ ps axu | grep java

Creating a thread dump

Next step is to create a thread dump using the command “jstack”. This command is part of the JDK, so it is normally available per default. “jstack” needs the PID of the Java process given as parameter. If “jstack” hangs or does not respond it has to be forced to take the dump using the “-F” parameter. It is also a good idea write its output into a file, an example with PID=12345 that writes a file into “/tmp” directory:

$ jstack -F 12345 > /tmp/heapdump

Identifying the problematic Java thread(s)

The “top” command is also able to display all threads of a task by pressing “H” when started or using the “-H” parameter:

$ top -H

So this time the PID(s) of problematic thread(s) can also be found out.

Now it is possible to find information about the thread(s) causing the problem by searching the heapdump file for these PIDs, assuming a PID is 32145:

-in a dump created with Java 7 and newer a PID can be found directly, for example:
Thread 32145: (state = IN_NATIVE)

-in a dump created with Java 6 and earlier it is necessary to convert the PID to a HEX value, also using BASH for example:

$ printf '%x\n' 32145

The result for this example is “7d91” and can be found in heapdump file as an “nid”, for example:
“Thread-0” prio=10 tid=0x00002f6a98760000 nid=0x7d91 runnable [0x00001f9a4d123000]