THREAD DUMP ANALYSIS PATTERN – ALL ROADS LEAD TO ROME

Description

If several threads in a thread dump ends up working on one single method, then it might be of concern. Most of the times, if there is a problem (say poorly responding data source, un-relinquished lock, infinite looping threads …) then significant number of threads will end up in one single method. That particular method has to be analyzed in detail.

Example

This application was connecting with Apache Cassandra NoSQL Database. Application uses DataStax java driver to connect with Cassandra. DataStax has dependency on the netty library. To be specific following are the libraries that application uses:

• cassandra-driver-core-2.0.1.jar
• netty-3.9.0.Final.jar

This application all of sudden ran in to ‘java.lang.OutOfMemoryError: unable to create new native thread‘. When thread dump was taken on the application, around 2460 threads were in ‘runnable’ state stuck in the method: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method). Below is the stack trace of one of the thread:

"New I/O worker #211" prio=10 tid=0x00007fa06424d000 nid=0x1a58 runnable [0x00007f9f832f6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked  (a sun.nio.ch.Util$2)
	- locked  (a java.util.Collections$UnmodifiableSet)
	- locked  (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

Oh boy! This is way too many threads. All of them are netty library threads. Apparently issue turned out that Cassandra NoSQL DB ran out of space. This issue was cascading in the application as OutOfMemoryError. When more space was allocated in Cassandra DB, problem went away.

Thus always look out for the method(s) where most of the threads are working.

Why named as ‘All roads lead to Rome’?

‘All Roads lead to Rome’ is a famous proverb to indicate different paths can take to one final end. Similarly when there is a problem there is a high chance that several threads will finally end-up in one problematic method.