THREAD DUMP ANALYSIS PATTERN – ADDITIVES

Description

It’s highly recommended to capture 3 threads dumps in an interval of 10 seconds to uncover any problem in the JVM. If in the 2^nd and 3^rd thread dump if additional threads starts to go in to a particular state, then those threads and their stack traces has to be studied in detail. It may or may not be indicative of certain problem in the application, but definitely a good lead to follow.

Example

This problem surfaced because of a thread leak in Oracle JDBC Driver when ONS feature was turned ON. This problem happend in a old version of Oracle JDBC Driver (almost in 2011). Because of the bug in the driver, under certain scenarios it started to spawn tonnes of new threads. In every captured thread dump new threads in RUNNABLE stated with below stack trace got added. Around 1700 threads with the same stack trace got created.

Thread-6805 - priority:8 - threadId:0x07768000 - nativeId:10966 - addressSpace:null - state:RUNNABLE
stackTrace:
	java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:155)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:268)
	at oracle.ons.InputBuffer.getNextString(InputBuffer.java:223)
	at oracle.ons.ReceiverThread.run(ReceiverThread.java:266)

Why named as additives?

Additives is defined as ‘a substance added to something in small quantities’. Similarly this pattern talks above addition of new threads to an existing state.