Tread dump:- Thread dump provides a snapshot of the current active live threads. It provides the stack trace of all the java threads in the JVM. It is used when the server is hung and we want to see the threads executing and take their dump.
A java thread dump is a way finding out way what every thread in JVM is doing at a particular point of time. This is especially useful if your Java application sometimes seems to hang when running under load as analysis of the dump will show where the threads are stuck.
There are different ways to take thread dump.
In unix: kill -3 <pid>
In windows: ctrl+break
Weblogic.Admin utility: java weblogic.Admin -url t3://localhost:7001 -username weblogic -password weblogic THREAD_DUMP
WLST Scripting:
connect(‘weblogic’,’weblogic’,’t3://localhost:7001′)
cd(‘server’)
cd(‘AdminServer’)
ThreadDump()
disconnect()
exit()
Admin console:
Step1: login to the admin console
Step2: Click on server
Step3: Navigate to servers
Step4: Click monitor tab
Step5: Click on thread
Step6: Click on the dump thread stack.
Locating the Thread Dump: The thread dump is placed in the WebLogic log file. The log file location varies depending on the OS platform:
For UNIX: the output is sent to:
<PS_HOME>/webserv/<DOMAIN_NAME>/servers/logs/PIA_stdout.log
For Linux: the output is sent to:
<PS_HOME>/webserv/<DOMAIN_NAME>/servers/logs/PIA_stderr.log
For Windows: the output is sent to:
<PS_HOME>\webserv\<DOMAIN_NAME>\servers\PIA\logs\NTservice-<DOMAIN_NAME>-PIA.log
Analyzing a Thread Dump: The thread dump can be a bit challenging to analyze, and you may need assistance from an Oracle Support Engineer. Below are some tips on how to analyze the thread dump. This information is broken out into the following sections:
- General Information about the thread dump
- Overview of types of threads commonly seen in thread dump
- Examples of different issues you may observe in the thread dump
1) General Information about the Thread Dump: Note that the thread dump always begins with this line:
===== FULL THREAD DUMP ===============
And ends with this line:
===== END OF THREAD DUMP ===============
The first line of the thread dump shows when the thread dump was created, followed by the exact java version you are using.
Example:
Mon Apr 18 12:46:56 2011
Oracle JRockit(R) R28.0.0-679-130297-1.6.0_17-20100312-2123-windows-ia32
2) Overview of Types of Threads commonly seen in Thread Dump:
- i) Threads waiting for Requests: You will always see some threads that are just waiting for work, as WebLogic always allocates some threads to be available and ready to process any incoming requests. These threads can easily be identified because you’ll see “ExecuteThread.waitForRequest”in the call stack. These threads will be in ‘ACTIVE’ or ‘STANDBY’ mode. These threads do not have much significance when troubleshooting. However, if you see a lot of these threads waiting for requests (20 or more), it most likely indicates that the environment is just recovering from a very heavy load, when the thread dump was taken (and as the load diminishes, WebLogic will remove many of these extra threads that are waiting for requests)
Ex: at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
- ii) Socket Muxer Threads: You will also see approximately two to five socket muxer threads. These threads’ main responsibility is to read the request off the socket and pass the work to the appropriate thread. WebLogic allocates a percentage of execute threads from the self-tuning thread pool to be Muxer threads. Usually you will see three or four of these threads:
“ExecuteThread: ‘0’ for queue: ‘weblogic.socket.Muxer’” id=25 idx=0x60 tid=2068 prio=5 alive, in native
iii) ListenThreads: You will also see approximately six “listen threads”, usually three for SSL and three for non-SSL. The purpose of these threads is to wait for connections to arrive. All browser requests enter the WebLogic server through these threads.
“DynamicListenThread[Default]” id=39 idx=0x90 tid=2812 prio=9 alive, in native
“DynamicSSLListenThread[DefaultSecure]” id=40 idx=0x94 tid=3148 prio=9 alive, in native
- iv) Jolt Connection Threads: WebLogic Server and the Tuxedo Application Server use Jolt to communicate with each other. PIA creates two threads inside the WebLogic’s JVM per Jolt connection. For each Jolt connection made between WebLogic and the Tuxedo Application Servers, you will see a LLENwReader and a LLENwWriter thread in the thread dump:
“LLENwReader” id=52 idx=0xc4 tid=4408 prio=5 alive, in native, daemon
“LLENwWriter” id=53 idx=0xc8 tid=7828 prio=5 alive, waiting, native_blocked, daemon
- v) Threads waiting on Application Server: If the web server is waiting on the app server to process a request, you will see the following thread (below)
at bea/jolt/IOBuf.waitOnBuf(IOBuf.java:119)
3) Examples of Different Issues you may Observe in Thread Dump: Below are examples of different issues and the thread stacks you may observe.
Many threads waiting on App Server: If you see a lot of threads such as the one below, then this means that many of the WebLogic threads are waiting on the application server to finish processing the request:
at bea/jolt/IOBuf.waitOnBuf(IOBuf.java:119)
- i) Many threads processing the same call stack: If you see many threads all processing the same call stack, then you may need to review contents of the call stack in order to troubleshoot the issue. For example, in one case, the web server hung and the thread dump showed hundreds of threads like the one below. This was caused by an issue with a proxy server configuration, causing all threads to get hung up at logout:
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
psft.pt8.psp.logoutAccessedPIAs(Unknown Source)
- ii) All threads busy and waiting on one thread: By design, the PIA does not allow more than one request per HTTP session, to be submitted to the application server. If the PIA receives multiple requests from the same HTTP session, it will queue up all subsequent requests and process just one at a time. Typically, there should not be situations where the PIA receives multiples requests from the same HTTP session. However, this can occur in the following situations:
- You are using a proxy server that is re-submitting requests to the web server if a response is not received within a certain time.
-OR-
- A user submits a long-running request, and while waiting for the request to finish, the user continuously attempts to submit more requests.
When one of the above scenarios occurs, in the thread-dump you see one request waiting on Jolt to get response from the App-Server and many other threads waiting for the lock on the session to be released. Below are excerpts from a thread dump, showing this situation:
- a)There are many threads like this that are “blocked”, and all the threads are waiting on the same lock #.
— Blocked trying to get lock: java/lang/String@0x27D36AC0[thin lock]
- b)The thread that is holding the lock on “0x27D36AC0” (that all blocked threads are waiting on), is usually processing a jolt request (ie it is waiting on the application server):
at bea/jolt/IOBuf.waitOnBuf(IOBuf.java:119)
^– Holding lock: java/lang/String@0x27D36AC0[thin lock]
- c)At the end of the thread dump, you may see a list of “blocked locked chains”. In this list, you’ll notice that all threads are waiting on one thread: “Thread #0” in this example. Which happens to be a jolt request (ie it is waiting on application server)
Blocked lock chains
===================
Chain 2: “[ACTIVE] ExecuteThread: ‘2’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=35 idx=0x80 tid=3964 waiting for java/lang/String@0x27D36AC0 held by:
“[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=16 idx=0x48 tid=180 in chain 1
Chain 3: “[ACTIVE] ExecuteThread: ‘3’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=44 idx=0xa4 tid=4620 waiting for java/lang/String@0x27D36AC0 held by:
“[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=16 idx=0x48 tid=180 in chain 1
Chain 4: “[ACTIVE] ExecuteThread: ‘4’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=49 idx=0xb8 tid=1120 waiting for java/lang/String@0x27D36AC0
held by:
“[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=16 idx=0x48 tid=180 in chain 1
Analysing ThreadDump by using Summari tool:
Download: The binary is available for download at http://yusuke.homeip.net/samurai/en/samurai.jar
How to launch samurai: You can simply double-click to launch Samurai on your desktop or type as following in your command prompt.
$java -jar samurai.jar
Automatic update is not available with this way. Please check and download latest version manually.
Step1: Drag and drop the ThreadDump into summary tool
Step2: When Samurai detects a thread dump in your log, a tab named “Thread Dump” will appear.
Step3: You can just click “Thread dumps” tab to see the analysis result. Samurai colors idle threads in gray, blocked threads in red and running threads in green.
There are two resultant views and Samurai shows “Table view” by default.
In many cases, you are just interested in the table view and the sequence view. Use the table view to decide which thread needs be inspected, the sequence view to understand the thread’s behavior.
Result1:
Result2:
Happy Learning !!
15 Best Jenkins Alternatives in 2022
Weblogic Issues And Resolutions