How to Find the PIDs for Swapped Processes

zq9711

How to Find the PIDs for Swapped Processes on a Solaris System
This is one way of finding out which processes are currently swapped on a Solaris system. There are supposedly other ways of reaching this goal but none of these are currently known to the author.
To use this method you need to able to execute the Modular Debugger with root permissions. The Modular Debugger debugger is available if you have the package SUNWmdb installed. File is named '/usr/bin/mdb'.
Your 'vmstat' output is listing processes as swapped. Something like this:   # vmstat 1 5
   procs     memory            page            disk          faults      cpu
   r b w   swap  free  re  mf pi po fr de sr s0 s1 sd sd   in   sy   cs us sy id
   0 0 10 3552488 58856 39 222 124 123 157 0 74 14 14 36 2 484   6   37 12  8 80
   0 0 15 3418184 15600 31 326 64 224 264 0 344 7 7 0  0  374 1418 1337  3  4 93
   0 0 15 3418184 15784 1   1  0 152 160 0 91 2  2  0  0  342  956 1238  0  0 100
   0 0 15 3418168 15768 0   1  8  0  0  0  0  0  1  4  0  318 1289 1319  2  0 98
   0 0 15 3418168 15768 0   0  0  0  0  0  0  0  0 16  0  325 1081 1262 23  1 76
The numbers in the third column above show that we have 15 swapped processes, but how do we identify the process IDs for these processes? Using the Module Debugger and a lot of patience is it possible track down these PIDs. We do this following the below steps:

In a full thread list, search out the threads where the flag 'schflag' is set to 0 (zero). This means that the thread is not in physical memory.

For each of these threads, get the 'procp' value from the 'thread.brief' output.

For this 'procp' value, get the 'pipd' value from the 'proc' output.

For this 'pipd' value, get the 'id' value from the 'pid' output.

These 'id' value is the PID to which the thread in question belong.

Now repeat for all threads which have 'schedflag = 0'.
This an example meant to illustrate above steps:
(Step #1)
First we generate the full thread list. This amount of output can be massive so we will redirect the output to a file named 'allthreads.list'.   # mdb -k
  Loading modules: [ unix krtld genunix ip ptm cpc ipc random nfs ]
  > ::walk thread | $ allthreads.list
This is part of the output:   ...
  0x3000ab09b28:  sleepq          panic_trap      upimutex
                  1043fc70        0               0
  0x3000ab09b48:  nupinest
                  0
  0x3000ab09b50:  delay_lock
  0x3000ab09b50:  owner/waiters
                  0
  0x3000ab09b58:  unpark  thlink
                  0       0
  0x30003932bc0:  link            stk             startpc
                  0               2a1003bdaf0     0
  0x30003932bd8:  bound_cpu       affinitycnt     bind_cpu
                  0               0               -1
  0x30003932be4:  flag    proc_flag       schedflag
                  2       0               0
  0x30003932bea:  preempt preempt_lk      state
                  0       0               1
  0x30003932bf0:  pri     epri
                  29      0
  0x30003932bf8:
                  pc              sp
                  10079ed4        2a1003bcfb1
  0x30003932c08:  wchan0          wchan           sobj_ops
                  0               3000a4dbafc     1042e238
  0x30003932c20:  cid             clfuncs         cldata
                  1               10464fc8        30009f9d418
  0x30003932c38:  ctx             lofault         onfault
                  300035450e0     0               0
  0x30003932c50:  ontrap          swap            lock
                  0               2a1003ba000     0
  0x30003932c62:  pil     pi_lock cpu
                  0       0       1041b428
  0x30003932c70:  intr            did             tnf_tpdp
                  0               9778913         30005b0f7f0
  0x30003932c88:  tid             waitfor         alarmid
                  17              -1              0
  0x30003932c98:  realitimer
  0x30003932c98:  interval.tv_sec interval.tv_usec        value.tv_sec
                  0               0                       0
  0x30003932cb0:  value.tv_usec
  ...
The line starting with '0x30003932be4' has the value of 'schedflag' set to 0. The address for this thread can be found at the start of the line 4 lines up in the line which also contains the flag 'link'. Fortunately for us the distance back to the 'link' address is always the same, namely '0x24' (this easies the job of automating the process quite a lot). This means that the address to use in the next step can be written as '0x30003932bc0' or '0x30003932be4-0x24'.
(Step #2)
Now we fetch the 'procp' value from the 'thread.brief' output like this:   # mdb -k
  Loading modules: [ unix krtld genunix ip ptm cpc ipc random nfs ]
  > 0x30003932be4-0x24 $
The 'procp' has a value of '3000c277568'. This value is needed for our next lookup.
(Step #3)
Still from inside the same Module Debugger session, fetch the output of 'proc' using the 'procp' value of '3000c277568':   > 3000c277568 $
(Step #4)
Looking at the above ouput we can see that the value of 'pidp' is '300005bc440'. Using this value we can finally find the PIDs which this thread belongs to.   > 300005bc440 $
(Step #5)
Viola, the PID '26125' is one the swapped processes!   # ps -ef | grep [2]6125
   precise 26125     1  0   May 23 ?       978:20 ./products/gui/../../java/1.4.2/JRE/bin/java -cp java/1.3.1/lib/psi3I3FP.jar:ja
Perl Script to automate this
To follow the above process gets tedious pretty quickly so I wrote a Perl script to automate the process. Script improvements are very welcome!
Download script [
swapped.plx
]
Thanks
A big thank you to the SUN engineer Michael Schuster who was very patient and friendly helping me with this problem!


本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u/12631/showart_185202.html