IBM P750 报错误代码3D32B80D

szx_oracle

# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
3D32B80D   0815073414 P S topsvcs        NIM thread blocked
3D32B80D   0815073414 P S topsvcs        NIM thread blocked
3D32B80D   0815020614 P S topsvcs        NIM thread blocked
3D32B80D   0815020614 P S topsvcs        NIM thread blocked
3D32B80D   0814230814 P S topsvcs        NIM thread blocked
3D32B80D   0814224014 P S topsvcs        NIM thread blocked
3D32B80D   0814193414 P S topsvcs        NIM thread blocked
3D32B80D   0814184914 P S topsvcs        NIM thread blocked
3D32B80D   0814183514 P S topsvcs        NIM thread blocked
3D32B80D   0814183514 P S topsvcs        NIM thread blocked


# errpt -aj 3D32B80D |more
-------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Fri Aug 15 07:34:26 GMT+08:00 2014
Sequence Number: 1799390
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.0SIvH/5t/0ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          11
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Fri Aug 15 07:34:24 GMT+08:00 2014
Sequence Number: 1799389
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx..SIvH/W7L.ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
receive thread
Interval in seconds during which process was blocked
           9
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Fri Aug 15 02:06:55 GMT+08:00 2014
Sequence Number: 1799388
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.zeDvH/I/f/ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          13
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Fri Aug 15 02:06:52 GMT+08:00 2014
Sequence Number: 1799387
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.weDvH/qDt/ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
receive thread
Interval in seconds during which process was blocked
          10
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 23:08:22 GMT+08:00 2014
Sequence Number: 1799386
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.a1BvH//05/ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          11
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 22:40:28 GMT+08:00 2014
Sequence Number: 1799385
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.QdAvH/wDp.ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          13
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 19:34:17 GMT+08:00 2014
Sequence Number: 1799384
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.tu7vH/9MI.ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          13
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 18:49:05 GMT+08:00 2014
Sequence Number: 1799383
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward
Standard input
        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.VE7vH/N/Z0ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          12
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 18:35:11 GMT+08:00 2014
Sequence Number: 1799382
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.T17vH/Rwx.ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
send thread
Interval in seconds during which process was blocked
          11
Interface name
rhdiskpower0
---------------------------------------------------------------------------
LABEL:          TS_NIM_ERROR_STUCK_
IDENTIFIER:     3D32B80D

Date/Time:       Thu Aug 14 18:35:08 GMT+08:00 2014
Sequence Number: 1799381
Machine Id:      00F6FEE74C00
Node Id:         I_LG_BFS_XZ1
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   topsvcs         

Description
NIM thread blocked

Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
The system clock was set forward

User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
The system clock was manually set forward

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O

        Recommended Actions
        Examine I/O and memory activity on the system
        Reduce load on the system
        Tune virtual memory parameters
        Call IBM Service if problem persists

Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.41,7916            
ERROR ID
6BUfAx.Q17vH/1EG1ZF4N.1...................
REFERENCE CODE
                                          
Thread which was blocked
receive thread
Interval in seconds during which process was blocked
          10
Interface name
rhdiskpower0

szx_oracle

# vmstat -v
              8093696 memory pages
              7825376 lruable pages
               135321 free pages
                    8 memory pools
              1467902 pinned pages
                 80.0 maxpin percentage
                  3.0 minperm percentage
                 90.0 maxperm percentage
                 71.5 numperm percentage
              5600292 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 71.5 numclient percentage
                 90.0 maxclient percentage
              5600292 client pages
                    0 remote pageouts scheduled
                    0 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2228 filesystem I/Os blocked with no fsbuf
                 1237 client filesystem I/Os blocked with no fsbuf
                 1312 external pager filesystem I/Os blocked with no fsbuf
                 29.1 percentage of memory used for computational pages

是否应该调优?

szx_oracle

有没有那位大侠给个建议啊！

orian

可能太忙了，io有block，os是多少？据说调io pacing可以缓解，让io线程轮换得快些，不过我不知道效果。

还有的是NTP的问题，时间同步时乱跳也有影响，遇到过几次。

Deafa

楼主是不是装了HACMP

szx_oracle

HACMP 的版本6.1.0.7

Deafa

回复 6# szx_oracle
我的机器也有这个问题。问了800，说这个是ha的心跳的一个information。也不属于报错。