请教一个问题：F4800宕机原因分析

dawningzh

我这儿有一台F4800，近期经常宕机，有些规律，只要执行一个Q3程序，系统就会宕机，让人头痛。看了看日志，看不太懂，好象是线程之间互相死锁，也不知道是硬件故障还是程序问题，更不知道怎么解决。请大家帮忙瞧瞧。日志详见下面：
......
Feb  2 17:51:18 sxmd6 unix: [ID 836849 kern.notice]
Feb  2 17:51:18 sxmd6 ^Mpanic[cpu11]/thread=300122826e0:
Feb  2 17:51:18 sxmd6 unix: [ID 783603 kern.notice] Deadlock: cycle in blocking chain
Feb  2 17:51:18 sxmd6 unix: [ID 100000 kern.notice]
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191070 genunix:turnstile_block+360 (300122826e0, 0, 10420808, 30011744030, 0, 300137b7d10)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 000000001004980c 0000030011496420 00000300124dd680 0000000000000000
Feb  2 17:51:18 sxmd6   %l4-7: 0000000010420808 0000000000000000 00000300117f5140 00000300124dd680
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191140 unix:mutex_vector_enter+3b8 (300137b7d10, 104266d0, 104647e0, 300000c0000, 0, 1d2)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: ffff14f051a593d0 00000300137b7d10 000003000574aab0 0000000000000640
Feb  2 17:51:18 sxmd6   %l4-7: 00000300124dd680 000003000574aab0 00000300117f5141 000002a102191ba0
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191200 genunix:set_qend+c (300137b7d10, 300137b7ca8, 30012e2d128, 30012e2d1a8, 30012e2d128, 300137de4d0)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000000 0000000000000000 000003001389d550 0000000000007fff
Feb  2 17:51:18 sxmd6   %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a1021912b0 genunix:removeq+230 (0, 300137de310, 300137de4d0, 30012e2d128, 3000bf60810, 300137de3f0)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000001 000003000a9e1378 0000000000010000 0000000000000000
Feb  2 17:51:18 sxmd6   %l4-7: 000000007846e908 0000000000000000 0000000000000000 0000000000000000
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191360 genunix:link_rempassthru+10 (300137de310, 300122826e0, 20, 3000a2b2cd0, 30010a556e0, 10)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000040 0000030010a55658 000002a102191558 0000000000000040
Feb  2 17:51:18 sxmd6   %l4-7: 0000030010a556d8 0000000000003a98 000003000a248d80 0000000000000000
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191410 genunix:mlink+40c (5000, 0, 104f04e0, ffffffff, 300106f7d88, 2a102191aec)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000030010a556d8 0000030008c57b80 0000030010a55658 000000000000530c
Feb  2 17:51:18 sxmd6   %l4-7: 0000000000000005 0000030012e2d128 00000300137b7ca8 0000000000002000
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a102191570 genunix:strioctl+2cc4 (30010a556d8, 0, 2a102191aec, 3000a2b2cd0, 1, 300135b2910)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000000000000005 0000030008c57b80 000000000000530c 0000000000202003
Feb  2 17:51:18 sxmd6   %l4-7: 0000030010a556d8 0000030010a55658 000002a102191aec 00000300135b2830
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a1021918e0 specfs:spec_ioctl+8c (30008c57b80, 530c, 5, 202003, 3000a2b2cd0, 2a102191aec)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000004482001204 0000000000000016 0000000000000000 0000000000000004
Feb  2 17:51:18 sxmd6   %l4-7: 0000000000000000 0000030005968000 00000300122826e0 000002a102191ba0
Feb  2 17:51:18 sxmd6 genunix: [ID 723222 kern.notice] 000002a1021919a0 genunix:ioctl+1e4 (300138737e0, 4, 5, ffffffff7eeb6f60, 530c, 0)
Feb  2 17:51:18 sxmd6 genunix: [ID 179002 kern.notice]   %l0-3: 0000000010160600 0000000000000000 0000000000000000 0000000000000000
Feb  2 17:51:18 sxmd6   %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Feb  2 17:51:18 sxmd6 unix: [ID 100000 kern.notice]
Feb  2 17:51:18 sxmd6 genunix: [ID 672855 kern.notice] syncing file systems...
Feb  2 17:51:21 sxmd6 genunix: [ID 733762 kern.notice]  162
Feb  2 17:51:23 sxmd6 genunix: [ID 733762 kern.notice]  110
Feb  2 17:51:25 sxmd6 genunix: [ID 733762 kern.notice]  83
Feb  2 17:52:00 sxmd6 last message repeated 20 times
Feb  2 17:52:01 sxmd6 genunix: [ID 622722 kern.notice]  done (not all i/o completed)
Feb  2 17:52:02 sxmd6 genunix: [ID 353387 kern.notice] dumping to /dev/dsk/c0t0d0s1, offset 859111424
Feb  2 17:53:02 sxmd6 genunix: [ID 409368 kern.notice] ^M100% done: 142029 pages dumped, compression ratio 3.02,
Feb  2 17:53:02 sxmd6 genunix: [ID 851671 kern.notice] dump succeeded
Feb  2 17:56:53 sxmd6 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.8 Version Generic_117350-16 64-bit
Feb  2 17:56:53 sxmd6 genunix: [ID 913632 kern.notice] Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.
Feb  2 17:56:53 sxmd6 genunix: [ID 678236 kern.info] Ethernet address = 0:3:ba:4:1e:ec
......

谢谢啊！
祝DJ新年快乐！

abcU

建议先作系统最大化监测 来确定是不是硬件问题，假如不是硬件问题，那么应该考虑是不是软件兼容性问题？

ga0feng

运行这个程序，立刻察看 mpstat 5的输出的第smtx（这个就是死锁）项，看是不是很快的增大？
一般的如果机子的内存和cpu很强的话，这个值肯定到30是没关系的。像我的Fire v120,一般的时候smtx都是0。

然后有一个命令是lockstat，你man一下，我也不太会用。


如果问题以后解决了，无论如何请告诉我们解决的方法，让大家都学习一下。

sophia_dotty

1\如果系统每次都报cpu11出现panic错,那就是此cpu 硬件有问题,可以考虑把他拔出来，让其他几个CPU工作,看是否还出问题。
2\ 如果每次都是在Q3运行的时候出问题，那应该是软件的问题，
3\如果以前运行Q3都没问题的话,那还是要怀疑是CPU硬件问题.

Philmoon

也可能是内存。尤其是供CPU11的内存条。看看messages中有没有关于内存的报错记录。

sophia_dotty

楼上的,会分析crash 文件吗?

june_cai

把补丁打到最新的看看，另外，如果以前运行该程序没有问题的话，那么就应该是硬件的问题。

lisuit

99% is a software problem. try to update your patch level.
If problem still exists, let program developer to debug your application.

dawningzh

重新安装了Q3软件，然后再运行，系统不再crash。前几次crash时，日志显示都是因为CPU11的线程死锁，我想找个机会，装个CAT软件再分析一下前几次的core文件。