sybase 异常宕机

wxr111

我的是aix5。2系统 安装了sybase 补丁打到11335  昨天业务量很大 结果发现宕机了 日志如下
02:00000:00600:2006/11/22 19:11:12.88 kernel  Spinlock Resource->rdesmgr_spin at address 00000001602e8580 owned by 1260192307
02:00000:00600:2006/11/22 19:11:13.04 kernel  End of spinlock display.
01:00000:00346:2006/11/22 19:11:13.04 kernel  ************************************
01:00000:00346:2006/11/22 19:11:13.05 kernel  SQL causing error : 0009999'  ) or  (SJ.F_DWBH between '900310010000' and '900320009999'  )  ))  
01:00000:00346:2006/11/22 19:11:13.05 kernel  ************************************
01:00000:00346:2006/11/22 19:11:13.05 server  SQL Text: 0009999'  ) or  (SJ.F_DWBH between '900310010000' and '900320009999'  )  ))  
01:00000:00346:2006/11/22 19:11:13.06 kernel  curdb = 5 pstat = 0x10100 lasterror = 0
01:00000:00346:2006/11/22 19:11:13.06 kernel  preverror = 0 transtate = 0
01:00000:00346:2006/11/22 19:11:13.06 kernel  curcmd = 195 program = LSCW                          
01:00000:00346:2006/11/22 19:11:13.10 kernel  pc: 0x0000000100355e80 .pcstkwalk+0x88()
02:00000:00600:2006/11/22 19:11:13.10 kernel  pc: 0x0000000100355e80 .pcstkwalk+0x88()
01:00000:00346:2006/11/22 19:11:13.10 kernel  pc: 0x0000000100356538 .ucstkgentrace+0x238()
02:00000:00600:2006/11/22 19:11:13.10 kernel  pc: 0x0000000100356538 .ucstkgentrace+0x238()
01:00000:00346:2006/11/22 19:11:13.10 kernel  pc: 0x00000001003552c8 .ucbacktrace+0x78()
02:00000:00600:2006/11/22 19:11:13.13 kernel  pc: 0x00000001003552c8 .ucbacktrace+0x78()
02:00000:00600:2006/11/22 19:11:13.18 kernel  pc: 0x00000001002225c0 .terminate_process[3]+0x2c()
01:00000:00346:2006/11/22 19:11:13.18 kernel  pc: 0x0000000100222d90 .terminate_process[3]+0x7fc()
01:00000:00346:2006/11/22 19:11:13.19 kernel  pc: 0x000000010059a5fc .kaclkintr[10]+0x15c()
02:00000:00600:2006/11/22 19:11:13.18 kernel  pc: 0x000000010059a5fc .kaclkintr[10]+0x15c()
01:00000:00346:2006/11/22 19:11:13.19 kernel  pc: 0x0000000000000001 ()
02:00000:00600:2006/11/22 19:11:13.18 kernel  pc: 0x0000000177a6d8f8 ()
01:00000:00346:2006/11/22 19:11:13.19 kernel  pc: 0x0000000100042214 .pg_allocate+0x40()
02:00000:00600:2006/11/22 19:11:13.19 kernel  pc: 0x00000001000910ac .des__grab+0x28()
01:00000:00346:2006/11/22 19:11:13.19 kernel  pc: 0x00000001000783a0 .pg__alloc_newpage+0x1e0()
02:00000:00600:2006/11/22 19:11:13.19 kernel  pc: 0x00000001000800e0 .des_get[6]+0x1c()
02:00000:00600:2006/11/22 19:11:13.20 kernel  pc: 0x0000000100035408 .parse_name+0x2fc()
02:00000:00600:2006/11/22 19:11:13.20 kernel  pc: 0x0000000100035ebc .recvchars[1]+0x110()
01:00000:00346:2006/11/22 19:11:13.20 kernel  [Handler pc: 0x000000010071dc74 .pg_getdatapage_forinsert_lkbit_hdlr installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.21 kernel  pc: 0x00000001000a6ed8 .getid+0x5c()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x00000001000725a8 .pg_getdatapage_forinsert[4]+0xb0()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x000000010002a2e8 .dol_data_insert[1]+0x18()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x000000010002aa44 .dol_insert+0x1c4()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x000000010002acd8 .ins_insert[5]+0x8()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x000000010001e42c +0x98()
01:00000:00346:2006/11/22 19:11:13.22 kernel  pc: 0x000000010001e340 .run[22]+0x48()
01:00000:00346:2006/11/22 19:11:13.23 kernel  pc: 0x000000010000b9c4 .ind__ides_get[1]+0x40()
01:00000:00346:2006/11/22 19:11:13.23 kernel  pc: 0x000000010001932c .exec_eop[6]+0x4()
01:00000:00346:2006/11/22 19:11:13.23 kernel  pc: 0x000000010001932c .exec_eop[6]+0x4()
01:00000:00346:2006/11/22 19:11:13.23 kernel  pc: 0x000000010001932c .exec_eop[6]+0x4()
01:00000:00346:2006/11/22 19:11:13.23 kernel  [Handler pc: 0x00000001003bc4a4 .execerr installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.24 kernel  [Handler pc: 0x000000010017858c .droperr installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.25 kernel  pc: 0x00000001001162b4 .dropo[3]+0x3c()
02:00000:00600:2006/11/22 19:11:13.25 kernel  pc: 0x0000000100116188 .dropobject[1]+0x4c()
01:00000:00346:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x00000001004103c4 .jcsExHandler installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.26 kernel  pc: 0x000000010065b200 .s_execute[67]+0x3e7c()
02:00000:00600:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x0000000100645230 .s_handle installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.26 kernel  pc: 0x0000000100037260 .sequencer[1]+0x174()
02:00000:00600:2006/11/22 19:11:13.26 kernel  pc: 0x000000010007335c .tdsrecv_language[1]+0x58()
02:00000:00600:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x00000001002233dc .hdl_backout installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x000000010020a784  installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x00000001003bc4a4 .execerr installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.26 kernel  [Handler pc: 0x00000001003bc4a4 .execerr installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x00000001000732ec .result[4]+0x6c()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000100015970 +0x1c()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x000000010001594c .kpinitproc[1]+0x1c()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x00000001004478cc .aritherr installed by the following function:-]
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
02:00000:00600:2006/11/22 19:11:13.27 kernel  end of stack trace, spid 600, kpid 1260192307, suid 2638
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000100007f54 .execute[1]+0x1c()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000100012108 .s_execute[7]+0x84()
01:00000:00346:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x0000000100645230 .s_handle installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000100037260 .sequencer[1]+0x174()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x000000010007335c .tdsrecv_language[1]+0x58()
01:00000:00346:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x00000001002233dc .hdl_backout installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.27 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x00000001000732ec .result[4]+0x6c()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000100015970 +0x1c()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x000000010001594c .kpinitproc[1]+0x1c()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  pc: 0x0000000000000000 ()
01:00000:00346:2006/11/22 19:11:13.27 kernel  end of stack trace, spid 346, kpid 1261240563, suid 479
02:00000:00600:2006/11/22 19:11:13.27 kernel  ueshutdown: exiting
00:00000:00007:2006/11/22 19:11:34.09 kernel  timeslice -501, current process infected
00:00000:00007:2006/11/22 19:11:34.09 kernel  ************************************
00:00000:00007:2006/11/22 19:11:34.09 kernel  SQL causing error :
00:00000:00007:2006/11/22 19:11:34.09 kernel  ************************************
00:00000:00007:2006/11/22 19:11:34.09 kernel  curdb = 16 pstat = 0x400200 lasterror = 0
00:00000:00007:2006/11/22 19:11:34.09 kernel  preverror = 0 transtate = 0
00:00000:00007:2006/11/22 19:11:34.09 kernel  curcmd = 235 program =                              
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0x0000000100355e80 .pcstkwalk+0x88()
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0x0000000100356538 .ucstkgentrace+0x238()
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0x00000001003552c8 .ucbacktrace+0x78()
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0x0000000100222d90 .terminate_process[3]+0x7fc()
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0x000000010059a5fc .kaclkintr[10]+0x15c()
00:00000:00007:2006/11/22 19:11:34.10 kernel  pc: 0xffffffffffffffff ()
00:00000:00007:2006/11/22 19:11:34.10 kernel  [Handler pc: 0x0000000100223390 .hdl_backout_msg installed by the following function:-]
00:00000:00007:2006/11/22 19:11:34.13 kernel  pc: 0x00000001000b2438 .checkpoint[1]+0x240()
00:00000:00007:2006/11/22 19:11:34.14 kernel  [Handler pc: 0x00000001006f784c .dpdb_handle installed by the following function:-]
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x00000001000a308c .dmpx__truncatemain+0x828()
00:00000:00007:2006/11/22 19:11:34.14 kernel  [Handler pc: 0x000000010089a934  installed by the following function:-]
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x00000001000f3778 .j_adjust_cachesize[15]+0x20()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000100114358 .ckptproc[1]+0x18()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000100015970 +0x1c()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x000000010001594c .kpinitproc[1]+0x1c()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  pc: 0x0000000000000000 ()
00:00000:00007:2006/11/22 19:11:34.14 kernel  end of stack trace, spid 7, kpid 1441814, suid 0
01:00000:00477:2006/11/22 19:12:03.41 kernel  timeslice -501, current process infected
01:00000:00477:2006/11/22 19:12:03.41 kernel  ************************************
01:00000:00477:2006/11/22 19:12:03.41 kernel  SQL causing error : ccur51

01:00000:00477:2006/11/22 19:12:03.41 kernel  ************************************
01:00000:00477:2006/11/22 19:12:03.41 server  SQL Text: ccur51

01:00000:00477:2006/11/22 19:12:03.41 kernel  curdb = 5 pstat = 0x10000 lasterror = 0
01:00000:00477:2006/11/22 19:12:03.41 kernel  preverror = 0 transtate = 0
01:00000:00477:2006/11/22 19:12:03.41 kernel  curcmd = 0 program = LSCW                          
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000100355e80 .pcstkwalk+0x88()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000100356538 .ucstkgentrace+0x238()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x00000001003552c8 .ucbacktrace+0x78()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000100222d90 .terminate_process[3]+0x7fc()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x000000010059a5fc .kaclkintr[10]+0x15c()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000000008000 ()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000100035408 .parse_name+0x2fc()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x0000000100035ebc .recvchars[1]+0x110()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x000000010005f180 .get_range_sdes[2]+0x20()
01:00000:00477:2006/11/22 19:12:03.41 kernel  pc: 0x000000010005f1dc .get_range_sdes[2]+0x7c()
01:00000:00477:2006/11/22 19:12:03.42 kernel  pc: 0x0000000100051e10 .col_open_range+0x48()
01:00000:00477:2006/11/22 19:12:03.42 kernel  pc: 0x0000000100051ea4 .colnames_driver+0x3c()
01:00000:00477:2006/11/22 19:12:03.42 kernel  pc: 0x0000000100051fa0 .s_normquery+0x74()
01:00000:00477:2006/11/22 19:12:03.42 kernel  pc: 0x0000000100052eb8 .s_normalize[2]+0x6c()
01:00000:00477:2006/11/22 19:12:03.42 kernel  [Handler pc: 0x0000000100645230 .s_handle installed by the following function:-]
01:00000:00477:2006/11/22 19:12:03.42 kernel  pc: 0x000000010006f934 .sequencer[2]+0x48()
01:00000:00477:2006/11/22 19:12:03.43 kernel  pc: 0x00000001003ea460 .tdsrecv_declare[1]+0x6ec()
01:00000:00477:2006/11/22 19:12:03.43 kernel  [Handler pc: 0x00000001002233dc .hdl_backout installed by the following function:-]
01:00000:00477:2006/11/22 19:12:03.43 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
01:00000:00477:2006/11/22 19:12:03.43 kernel  [Handler pc: 0x0000000100453e78 .ut_handle installed by the following function:-]
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x00000001001da6e8 .conn_hdlr[8]+0xa0()
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x0000000100015970 +0x1c()
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x000000010001594c .kpinitproc[1]+0x1c()
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x0000000000000000 ()
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x0000000000000000 ()
01:00000:00477:2006/11/22 19:12:03.45 kernel  pc: 0x0000000000000000 ()

以前也有过这个情况 感觉是系统内存小 就分了3G内存给系统 sybase 7G 好了一段时间   现在又出现了  什么问题啊

echoaix

感觉因该是cpu处理能力不够，应该不是宕机，而是没有相应

Blackrose

几个建议：
1. 通过sysmon观察你的cache partition的spinlock情况；
2. 如果有更新的补丁就安装；
3. 可以适当增加 "cpu grace time"参数到1000
4. 对造成此故障的语句尽可能的进行优化