HP-UNIX+oracle报RA-27300: 操作系统系统相关操作: fork 失败, 求高人解答

cycwll

HPRX7640   物理内存32G
oracle：  compatible               = 10.2.0.3.0

刚刚启动时swapinfo  user86%。之后持续增长，持续3小时左右变为98%后oracle开始报错。

umpp-db3>swapinfo -amt
             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev        8192       0    8192    0%       0       -    1  /dev/vg00/lvol2
reserve       -    8192   -8192
memory    31040   29240    1800   94%
total     39232   37432    1800   95%       -       0    -


95% 状态下session291。98%的状态是seesion370左右。
SQL> select count(*) from v$session;

  COUNT(*)
----------
       291


oracle启动后持续3个小时就会报如下错误。


Errors in file /oracle/app/admin/umppdb/udump/umppdb_ora_17693.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-12414: 内部 LBAC 错误: zllcsi:OCIStmtExecute
错误: ORA-04030: 在尝试分配 63904 字节 (kxs-heap-f,frame segment) 时进程内存不足
ORA-06512: 在 "LBACSYS.LBAC_CACHE", line 99
ORA-06512: 在 "LBACSYS.LBAC_EVENTS", line 137
ORA-06512: 在 line 2
Thu Jun 14 08:00:25 2012
Thread 1 advanced to log sequence 23802 (LGWR switch)
  Current log# 3 seq# 23802 mem# 0: /oradata/umppdb/redo03.log
  Current log# 3 seq# 23802 mem# 1: /oradata/umppdb/redo13.log
Thu Jun 14 08:43:24 2012
Thread 1 advanced to log sequence 23803 (LGWR switch)
  Current log# 2 seq# 23803 mem# 0: /oradata/umppdb/redo02.log
  Current log# 2 seq# 23803 mem# 1: /oradata/umppdb/redo12.log
Thu Jun 14 09:27:56 2012
Thread 1 advanced to log sequence 23804 (LGWR switch)
  Current log# 1 seq# 23804 mem# 0: /oradata/umppdb/redo01.log
  Current log# 1 seq# 23804 mem# 1: /oradata/umppdb/redo11.log
Thu Jun 14 10:00:57 2012
Process startup failed, error stack:
Thu Jun 14 10:00:57 2012
Errors in file /oracle/app/admin/umppdb/bdump/umppdb_psp0_7880.trc:
ORA-27300: 操作系统系统相关操作: fork 失败, 状态为: 11
ORA-27301: 操作系统故障消息: Resource temporarily unavailable
ORA-27302: 错误发生在: skgpspawn5
ORA-27303: 附加信息: skgpspawn5
alert_umppdblog.txt (92%)Thu Jun 14 10:00:58 2012
Process m000 died, see its trace file
Thu Jun 14 10:00:58 2012
ksvcreate: Process(m000) creation failed
Thu Jun 14 10:01:58 2012
Process startup failed, error stack:
Thu Jun 14 10:01:58 2012
Errors in file /oracle/app/admin/umppdb/bdump/umppdb_psp0_7880.trc:
ORA-27300: 操作系统系统相关操作: fork 失败, 状态为: 11
ORA-27301: 操作系统故障消息: Resource temporarily unavailable
ORA-27302: 错误发生在: skgpspawn5
ORA-27303: 附加信息: skgpspawn5
Thu Jun 14 10:01:59 2012
Process m000 died, see its trace file
Thu Jun 14 10:01:59 2012
ksvcreate: Process(m000) creation failed
Thu Jun 14 10:02:59 2012
Process startup failed, error stack:
Thu Jun 14 10:02:59 2012
Errors in file /oracle/app/admin/umppdb/bdump/umppdb_psp0_7880.trc:
ORA-27300: 操作系统系统相关操作: fork 失败, 状态为: 11
ORA-27301: 操作系统故障消息: Resource temporarily unavailable
ORA-27302: 错误发生在: skgpspawn5
ORA-27303: 附加信息: skgpspawn5
Thu Jun 14 10:03:00 2012
Process m000 died, see its trace file
Thu Jun 14 10:03:00 2012
ksvcreate: Process(m000) creation failed
Thu Jun 14 10:04:00 2012
Process startup failed, error stack:
Thu Jun 14 10:04:00 2012
Errors in file /oracle/app/admin/umppdb/bdump/umppdb_psp0_7880.trc:
ORA-27300: 操作系统系统相关操作: fork 失败, 状态为: 11
ORA-27301: 操作系统故障消息: Resource temporarily unavailable
ORA-27302: 错误发生在: skgpspawn5
ORA-27303: 附加信息: skgpspawn5
Thu Jun 14 10:04:01 2012
Process m000 died, see its trace file
Thu Jun 14 10:04:01 2012
ksvcreate: Process(m000) creation failed



oracle信息
System parameters with non-default values:
  processes                = 600
  sessions                 = 665
  sga_max_size             = 20619198464
  __shared_pool_size       = 2315255808
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  nls_language             = SIMPLIFIED CHINESE
  nls_territory            = CHINA
  disk_asynch_io           = FALSE
  sga_target               = 20619198464
  control_files            = /oradata/umppdb/control01.ctl, /oradata/umppdb/control02.ctl, /oradata/umppdb/control03.ctl
  db_block_size            = 8192
  __db_cache_size          = 18220056576
  compatible               = 10.2.0.3.0
  log_archive_dest         = /orabackup/rman
  log_buffer               = 31457280
  db_file_multiblock_read_count= 128
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=umppdbXDB)
  job_queue_processes      = 10
  cursor_sharing           = SIMILAR
  background_dump_dest     = /oracle/app/admin/umppdb/bdump
  user_dump_dest           = /oracle/app/admin/umppdb/udump
  core_dump_dest           = /oracle/app/admin/umppdb/cdump
  audit_file_dest          = /oracle/app/admin/umppdb/adump
  db_name                  = umppdb
  open_cursors             = 300
  optimizer_index_cost_adj = 30
  optimizer_index_caching  = 90
  pga_aggregate_target     = 5153751040

InfoSVC

唉,你这是个oracle的问题吧

InfoSVC

贴一下报错的trc文件呢

cycwll

# more /oracle/app/admin/umppdb/udump/umppdb_ora_17693.trc
/oracle/app/admin/umppdb/udump/umppdb_ora_17693.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining Scoring Engine
and Real Application Testing options
ORACLE_HOME = /oracle/app/product/10.2.0
System name:    HP-UX
Node name:      umpp-db3
Release:        B.11.31
Version:        U
Machine:        ia64
Instance name: umppdb
Redo thread mounted by this instance: 1
Oracle process number: 411
Unix process pid: 17693, image: oracle@umpp-db3

*** ACTION NAME) 2012-06-14 07:40:11.360
*** MODULE NAMEruby@ZB-UMPP-APP-01 (TNS V1-V3)) 2012-06-14 07:40:11.360
*** SERVICE NAMEumppdb) 2012-06-14 07:40:11.360
*** SESSION ID440.1382 2012-06-14 07:40:11.360
Skipped error 604 during the execution of LBACSYS.LBAC$LOGON
*** 2012-06-14 07:40:11.373
ksedmp: internal or fatal error
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-12414: 内部 LBAC 错误: zllcsi:OCIStmtExecute
错误: ORA-04030: 在尝试分配 63904 字节 (kxs-heap-f,frame segment) 时进程内存不足
ORA-06512: 在 "LBACSYS.LBAC_CACHE", line 99
ORA-06512: 在 "LBACSYS.LBAC_EVENTS", line 137
ORA-06512: 在 line 2
umppdb_ora_17693.trc: 结束