- 论坛徽章:
- 0
|
OS: CentOS 5.3
Linux CentOS2.localdomain 2.6.18-164.el5 #1 SMP Thu Sep 3 03:33:56 EDT 2009 i686 i686 i386 GNU/Linux
常运行程序:
fail2ban
mldonkey
hadoop
某次在putty操作时发现终端错误信息,还有运行hadoop dfs命令后JVM出错,
Linux死机时该看哪些东西?
CentOS 死机 目前看到两处:
1. kernel 的错误信息,mldonkey导致的。
2.JVM挂起,hadoop开始可能配置错误,分配的内存过多了。
- Message from syslogd@ at Mon Sep 28 21:39:25 2009 ...
- CentOS2 kernel: Bad page state in process 'mlnet'
- Message from syslogd@ at Mon Sep 28 21:39:25 2009 ...
- CentOS2 kernel: page:c118dfe0 flags:0x80000000 mapping:00000000 mapcount:-64 count:0 (Not tainted)
- Message from syslogd@ at Mon Sep 28 21:39:25 2009 ...
- CentOS2 kernel: Trying to fix it up, but a reboot is needed
- Message from syslogd@ at Mon Sep 28 21:39:25 2009 ...
- CentOS2 kernel: Backtrace:
- [hadoop@CentOS2 logs]$
复制代码
- [hadoop@CentOS2 ~]$ hadoop/bin/hadoop dfs -put file01 input
- [hadoop@CentOS2 ~]$ hadoop/bin/hadoop dfs -put file02 input
- #
- # A fatal error has been detected by the Java Runtime Environment:
- #
- # SIGSEGV (0xb) at pc=0x4fffffff, pid=3042, tid=3031935888
- #
- # JRE version: 6.0_16-b01
- # Java VM: Java HotSpot(TM) Client VM (14.2-b01 mixed mode, sharing linux-x86 )
- # Problematic frame:
- # C 0x4fffffff
- #
- # An error report file with more information is saved as:
- # /home/hadoop/hs_err_pid3042.log
- #
- # If you would like to submit a bug report, please visit:
- # [url]http://java.sun.com/webapps/bugreport/crash.jsp[/url]
- #
- Aborted
- [hadoop@CentOS2 ~]$
- [hadoop@CentOS2 ~]$
- [hadoop@CentOS2 ~]$ ls
- file01 file02 hadoop hdfs hs_err_pid3042.log mapred
- [hadoop@CentOS2 ~]$ cat hs_err_pid3042.log
- #
- # A fatal error has been detected by the Java Runtime Environment:
- #
- # SIGSEGV (0xb) at pc=0x4fffffff, pid=3042, tid=3031935888
- #
- # JRE version: 6.0_16-b01
- # Java VM: Java HotSpot(TM) Client VM (14.2-b01 mixed mode, sharing linux-x86 )
- # Problematic frame:
- # C 0x4fffffff
- #
- # If you would like to submit a bug report, please visit:
- # [url]http://java.sun.com/webapps/bugreport/crash.jsp[/url]
- #
- --------------- T H R E A D ---------------
- Current thread (0x0942d000): JavaThread "CompilerThread0" daemon [_thread_in_vm, id=3069, stack(0xb 4afa000,0xb4b7b000)]
- siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x4fffffff
- Registers:
- EAX=0xb4b795b8, EBX=0x00000000, ECX=0x00000001, EDX=0x00000000
- ESP=0xb4b795d0, EBP=0x47000000, ESI=0xb4b79d7f, EDI=0xb4b79980
- EIP=0x4fffffff, CR2=0x4fffffff, EFLAGS=0x00010246
- Top of Stack: (sp=0xb4b795d0)
- 0xb4b795d0: b4b79604 00000000 b4b79638 b78e7d6c
- 0xb4b795e0: b4b79980 4fffffff 00000000 00000000
- 0xb4b795f0: 0942dda4 b4b79fc0 09433530 b4b79610
- 0xb4b79600: 4fffffff 00000000 b7ba5d60 b4b79b64
- 0xb4b79610: 46000003 00000000 94150978 0000000e
- 0xb4b79620: 94151360 00000063 b4b79658 00000a00
- 0xb4b79630: b4b79980 00000003 b4b79848 b78e6bf3
- 0xb4b79640: b4b79980 00000001 00000000 00000a00
- Instructions: (pc=0x4fffffff)
- 0x4fffffef:
- [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb]
- Stack: [0xb4afa000,0xb4b7b000], sp=0xb4b795d0, free space=509k
- Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
- C 0x4fffffff
- [error occurred during error reporting (printing native stack), id 0xb]
- Current CompileTask:
- C1: 14 ! java.io.BufferedReader.readLine(Z)Ljava/lang/String; (304 bytes)
- --------------- P R O C E S S ---------------
- Java Threads: ( => current thread )
- 0x09430400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3070, stack(0xb4aa9000,0xb 4afa000)]
- =>0x0942d000 JavaThread "CompilerThread0" daemon [_thread_in_vm, id=3069, stack(0xb4afa000,0xb4b7b00 0)]
- 0x0942b800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3068, stack(0xb4b7b000,0xb4b cc000)]
- 0x0941ac00 JavaThread "Finalizer" daemon [_thread_blocked, id=3067, stack(0xb50aa000,0xb50fb000)]
- 0x09419800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3066, stack(0xb50fb000,0xb51 4c000)]
- 0x093f3800 JavaThread "main" [_thread_blocked, id=3064, stack(0xb76ba000,0xb770b000)]
- Other Threads:
- 0x09417c00 VMThread [stack: 0xb514c000,0xb51cd000] [id=3065]
- 0x09432000 WatcherThread [stack: 0xb4a28000,0xb4aa9000] [id=3071]
- VM state:not at safepoint (normal execution)
- VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
- [0x093f22f8] Heap_lock - owner thread: 0x093f3800
- Heap
- def new generation total 960K, used 89K [0x51150000, 0x51250000, 0x55e30000)
- eden space 896K, 2% used [0x51150000, 0x51156518, 0x51230000)
- from space 64K, 100% used [0x51240000, 0x51250000, 0x51250000)
- to space 64K, 0% used [0x51230000, 0x51230000, 0x51240000)
- tenured generation total 4096K, used 162K [0x55e30000, 0x56230000, 0x8f950000)
- the space 4096K, 3% used [0x55e30000, 0x55e589a8, 0x55e58a00, 0x56230000)
- compacting perm gen total 12288K, used 168K [0x8f950000, 0x90550000, 0x93950000)
- the space 12288K, 1% used [0x8f950000, 0x8f97a308, 0x8f97a400, 0x90550000)
- ro space 8192K, 74% used [0x93950000, 0x93f4a2a8, 0x93f4a400, 0x94150000)
- rw space 12288K, 59% used [0x94150000, 0x94867878, 0x94867a00, 0x94d50000)
- Dynamic libraries:
- 00149000-0015c000 r-xp 00000000 fd:00 3342405 /lib/libnsl-2.5.so
- 0015c000-0015d000 r-xp 00012000 fd:00 3342405 /lib/libnsl-2.5.so
- 0015d000-0015e000 rwxp 00013000 fd:00 3342405 /lib/libnsl-2.5.so
- 0015e000-00160000 rwxp 0015e000 00:00 0
- 00ab8000-00abf000 r-xp 00000000 fd:00 3342415 /lib/librt-2.5.so
- 00abf000-00ac0000 r-xp 00006000 fd:00 3342415 /lib/librt-2.5.so
- 00ac0000-00ac1000 rwxp 00007000 fd:00 3342415 /lib/librt-2.5.so
- 00ad7000-00af1000 r-xp 00000000 fd:00 3342339 /lib/ld-2.5.so
- 00af1000-00af2000 r-xp 00019000 fd:00 3342339 /lib/ld-2.5.so
- 00af2000-00af3000 rwxp 0001a000 fd:00 3342339 /lib/ld-2.5.so
- 00af5000-00af7000 r-xp 00000000 fd:00 3342422 /lib/libdl-2.5.so
- 00af7000-00af8000 r-xp 00001000 fd:00 3342422 /lib/libdl-2.5.so
- 00af8000-00af9000 rwxp 00002000 fd:00 3342422 /lib/libdl-2.5.so
- 00aff000-00c3d000 r-xp 00000000 fd:00 3342353 /lib/libc-2.5.so
- 00c3d000-00c3f000 r-xp 0013e000 fd:00 3342353 /lib/libc-2.5.so
- 00c3f000-00c40000 rwxp 00140000 fd:00 3342353 /lib/libc-2.5.so
- 00c40000-00c43000 rwxp 00c40000 00:00 0
- 00c45000-00c6a000 r-xp 00000000 fd:00 3342426 /lib/libm-2.5.so
- 00c6a000-00c6b000 r-xp 00024000 fd:00 3342426 /lib/libm-2.5.so
- 00c6b000-00c6c000 rwxp 00025000 fd:00 3342426 /lib/libm-2.5.so
- 00c6e000-00c81000 r-xp 00000000 fd:00 3342410 /lib/libpthread-2.5.so
- 00c81000-00c82000 r-xp 00012000 fd:00 3342410 /lib/libpthread-2.5.so
- 00c82000-00c83000 rwxp 00013000 fd:00 3342410 /lib/libpthread-2.5.so
- 00c83000-00c85000 rwxp 00c83000 00:00 0
- 08048000-08052000 r-xp 00000000 fd:02 115762 /opt/jdk1.6.0_16/bin/java
- 08052000-08053000 rwxp 00009000 fd:02 115762 /opt/jdk1.6.0_16/bin/java
- 093ed000-094d9000 rwxp 093ed000 00:00 0 [heap]
- 51150000-51250000 rwxp 51150000 00:00 0
- 51250000-55e30000 rwxp 51250000 00:00 0
- 55e30000-56230000 rwxp 55e30000 00:00 0
- 56230000-8f950000 rwxp 56230000 00:00 0
- 8f950000-90550000 rwxp 8f950000 00:00 0
- 90550000-93950000 rwxp 90550000 00:00 0
- 93950000-93f4b000 r-xs 00001000 fd:02 115465 /opt/jdk1.6.0_16/jre/lib/i386/client/classes.jsa
- 93f4b000-94150000 rwxp 93f4b000 00:00 0
- 94150000-94868000 rwxp 005fc000 fd:02 115465 /opt/jdk1.6.0_16/jre/lib/i386/client/classes.jsa
- 94868000-94d50000 rwxp 94868000 00:00 0
- 94d50000-94e2e000 rwxp 00d14000 fd:02 115465 /opt/jdk1.6.0_16/jre/lib/i386/client/classes.jsa
- 94e2e000-95150000 rwxp 94e2e000 00:00 0
- 95150000-95158000 r-xs 00df2000 fd:02 115465 /opt/jdk1.6.0_16/jre/lib/i386/client/classes.jsa
- 95158000-95550000 rwxp 95158000 00:00 0
- b49a5000-b49c9000 r-xs 0026b000 fd:02 1364148 /opt/hadoop-0.20.1/hadoop-0.20.1-core.jar
- b49c9000-b4a28000 r-xs 00ba3000 fd:02 115467 /opt/jdk1.6.0_16/lib/tools.jar
- b4a28000-b4a29000 --xp b4a28000 00:00 0
- b4a29000-b4aa9000 rwxp b4a29000 00:00 0
- b4aa9000-b4aac000 --xp b4aa9000 00:00 0
- b4aac000-b4afa000 rwxp b4aac000 00:00 0
- b4afa000-b4afd000 --xp b4afa000 00:00 0
- b4afd000-b4b7b000 rwxp b4afd000 00:00 0
- b4b7b000-b4b7e000 --xp b4b7b000 00:00 0
- b4b7e000-b4bff000 rwxp b4b7e000 00:00 0
- b4bff000-b4d95000 r-xs 02fb3000 fd:02 119193 /opt/jdk1.6.0_16/jre/lib/rt.jar
- b4d95000-b4d96000 r-xp 00cdd000 fd:00 2138016 /usr/lib/locale/locale-archive
- b4d96000-b4e6e000 r-xp 00596000 fd:00 2138016 /usr/lib/locale/locale-archive
- b4e6e000-b4eaa000 r-xp 00525000 fd:00 2138016 /usr/lib/locale/locale-archive
- b4eaa000-b50aa000 r-xp 00000000 fd:00 2138016 /usr/lib/locale/locale-archive
- b50aa000-b50ad000 --xp b50aa000 00:00 0
- b50ad000-b50fb000 rwxp b50ad000 00:00 0
- b50fb000-b50fe000 --xp b50fb000 00:00 0
- b50fe000-b514c000 rwxp b50fe000 00:00 0
- b514c000-b514d000 --xp b514c000 00:00 0
- b514d000-b51db000 rwxp b514d000 00:00 0
- b51db000-b51f5000 rwxp b51db000 00:00 0
- b51f5000-b51f8000 rwxp b51f5000 00:00 0
- b51f8000-b53c3000 rwxp b51f8000 00:00 0
- b53c3000-b53c4000 rwxp b53c3000 00:00 0
- b53c4000-b53e9000 rwxp b53c4000 00:00 0
- b53e9000-b53ec000 rwxp b53e9000 00:00 0
- b53ec000-b55b7000 rwxp b53ec000 00:00 0
- b55b7000-b55bd000 rwxp b55b7000 00:00 0
- b55bd000-b55d7000 rwxp b55bd000 00:00 0
- b55d7000-b55e5000 rwxp b55d7000 00:00 0
- b55e5000-b5662000 rwxp b55e5000 00:00 0
- b5662000-b56ea000 rwxp b5662000 00:00 0
- b56ea000-b7662000 rwxp b56ea000 00:00 0
- b7662000-b7671000 r-xp 00000000 fd:02 114734 /opt/jdk1.6.0_16/jre/lib/i386/libzip.so
- b7671000-b7673000 rwxp 0000e000 fd:02 114734 /opt/jdk1.6.0_16/jre/lib/i386/libzip.so
- b7673000-b767c000 r-xp 00000000 fd:00 3342375 /lib/libnss_files-2.5.so
- b767c000-b767d000 r-xp 00008000 fd:00 3342375 /lib/libnss_files-2.5.so
- b767d000-b767e000 rwxp 00009000 fd:00 3342375 /lib/libnss_files-2.5.so
- b7681000-b7686000 rwxp b7681000 00:00 0
- b7686000-b768e000 rwxs 00000000 fd:00 1966157 /tmp/hsperfdata_hadoop/3042
- b768e000-b7694000 r-xp 00000000 fd:02 114716 /opt/jdk1.6.0_16/jre/lib/i386/native_threads/libhpi .so
- b7694000-b7695000 rwxp 00006000 fd:02 114716 /opt/jdk1.6.0_16/jre/lib/i386/native_threads/libhpi .so
- b7695000-b76b8000 r-xp 00000000 fd:02 114730 /opt/jdk1.6.0_16/jre/lib/i386/libjava.so
- b76b8000-b76ba000 rwxp 00023000 fd:02 114730 /opt/jdk1.6.0_16/jre/lib/i386/libjava.so
- b76ba000-b76bd000 --xp b76ba000 00:00 0
- b76bd000-b770b000 rwxp b76bd000 00:00 0
- b770b000-b7b71000 r-xp 00000000 fd:02 114723 /opt/jdk1.6.0_16/jre/lib/i386/client/libjvm.so
- b7b71000-b7b91000 rwxp 00465000 fd:02 114723 /opt/jdk1.6.0_16/jre/lib/i386/client/libjvm.so
- b7b91000-b7fb3000 rwxp b7b91000 00:00 0
- b7fb3000-b7fba000 r-xp 00000000 fd:02 114732 /opt/jdk1.6.0_16/jre/lib/i386/jli/libjli.so
- b7fba000-b7fbc000 rwxp 00006000 fd:02 114732 /opt/jdk1.6.0_16/jre/lib/i386/jli/libjli.so
- b7fbe000-b7fbf000 rwxp b7fbe000 00:00 0
- b7fbf000-b7fc0000 r-xp b7fbf000 00:00 0
- b7fc0000-b7fcb000 r-xp 00000000 fd:02 114729 /opt/jdk1.6.0_16/jre/lib/i386/libverify.so
- b7fcb000-b7fcc000 rwxp 0000b000 fd:02 114729 /opt/jdk1.6.0_16/jre/lib/i386/libverify.so
- b7fcc000-b7fcd000 rwxp b7fcc000 00:00 0
- b7fcd000-b7fce000 r-xp b7fcd000 00:00 0 [vdso]
- bfb58000-bfb6d000 rwxp bffea000 00:00 0 [stack]
- VM Arguments:
- jvm_args: -Xmx1000m -Dhadoop.log.dir=/home/hadoop/hadoop/bin/../logs -Dhadoop.log.file=hadoop.log -D hadoop.home.dir=/home/hadoop/hadoop/bin/.. -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava .library.path=/home/hadoop/hadoop/bin/../lib/native/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy .xml
- java_command: org.apache.hadoop.fs.FsShell -put file02 input
- Launcher Type: SUN_STANDARD
- Environment Variables:
- JAVA_HOME=/opt/jdk1.6.0_16
- PATH=/opt/jdk1.6.0_16/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/oracle/product/11.2.0/ dbhome_1/bin:/home/hadoop/bin
- LD_LIBRARY_PATH=/opt/jdk1.6.0_16/jre/lib/i386/client:/opt/jdk1.6.0_16/jre/lib/i386:/opt/jdk1.6.0_16/ jre/../lib/i386
- SHELL=/bin/bash
- DISPLAY=localhost:10.0
- Signal Handlers:
- SIGSEGV: [libjvm.so+0x3fc370], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGBUS: [libjvm.so+0x3fc370], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGFPE: [libjvm.so+0x325f30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGPIPE: [libjvm.so+0x325f30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGXFSZ: [libjvm.so+0x325f30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGILL: [libjvm.so+0x325f30], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
- SIGUSR2: [libjvm.so+0x328b20], sa_mask[0]=0x00000000, sa_flags=0x10000004
- SIGHUP: [libjvm.so+0x328850], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGINT: [libjvm.so+0x328850], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGTERM: [libjvm.so+0x328850], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- SIGQUIT: [libjvm.so+0x328850], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
- --------------- S Y S T E M ---------------
- OS:CentOS release 5.3 (Final)
- uname:Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:33:56 EDT 2009 i686
- libc:glibc 2.5 NPTL 2.5
- rlimit: STACK 10240k, CORE 0k, NPROC 8191, NOFILE 1024, AS infinity
- load average:0.00 0.02 0.00
- CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 2 stepping 9, cmov, cx8, fxsr, mmx , sse, sse2
- Memory: 4k page, physical 515348k(153864k free), swap 1015800k(1015800k free)
- vm_info: Java HotSpot(TM) Client VM (14.2-b01) for linux-x86 JRE (1.6.0_16-b01), built on Jul 31 200 9 06:19:30 by "java_re" with gcc 3.2.1-7a (J2SE release)
- time: Mon Sep 28 21:49:22 2009
- elapsed time: 0 seconds
复制代码 |
|