RHCS双机，node2重启导致双机切换，日志有点看不明白

hankewhl

Jan 11 10:44:58 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:44:58 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:44:58 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 270 seconds.
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 274 seconds.
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 291 seconds.
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 263 seconds.
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 242 seconds.
Jan 11 11:07:17 AMC-S-X3650-2 openais[4179]: [TOTEM] The token was lost in the OPERATIONAL stat
e.
Jan 11 11:07:17 AMC-S-X3650-2 openais[4179]: [TOTEM] Receive multicast socket recv buffer size
(288000 bytes).
Jan 11 11:07:17 AMC-S-X3650-2 openais[4179]: [TOTEM] Transmit multicast socket send buffer size
(262142 bytes).
Jan 11 11:07:17 AMC-S-X3650-2 openais[4179]: [TOTEM] entering GATHER state from 2.
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 280 seconds.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] entering GATHER state from 0.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] Creating commit token because I am the rep
.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] Saving state aru 57 high seq received 57
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] Storing new sequence id for ring e834
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] entering COMMIT state.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] entering RECOVERY state.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] position [0] member 192.168.70.2:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] previous ring seq 59440 rep 192.168.70.1
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] aru 57 high delivered 57 received flag 1
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] Did not need to originate any messages in
recovery.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] Sending initial ORF token
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] New Configuration:
Jan 11 11:07:22 AMC-S-X3650-2 kernel: dlm: closing connection to node 1
Jan 11 11:07:22 AMC-S-X3650-2 fenced[4198]: node1 not a cluster member after 0 sec post_fail_de
lay
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ]    r(0) ip(192.168.70.2)
Jan 11 11:07:22 AMC-S-X3650-2 fenced[4198]: fencing node "node1"
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] Members Left:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] Members Joined:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] New Configuration:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ]    r(0) ip(192.168.70.2)
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] Members Left:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] Members Joined:
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [SYNC ] This node is within the primary component
and will provide service.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [TOTEM] entering OPERATIONAL state.
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CLM  ] got nodejoin message 192.168.70.2
Jan 11 11:07:22 AMC-S-X3650-2 openais[4179]: [CPG  ] got joinlist message from node 2
Jan 11 11:07:32 AMC-S-X3650-2 fenced[4198]: agent "fence_ipmilan" reports: Rebooting machine @
IPMI:192.168.70.126...ipmilan: Failed to connect after 10 seconds Failed
Jan 11 11:07:32 AMC-S-X3650-2 fenced[4198]: fence "node1" failed
Jan 11 11:07:33 AMC-S-X3650-2 shutdown[32619]: shutting down for system halt
Jan 11 11:07:33 AMC-S-X3650-2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Jan 11 11:07:33 AMC-S-X3650-2 kernel: gdm-rh-security[5016]: segfault at 0000000000000000 rip 0
0002b95b9a8c19c rsp 0000000041bbfeb0 error 4
Jan 11 11:07:34 AMC-S-X3650-2 rgmanager: [32688]: <notice> Shutting down Cluster Service Manage
r...
Jan 11 11:07:34 AMC-S-X3650-2 clurgmgrd[4712]: <notice> Shutting down
Jan 11 11:07:37 AMC-S-X3650-2 fenced[4198]: fencing node "node1"
Jan 11 11:12:58 AMC-S-X3650-2 syslogd 1.4.1: restart.
Jan 11 11:12:58 AMC-S-X3650-2 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 11 11:12:58 AMC-S-X3650-2 kernel: Linux version 2.6.18-164.el5 (mockbuild@x86-003.build.bos
.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:51:48 EDT 200
9
Jan 11 11:12:58 AMC-S-X3650-2 kernel: Command line: ro root=/dev/VolGroup01/LogVol00 rhgb quiet
Jan 11 11:12:58 AMC-S-X3650-2 kernel: BIOS-provided physical RAM map:
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000000010000 - 000000000009c000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000000009c000 - 00000000000a0000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000000100000 - 000000007d052000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d052000 - 000000007d116000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d116000 - 000000007d6dc000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d6dc000 - 000000007d78c000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d78c000 - 000000007f68f000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f68f000 - 000000007f6df000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f6df000 - 000000007f7df000 (ACPI NVS
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f7df000 - 000000007f7ff000 (ACPI dat
a)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f7ff000 - 000000007f800000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f800000 - 0000000090000000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000fc000000 - 00000000fd000000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved
)
Jan 11 11:12:59 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
Jan 11 11:12:59 AMC-S-X3650-2 kernel: DMI 2.5 present.
Jan 11 11:12:59 AMC-S-X3650-2 rpc.statd[4029]: Version 1.0.9 Starting
Jan 11 11:12:59 AMC-S-X3650-2 rpc.statd[4029]: statd running as root. chown /var/lib/nfs/statd/
sm to choose different user
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 16 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 18 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 20 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 22 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 7 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 17 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 19 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 21 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 23 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: Node 0 PXM 0 0-80000000
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: Node 1 PXM 1 100000000-180000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd[4192]: Starting ccsd 2.0.115:
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Bootmem setup node 0 0000000000000000-0000000080000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd[4192]:  Built: Aug  5 2009 08:24:53
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Bootmem setup node 1 0000000100000000-0000000180000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd[4192]:  Copyright (C) Red Hat, Inc.  2004  All rights reserv
ed.
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissibl
e range
Jan 11 11:12:59 AMC-S-X3650-2 ccsd[4192]: cluster.conf (cluster name = amc_cluster, version = 3
0) found.
Jan 11 11:12:59 AMC-S-X3650-2 kernel: disabling kdump
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: PM-Timer IO Port: 0x588
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #0 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #2 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #4 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #6 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #16 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #18 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #20 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #22 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x01] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #1 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x03] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #3 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x05] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #5 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x07] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #7 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x11] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #17 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x13] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #19 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x15] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #21 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #23 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Jan 11 11:13:00 AMC-S-X3650-2 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI
0-23
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
Jan 11 11:13:00 AMC-S-X3650-2 kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI
24-47
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high leve
l)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Setting APIC routing to clustered
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Using ACPI (MADT) for SMP configuration information
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000000009c000 - 00000000000a0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000000a0000 - 00000000000e0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000000e0000 - 0000000000100000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007d052000 - 000000007d116000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007d6dc000 - 000000007d78c000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f68f000 - 000000007f6df000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f6df000 - 000000007f7df000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f7df000 - 000000007f7ff000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f800000 - 0000000090000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 0000000090000000 - 00000000fc000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fc000000 - 00000000fd000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fd000000 - 00000000fed1c000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fed1c000 - 00000000fed20000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fed20000 - 00000000ff800000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000ff800000 - 0000000100000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Allocating PCI resources starting at 98000000 (gap: 90000
000:6c000000)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: SMP: Allowing 16 CPUs, 0 hotplug CPUs
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Built 2 zonelists.  Total pages: 1030045
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Kernel command line: ro root=/dev/VolGroup01/LogVol00 rhg
b quiet
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Initializing CPU#0
Jan 11 11:13:01 AMC-S-X3650-2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Console: colour VGA+ 80x25
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Dentry cache hash table entries: 524288 (order: 10, 41943
04 bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Inode-cache hash table entries: 262144 (order: 9, 2097152
bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Checking aperture...
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: DMAR not present
Jan 11 11:13:01 AMC-S-X3650-2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Placing software IO TLB between 0x29d0000 - 0x69d0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Memory: 4043748k/6291456k available (2547k kernel code, 1
38940k reserved, 1289k data, 208k init)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Calibrating delay loop (skipped), value calculated using
timer frequency.. 4533.62 BogoMIPS (lpj=2266812)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Security Framework v1.0.0 initialized
Jan 11 11:13:01 AMC-S-X3650-2 kernel: SELinux:  Initializing.
Jan 11 11:13:01 AMC-S-X3650-2 kernel: selinux_register_security:  Registering secondary module
capability
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Capability LSM initialized as secondary
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Mount-cache hash table entries: 256
Jan 11 11:13:01 AMC-S-X3650-2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 11 11:13:01 AMC-S-X3650-2 kernel: CPU: L2 cache: 256K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L3 cache: 8192K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU 0/0 -> Node 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: using mwait in idle threads.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Physical Processor ID: 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Processor Core ID: 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 11 11:13:02 AMC-S-X3650-2 kernel: SMP alternatives: switching to UP code
Jan 11 11:13:02 AMC-S-X3650-2 kernel: ACPI: Core revision 20060707
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Using local APIC timer interrupts.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: result 8333842
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Detected 8.333 MHz APIC timer.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: SMP alternatives: switching to SMP code
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Booting processor 1/16 APIC 0x2
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Initializing CPU#1
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Calibrating delay using timer specific routine.. 4533.40
BogoMIPS (lpj=2266701)
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L2 cache: 256K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L3 cache: 8192K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU 1/2 -> Node 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Physical Processor ID: 0
--More--(84%)

hankewhl

什么原因导致node2重启了呢

yjs_sh

可能node1和node2之间的心跳通讯出现问题，因此RHCS会触发fence。由于心跳相互都不通，因此双方都会fence对方，但是从日志里看，node2在fence node1的时候失败，而node2的重启应该是由于node1 fence node2造成的。
最终的原因可能是你的node2的网络出问题造成的，你可以检查一下node1的日志看看。这个结果应该是rhcs的正常反应。

hankewhl

Jan 11 00:50:58 AMC-S-X3650-1 last message repeated 8 times
Jan 11 01:15:56 AMC-S-X3650-1 last message repeated 8 times
Jan 11 07:12:18 AMC-S-X3650-1 last message repeated 8 times
Jan 11 10:01:18 AMC-S-X3650-1 last message repeated 9 times
Jan 11 10:01:58 AMC-S-X3650-1 last message repeated 7 times
Jan 11 11:10:05 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 12.
Jan 11 11:10:05 AMC-S-X3650-1 openais[4074]: [TOTEM] Creating commit token because I am the rep.
Jan 11 11:10:05 AMC-S-X3650-1 openais[4074]: [TOTEM] Saving state aru 57 high seq received 57
Jan 11 11:10:05 AMC-S-X3650-1 openais[4074]: [TOTEM] Storing new sequence id for ring e834
Jan 11 11:10:05 AMC-S-X3650-1 openais[4074]: [TOTEM] entering COMMIT state.
Jan 11 11:10:15 AMC-S-X3650-1 openais[4074]: [TOTEM] The token was lost in the COMMIT state.
Jan 11 11:10:15 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 4.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 0.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] Creating commit token because I am the rep.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] Storing new sequence id for ring e838
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] entering COMMIT state.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] entering RECOVERY state.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] position [0] member 192.168.70.1:
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] previous ring seq 59440 rep 192.168.70.1
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] aru 57 high delivered 57 received flag 1
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] Did not need to originate any messages in recovery.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] Sending initial ORF token
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:20 AMC-S-X3650-1 kernel: dlm: closing connection to node 2
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 fenced[4093]: node2 not a cluster member after 0 sec post_fail_delay
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 fenced[4093]: fencing node "node2"
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.2)
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [SYNC ] This node is within the primary component and will provide service.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] entering OPERATIONAL state.
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] got nodejoin message 192.168.70.1
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CPG  ] got joinlist message from node 1
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 9.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 0.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] Creating commit token because I am the rep.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] Saving state aru e high seq received e
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] Storing new sequence id for ring e83c
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] entering COMMIT state.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] entering RECOVERY state.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] position [0] member 192.168.70.1:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] previous ring seq 59448 rep 192.168.70.1
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] aru e high delivered e received flag 1
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] Did not need to originate any messages in recovery.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] Sending initial ORF token
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [SYNC ] This node is within the primary component and will provide service.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] entering OPERATIONAL state.
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CLM  ] got nodejoin message 192.168.70.1
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [CPG  ] got joinlist message from node 1
Jan 11 11:10:25 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 9.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 0.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] Creating commit token because I am the rep.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] Saving state aru c high seq received c
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] Storing new sequence id for ring e840
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] entering COMMIT state.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] entering RECOVERY state.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] position [0] member 192.168.70.1:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] previous ring seq 59452 rep 192.168.70.1
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] aru c high delivered c received flag 1
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] Did not need to originate any messages in recovery.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] Sending initial ORF token
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [SYNC ] This node is within the primary component and will provide service.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [TOTEM] entering OPERATIONAL state.
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CLM  ] got nodejoin message 192.168.70.1
Jan 11 11:10:30 AMC-S-X3650-1 openais[4074]: [CPG  ] got joinlist message from node 1
Jan 11 11:10:30 AMC-S-X3650-1 fenced[4093]: fence "node2" success
Jan 11 11:10:31 AMC-S-X3650-1 clurgmgrd[4627]: <notice> Taking over service service:amcapache from down member node2
Jan 11 11:10:31 AMC-S-X3650-1 avahi-daemon[4537]: Registering new address record for 172.20.16.2 on eth0.
Jan 11 11:10:32 AMC-S-X3650-1 clurgmgrd[4627]: <notice> Service service:amcapache started
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] entering GATHER state from 9.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] Creating commit token because I am the rep.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] Saving state aru 16 high seq received 16
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] Storing new sequence id for ring e844
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] entering COMMIT state.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] entering RECOVERY state.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] position [0] member 192.168.70.1:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] previous ring seq 59456 rep 192.168.70.1
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] aru 16 high delivered 16 received flag 1
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] position [1] member 192.168.70.2:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] previous ring seq 59448 rep 192.168.70.2
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] aru d high delivered d received flag 1
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] Did not need to originate any messages in recovery.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] Sending initial ORF token
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] CLM CONFIGURATION CHANGE
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] New Configuration:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.2)
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Joined:
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.2)
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [SYNC ] This node is within the primary component and will provide service.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [TOTEM] entering OPERATIONAL state.
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] got nodejoin message 192.168.70.1
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CLM  ] got nodejoin message 192.168.70.2
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CPG  ] got joinlist message from node 1
Jan 11 11:14:24 AMC-S-X3650-1 openais[4074]: [CPG  ] got joinlist message from node 2
Jan 11 11:14:35 AMC-S-X3650-1 kernel: dlm: connecting to 2
Jan 11 11:14:35 AMC-S-X3650-1 kernel: dlm: got connection from 2

你好，这是node1的日志信息

yjs_sh

Jan 11 11:10:20 AMC-S-X3650-1 kernel: dlm: closing connection to node 2
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ]    r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 fenced[4093]: node2 not a cluster member after 0 sec post_fail_delay
Jan 11 11:10:20 AMC-S-X3650-1 openais[4074]: [CLM  ] Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 fenced[4093]: fencing node "node2"
以上表明node1无法检测到node2的心跳后，认为node2已经出现问题，开始fence node2
Jan 11 11:10:30 AMC-S-X3650-1 fenced[4093]: fence "node2" success
Jan 11 11:10:31 AMC-S-X3650-1 clurgmgrd[4627]: <notice> Taking over service service:amcapache from down member node2
以上表明node1已经成功fence node2，接下来开始接管HA所管理的服务。

RHCS就是这样的机制，如果一个节点出现问题，那么其他节点将会fence这个节点。基本上fence就是将其重新启动，当然还有部分fence是将存储断开。

hankewhl

那node2是为什么重启的呢？

yjs_sh

从日志上看应该可能是node2的网络出现了问题，然后被node1 fence掉了。其实在node2的日志中也出现了fence node1的记录，但是没有成功，那基本就证明了是node2的网络问题。一般fence失败基本上都是网络问题，除非rhce的fence配置有问题。不过你还可以看看是不是当时服务出了问题，一般rhce会在本地重新启动服务，3次失败后会进行切换，如果切换的过程中比如：卸载文件系统失败，也会自动重启，不过应该日志中不会是被fence掉的日志。

zhaopingzi

RHCS垃圾中的战斗机