kvm虚拟机自动重启故障，请大神帮忙分析分析

xiaoniqiu26003

今天生产环境的kvm虚拟机发生了一次自动重启，查询了好久没有发现原因，chinaunix的大神帮我分析下，跪谢！！！

环境配置如下：
宿主机：
libvirt版本
virsh # version
Compiled against library: libvirt 1.2.21
Using library: libvirt 1.2.21
Using API: QEMU 1.2.21
Running hypervisor: QEMU 2.1.2

操作系统：

root@localhost:/var/log#  lsb_release -a
No LSB modules are available.
Distributor ID:        Debian
Description:        Debian GNU/Linux 8.3 (jessie)
Release:        8.3
Codename:        jessie

虚拟机也是debian系统

下面是我截取的log日志,由于内容较长，只截取了部分log，不够我再截取，请上眼：

1.虚拟机/var/log/message日志： 可以看到在Feb 26 18:49:06  时间点系统重启 了 debian shutdown[50043]: shutting down for system reboot
Feb 25 13:13:12 debian kernel: [    4.238137] ppdev: user-space parallel port driver
Feb 26 07:35:11 debian rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2118" x-info="h tt p://w ww.rsysl og.com"] rsyslogd was HUPed
Feb 26 18:49:06 debian shutdown[50043]: shutting down for system reboot
Feb 26 18:50:16 debian kernel: imklog 5.8.11, log source = /proc/kmsg started.
Feb 26 18:50:16 debian rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2121" x-info="h tt p://ww w.rs yslog.co m"] start
Feb 26 18:50:16 debian kernel: [    0.000000] Initializing cgroup subsys cpuset
Feb 26 18:50:16 debian kernel: [    0.000000] Initializing cgroup subsys cpu
Feb 26 18:50:16 debian kernel: [    0.000000] Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.63-2
Feb 26 18:50:16 debian kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.2.0-4-amd64 root=UUID=a41b73d7-63ca-46af-ba2f-fb80b6363dbe ro quiet
Feb 26 18:50:16 debian kernel: [    0.000000] BIOS-provided physical RAM map:
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000000100000 - 00000000bffdf000 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000bffdf000 - 00000000c0000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000100000000 - 0000000440000000 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000] NX (Execute Disable) protection: active
Feb 26 18:50:16 debian kernel: [    0.000000] SMBIOS 2.8 present.
Feb 26 18:50:16 debian kernel: [    0.000000] No AGP bridge found
Feb 26 18:50:16 debian kernel: [    0.000000] last_pfn = 0x440000 max_arch_pfn = 0x400000000
Feb 26 18:50:16 debian kernel: [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Feb 26 18:50:16 debian kernel: [    0.000000] last_pfn = 0xbffdf max_arch_pfn = 0x400000000
Feb 26 18:50:16 debian kernel: [    0.000000] found SMP MP-table at [ffff8800000f0e70] f0e70
Feb 26 18:50:16 debian kernel: [    0.000000] Using GB pages for direct mapping
Feb 26 18:50:16 debian kernel: [    0.000000] init_memory_mapping: 0000000000000000-00000000bffdf000
Feb 26 18:50:16 debian kernel: [    0.000000] init_memory_mapping: 0000000100000000-0000000440000000
Feb 26 18:50:16 debian kernel: [    0.000000] RAMDISK: 36bf0000 - 375f0000
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: RSDP 00000000000f0e20 00014 (v00 BOCHS )
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: RSDT 00000000bffe170f 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: FACP 00000000bffe08b7 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: DSDT 00000000bffdfdc0 00AF7 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: FACS 00000000bffdfd80 00040
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: SSDT 00000000bffe092b 00D1C (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: APIC 00000000bffe1647 00090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] ACPI: HPET 00000000bffe16d7 00038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
Feb 26 18:50:16 debian kernel: [    0.000000] No NUMA configuration found
Feb 26 18:50:16 debian kernel: [    0.000000] Faking a node at 0000000000000000-0000000440000000
Feb 26 18:50:16 debian kernel: [    0.000000] Initmem setup node 0 0000000000000000-0000000440000000
Feb 26 18:50:16 debian kernel: [    0.000000]   NODE_DATA [000000043fffb000 - 000000043fffffff]
Feb 26 18:50:16 debian kernel: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
Feb 26 18:50:16 debian kernel: [    0.000000] kvm-clock: cpu 0, msr 0:16a9701, boot clock
Feb 26 18:50:16 debian kernel: [    0.000000] Zone PFN ranges:
Feb 26 18:50:16 debian kernel: [    0.000000]   DMA      0x00000010 -> 0x00001000
Feb 26 18:50:16 debian kernel: [    0.000000]   DMA32    0x00001000 -> 0x00100000
Feb 26 18:50:16 debian kernel: [    0.000000]   Normal   0x00100000 -> 0x00440000
Feb 26 18:50:16 debian kernel: [    0.000000] Movable zone start PFN for each node
Feb 26 18:50:16 debian kernel: [    0.000000] early_node_map[3] active PFN ranges



2.虚拟机/var/log/syslog日志：同样有 Feb 26 18:49:06 debian shutdown[50043]: shutting down for system reboot

Feb 26 07:35:11 debian rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2118" x-info="ht tp://www.rsys log.com"] rsyslogd was HUPed
Feb 26 07:35:12 debian anacron[31956]: Job `cron.daily' terminated
Feb 26 07:35:12 debian anacron[31956]: Normal exit (1 job run)
Feb 26 08:17:01 debian /USR/SBIN/CRON[33323]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 09:17:01 debian /USR/SBIN/CRON[34830]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 10:17:01 debian /USR/SBIN/CRON[36337]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 11:17:01 debian /USR/SBIN/CRON[37935]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 12:17:01 debian /USR/SBIN/CRON[39442]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 13:17:01 debian /USR/SBIN/CRON[40949]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 14:17:01 debian /USR/SBIN/CRON[42656]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 15:17:01 debian /USR/SBIN/CRON[44436]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 16:17:01 debian /USR/SBIN/CRON[45959]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 17:17:01 debian /USR/SBIN/CRON[47689]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 18:17:01 debian /USR/SBIN/CRON[49214]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 26 18:49:06 debian shutdown[50043]: shutting down for system reboot
Feb 26 18:49:06 debian init: Switching to runlevel: 6
Feb 26 18:49:08 debian minissdpd[2862]: received signal 15, good-bye
Feb 26 18:49:08 debian avahi-daemon[2509]: Got SIGTERM, quitting.
Feb 26 18:49:08 debian avahi-daemon[2509]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::5054:ff:fe9a:d587.
Feb 26 18:49:08 debian avahi-daemon[2509]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.10.53.
Feb 26 18:49:08 debian bluetoothd[2495]: Terminating
Feb 26 18:49:08 debian NetworkManager[2601]: <info> caught signal 15, shutting down normally.
Feb 26 18:49:08 debian NetworkManager[2601]: <warn> quit request received, terminating...
Feb 26 18:49:08 debian NetworkManager[2601]: <info> exiting (success)
Feb 26 18:49:08 debian avahi-daemon[2509]: avahi-daemon 0.6.31 exiting.
Feb 26 18:49:08 debian bluetoothd[2495]: Stopping SDP server
Feb 26 18:49:08 debian bluetoothd[2495]: Exit
Feb 26 18:49:08 debian acpid: exiting
Feb 26 18:49:08 debian modem-manager[2755]: <info>  Caught signal 15, shutting down...
Feb 26 18:50:16 debian kernel: imklog 5.8.11, log source = /proc/kmsg started.
Feb 26 18:50:16 debian rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2121" x-info="htt p://www.rsy slog.com"] start
Feb 26 18:50:16 debian kernel: [    0.000000] Initializing cgroup subsys cpuset
Feb 26 18:50:16 debian kernel: [    0.000000] Initializing cgroup subsys cpu
Feb 26 18:50:16 debian kernel: [    0.000000] Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.63-2
Feb 26 18:50:16 debian kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.2.0-4-amd64 root=UUID=a41b73d7-63ca-46af-ba2f-fb80b6363dbe ro quiet
Feb 26 18:50:16 debian kernel: [    0.000000] BIOS-provided physical RAM map:
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000000100000 - 00000000bffdf000 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000bffdf000 - 00000000c0000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000]  BIOS-e820: 0000000100000000 - 0000000440000000 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000] NX (Execute Disable) protection: active
Feb 26 18:50:16 debian kernel: [    0.000000] SMBIOS 2.8 present.
Feb 26 18:50:16 debian kernel: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Feb 26 18:50:16 debian kernel: [    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
Feb 26 18:50:16 debian kernel: [    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
Feb 26 18:50:16 debian kernel: [    0.000000] No AGP bridge found
Feb 26 18:50:16 debian kernel: [    0.000000] last_pfn = 0x440000 max_arch_pfn = 0x400000000
Feb 26 18:50:16 debian kernel: [    0.000000] MTRR default type: write-back
Feb 26 18:50:16 debian kernel: [    0.000000] MTRR fixed ranges enabled:
Feb 26 18:50:16 debian kernel: [    0.000000]   00000-9FFFF write-back
Feb 26 18:50:16 debian kernel: [    0.000000]   A0000-BFFFF uncachable
Feb 26 18:50:16 debian kernel: [    0.000000]   C0000-FFFFF write-protect
Feb 26 18:50:16 debian kernel: [    0.000000] MTRR variable ranges enabled

3.物理机上的/var/log/libvirt/qemu里没有任何日志打印  奇怪！！！

4.物理机上的dmesg log 不知道有没有用 我也一并贴出来吧：

[99937.365372] device vmnet57 entered promiscuous mode
[99937.365515] br2: port 5(vmnet57) entered forwarding state
[99937.365521] br2: port 5(vmnet57) entered forwarding state
[101606.987514] device eth3 left promiscuous mode
[101606.987662] br3: port 1(eth3) entered disabled state
[101606.988446] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[101643.475357] device eth3 entered promiscuous mode
[101643.476880] tg3 0000:02:00.1: irq 143 for MSI/MSI-X
[101643.476907] tg3 0000:02:00.1: irq 144 for MSI/MSI-X
[101643.476926] tg3 0000:02:00.1: irq 145 for MSI/MSI-X
[101643.476943] tg3 0000:02:00.1: irq 146 for MSI/MSI-X
[101643.476963] tg3 0000:02:00.1: irq 147 for MSI/MSI-X
[101643.594365] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[101643.597221] IPv6: ADDRCONF(NETDEV_UP): br3: link is not ready
[125637.765295] perf interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[182871.580553] tg3 0000:02:00.1 eth3: Link is up at 1000 Mbps, full duplex
[182871.580566] tg3 0000:02:00.1 eth3: Flow control is off for TX and off for RX
[182871.580570] tg3 0000:02:00.1 eth3: EEE is disabled
[182871.580592] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
[182871.580757] br3: port 1(eth3) entered forwarding state
[182871.580763] br3: port 1(eth3) entered forwarding state
[182871.580928] IPv6: ADDRCONF(NETDEV_CHANGE): br3: link becomes ready
[186558.147190] br2: port 5(vmnet57) entered disabled state
[186558.148172] device vmnet57 left promiscuous mode
[186558.148186] br2: port 5(vmnet57) entered disabled state
[186561.165530] br2: port 4(vmnet5 entered disabled state
[186561.166552] device vmnet58 left promiscuous mode
[186561.166561] br2: port 4(vmnet5 entered disabled state
[186598.858855] br0: port 3(vmnet62) entered disabled state
[186598.859624] device vmnet62 left promiscuous mode
[186598.859630] br0: port 3(vmnet62) entered disabled state
[186633.621894] device vmnet62 entered promiscuous mode
[186633.638031] br0: port 3(vmnet62) entered forwarding state
[186633.638039] br0: port 3(vmnet62) entered forwarding state
[186637.479182] kvm: zapping shadow pages for mmio generation wraparound
[186638.801949] device vmnet57 entered promiscuous mode
[186638.818097] br3: port 2(vmnet57) entered forwarding state
[186638.818104] br3: port 2(vmnet57) entered forwarding state
[186642.412776] device vmnet58 entered promiscuous mode
[186642.444882] br3: port 3(vmnet5 entered forwarding state
[186642.444889] br3: port 3(vmnet5 entered forwarding state
[186642.805135] kvm: zapping shadow pages for mmio generation wraparound
[186643.532540] kvm [18557]: vcpu0 unhandled rdmsr: 0x345
[186646.392155] kvm: zapping shadow pages for mmio generation wraparound
[186648.889221] kvm [18575]: vcpu0 unhandled rdmsr: 0x345
[186652.538320] kvm [18593]: vcpu0 unhandled rdmsr: 0x345
[239353.815905] UDP: short packet: From 95.110.185.31:0 17959/1480 to 101.251.195.119:0
[244492.544933] kvm [8093]: vcpu0 unhandled rdmsr: 0x345
[244500.218354] kvm [8111]: vcpu0 unhandled rdmsr: 0x345
[321898.879301] kvm [18557]: vcpu0 unhandled rdmsr: 0x345
[351167.793461] kvm [8111]: vcpu0 unhandled rdmsr: 0x345
[351207.339178] kvm [8111]: vcpu0 unhandled rdmsr: 0x345

5、我的libvirt.xml配置如下：

<domain type='kvm' id='14'>
  <name>customer_10.0.10.53</name>
  <uuid>acca18bc-6a74-985a-ee59-771a88ef971d</uuid>
  <description>None</description>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/vmstorage/images/customer2.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:9a:d5:87'/>
      <source bridge='br0'/>
      <target dev='vmnet53'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>
Edit

6.宿主机网卡绑定信息：
root@localhost:/var/log# brctl show
bridge name        bridge id                STP enabled        interfaces
br0                8000.44a8424ad18a        no                eth0
                                                        vmnet0
                                                        vmnet62
br2                8000.44a8424ad18c        no                eth2
                                                        vmnet52
                                                        vmnet53
br3                8000.44a8424ad18d        no                eth3
                                                        vmnet57
                                                        vmnet58
virbr0                8000.5254009d8477        yes                virbr0-nic



以上，请各位大神救救我！！！！