CE6810下任意两台服务器httping和ping抖动

ruochen

现象描述:
同一台CE6810交换下任意两台服务器之间httping（170bytes）和ping（64bytes）的正常响应时间都是在0.0x~0.x毫秒级别
但是频繁出现超过20ms甚至更长的的建连时间
目标：tcp建连控制在5ms或者10ms以内


一）ping情况：
1）ping命令
      1.1）正常命令（过滤大于0ms的）
             ping -i 0.01 目的IP | awk -F "=" '{now=strftime("%m%d-%H:%M:%S"); if($4 > 0) print now,$0}'
            
0222-13:33:34 64 bytes from 目的IP: icmp_seq=26 ttl=64 time=0.070 ms
0222-13:33:34 64 bytes from 目的IP: icmp_seq=27 ttl=64 time=0.039 ms
0222-13:33:34 64 bytes from 目的IP: icmp_seq=28 ttl=64 time=0.038 ms
0222-13:33:34 64 bytes from 目的IP: icmp_seq=29 ttl=64 time=0.038 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=30 ttl=64 time=0.037 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=31 ttl=64 time=0.054 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=32 ttl=64 time=0.039 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=33 ttl=64 time=0.059 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=34 ttl=64 time=0.058 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=35 ttl=64 time=0.034 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=36 ttl=64 time=0.064 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=37 ttl=64 time=0.036 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=38 ttl=64 time=0.049 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=39 ttl=64 time=0.041 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=40 ttl=64 time=0.039 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=41 ttl=64 time=0.040 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=42 ttl=64 time=0.034 ms
0222-13:33:35 64 bytes from 目的IP: icmp_seq=43 ttl=64 time=0.039 ms

统计信息：
1207 packets transmitted, 1207 received, 0% packet loss, time 14710ms
rtt min/avg/max/mdev = 0.031/0.091/20.445/0.780 ms, pipe 2

                 
        
      1.2）异常命令（过滤大于1ms）
              ping -i 0.01 目的IP | awk -F "=" '{now=strftime("%m%d-%H:%M:%S"); if($4 > 1) print now,$0}'


0222-13:34:07 64 bytes from 目的IP: icmp_seq=773 ttl=64 time=11.6 ms
0222-13:34:07 64 bytes from 目的IP: icmp_seq=774 ttl=64 time=20.8 ms
0222-13:34:07 64 bytes from 目的IP: icmp_seq=775 ttl=64 time=2.54 ms
0222-13:34:07 64 bytes from 目的IP: icmp_seq=776 ttl=64 time=1.61 ms
0222-13:35:34 64 bytes from 目的IP: icmp_seq=7903 ttl=64 time=19.0 ms
0222-13:35:34 64 bytes from 目的IP: icmp_seq=7904 ttl=64 time=7.02 ms
0222-13:35:34 64 bytes from 目的IP: icmp_seq=7905 ttl=64 time=11.5 ms
0222-13:35:56 64 bytes from 目的IP: icmp_seq=9725 ttl=64 time=20.4 ms
0222-13:35:56 64 bytes from 目的IP: icmp_seq=9726 ttl=64 time=8.50 ms
0222-13:35:56 64 bytes from 目的IP: icmp_seq=9727 ttl=64 time=12.7 ms
0222-13:35:56 64 bytes from 目的IP: icmp_seq=9728 ttl=64 time=2.72 ms
0222-13:35:56 64 bytes from 目的IP: icmp_seq=9729 ttl=64 time=2.64 ms
0222-13:36:15 64 bytes from 目的IP: icmp_seq=11329 ttl=64 time=1.44 ms
0222-13:36:15 64 bytes from 目的IP: icmp_seq=11330 ttl=64 time=5.27 ms
0222-13:36:15 64 bytes from 目的IP: icmp_seq=11331 ttl=64 time=16.1 ms
0222-13:36:15 64 bytes from 目的IP: icmp_seq=11332 ttl=64 time=4.15 ms
0222-13:36:15 64 bytes from 目的IP: icmp_seq=11333 ttl=64 time=3.29 ms
0222-13:36:18 64 bytes from 目的IP: icmp_seq=11509 ttl=64 time=4.69 ms
0222-13:36:18 64 bytes from 目的IP: icmp_seq=11510 ttl=64 time=15.8 ms
0222-13:36:18 64 bytes from 目的IP: icmp_seq=11511 ttl=64 time=5.83 ms
0222-13:36:18 64 bytes from 目的IP: icmp_seq=11512 ttl=64 time=5.18 ms
0222-13:36:29 64 bytes from 目的IP: icmp_seq=12417 ttl=64 time=11.7 ms
0222-13:36:29 64 bytes from 目的IP: icmp_seq=12418 ttl=64 time=20.9 ms
0222-13:36:29 64 bytes from 目的IP: icmp_seq=12419 ttl=64 time=2.60 ms
0222-13:36:29 64 bytes from 目的IP: icmp_seq=12420 ttl=64 time=1.81 ms



二）httping情况：
2）httping命令
      2.1）正常命令（过滤大于0ms的）

             httping -i 0.01 -S  "http://目的IP：PORT/URL" |awk -F'[=+]'  
             '{now=strftime("%m%d-%H:%M:%S"); if($3 > 0) print now,$0}'
   
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=76 time=0.09+0.17=0.26 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=77 time=0.09+0.12=0.21 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=78 time=0.08+0.13=0.22 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=79 time=0.09+0.17=0.26 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=80 time=0.08+0.19=0.27 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=81 time=0.06+0.13=0.19 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=82 time=0.09+0.19=0.27 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=83 time=0.09+0.13=0.22 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=84 time=0.08+0.14=0.22 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=85 time=0.07+0.20=0.27 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=86 time=0.10+0.16=0.26 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=87 time=0.06+0.14=0.20 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=88 time=0.08+3.46=3.54 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=89 time=0.08+0.17=0.25 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=90 time=0.08+0.14=0.22 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=91 time=0.07+0.18=0.25 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=92 time=0.09+0.12=0.21 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=93 time=0.09+0.18=0.26 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=94 time=0.09+0.13=0.22 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=95 time=0.06+0.14=0.20 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=96 time=0.11+0.15=0.26 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=97 time=0.08+0.13=0.21 ms
0222-13:41:10 connected to 目的IP：PORT (170 bytes), seq=98 time=0.12+0.15=0.27 ms


      2.2）异常命令（过滤大于5ms的）
             httping -i 0.01 -S  "http://目的IP：PORT/URL" |awk -F'[=+]'  
             '{now=strftime("%m%d-%H:%M:%S"); if($3 > 5) print now,$0}'

   
0222-13:41:38 connected to 目的IP：PORT (170 bytes), seq=2150 time=17.87+0.13=18.00 ms
0222-13:42:02 connected to 目的IP：PORT (170 bytes), seq=4500 time=17.87+0.20=18.07 ms
0222-13:42:07 connected to 目的IP：PORT (170 bytes), seq=4915 time=11.73+0.21=11.93 ms
0222-13:42:15 connected to 目的IP：PORT (170 bytes), seq=5698 time=6.68+0.17=6.86 ms
0222-13:42:19 connected to 目的IP：PORT (170 bytes), seq=6098 time=12.74+0.18=12.92 ms
0222-13:43:02 connected to 目的IP：PORT (170 bytes), seq=10227 time=11.08+0.19=11.26 ms
0222-13:43:30 connected to 目的IP：PORT (170 bytes), seq=12963 time=9.97+0.18=10.14 ms
0222-13:43:35 connected to 目的IP：PORT (170 bytes), seq=13500 time=19.50+0.17=19.67 ms
0222-13:44:22 connected to 目的IP：PORT (170 bytes), seq=18012 time=5.12+0.37=5.49 ms





网络结构：
服务器1/2/3/4------CE6810


服务器配置：
OS：debian 8 kernel 3.16.7
OS压力：E5-2658*2 load 10左右
网卡：82599ES 10G 戴尔原装光模块
网卡驱动：3.16.7内核自带3.19.1-k（升级到4.6.4没有改善）
网卡模式：bond mode 0（变更到mode 4没有改善）
网卡压力：单网卡带宽最高300M，单网卡pps 最高10万
CE6810：华为原装光模块，cpu 使用率在20%左右                                            每个口上错误包都是0，光衰在标称最大、最小范围内