squid负载过高,必然DOWN机,垦请高手来分析

fugangyun

操作系统:solaris 10
服务器:SUN E2900   16G内存    4CPU   2个10000转SCSI硬盘
df -h

/dev/dsk/c1t0d0s3       20G   4.4G    15G    23%    /var
swap                    27G     0K    27G     0%    /tmp
swap                    27G    16K    27G     1%    /var/run
/dev/dsk/c1t1d0s1       16G    11G   4.3G    73%    /proxy
/dev/dsk/c1t1d0s3       20G    14G   5.6G    72%    /cache1

squid版本是2.6,安装在/proxy下
squid.conf配置如下:

http_port 10.1.1.1:8081
hierarchy_stoplist cgi-bin ?
hierarchy_stoplist -i ^https:\\ ?
acl QUERY urlpath_regex -i cgi-bin \? \.asp \.php \.jsp \.cgi
acl denyssl urlpath_regex -i ^https:\\
cache deny QUERY
cache deny denyssl
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

cache_mem 1024 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 8192 KB
maximum_object_size_in_memory 32 KB

cache_dir ufs /proxy/var/cache 12000 48 256
cache_dir ufs /cache1 15000 48 256
access_log none
cache_log /cmproxy/var/logs/cache.log
cache_store_log none
mime_table /proxy/etc/mime.conf
pid_filename /proxy/var/logs/squid.pid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320

acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80                # http
acl Safe_ports port 21                # ftp
acl Safe_ports port 443 563           # https, snews
acl Safe_ports port 70                # gopher
acl Safe_ports port 210               # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488                # gss-http
acl Safe_ports port 591                # filemaker
acl Safe_ports port 777                # multiling http
acl CONNECT method CONNECT
acl SSL_gmcc_port port 81 443 563 7001 8080 8888 9087 9088 9315
acl our_networks src 10.0.0.0/8 192.168.0.0/16
http_access allow our_networks
http_access allow manager localhost
http_access deny manager
http_access allow SSL_gmcc_port
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all
http_reply_access allow all

logfile_rotate 7
visible_hostname none
forwarded_for off
store_objects_per_bucket 50
icon_directory /proxy/share/icons
coredump_dir /cache1

fugangyun

同时在线连接数超过3K后,终端打开网页极慢,且squid进程僵死,无法通过squid -k shutdown停止,即使用squid -k kill 或kill -9 进程id也无效,只有重启机器

连接数

netstat -an|grep 8081|grep EST|wc -l
3096

等待TIME_WAIT数

netstat -an|grep TIME_WAIT|wc -l
5300

CPU占用率并不高

#sar -u 3 5

SunOS 5.10 Generic_118833-02 sun4u    03/21/2007

08:16:10    %usr    %sys    %wio   %idle
08:16:13       3       5       0      92
08:16:16       3       5       0      92
08:16:19       2       6       0      92
08:16:22       2       4       0      94
08:16:25       2       4       0      93

Average        2       5       0      93

磁盘IO使用率较高

iostat -cxn 5
                 extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d20
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d21
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 d22
    0.0    0.6    0.0    0.4  0.0  0.0    0.0    9.1   0   0 c1t0d0
   18.6  128.2   70.8 1440.9  0.0  3.8    0.0   25.9   0  81 c1t1d0

内存使用及机器负荷

prstat
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP      
   279 root     1577M 1573M cpu1     0    0   3:45:19 3.7% squid/1
   125 named      10M 7752K sleep   59    0   0:04:56 0.1% named/11
   280 nobody   1168K  840K sleep   60    0   0:04:15 0.1% unlinkd/1

Total: 26 processes, 134 lwps, load averages: 0.50, 0.55, 0.54

[ 本帖最后由 fugangyun 于 2007-3-21 09:49 编辑 ]

fugangyun

squid的信息

squidclient -p 8081 mgr:info

HTTP/1.0 200 OK
Server: squid/2.6.STABLE10
Date: Wed, 21 Mar 2007 00:23:01 GMT
Content-Type: text/plain
Expires: Wed, 21 Mar 2007 00:23:01 GMT
Last-Modified: Wed, 21 Mar 2007 00:23:01 GMT
X-Cache: MISS from none
Via: 1.0 none:8081 (squid/2.6.STABLE10)
Proxy-Connection: close

Squid Object Cache: Version 2.6.STABLE10
Start Time:     Tue, 20 Mar 2007 04:32:03 GMT
Current Time:   Wed, 21 Mar 2007 00:23:01 GMT
Connection information for squid:
        Number of clients accessing cache:      4514
        Number of HTTP requests received:       7291534
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   6122.4
        Average ICP messages per minute since start:    0.0
        Select loop called: 39501660 times, 1.809 ms avg
Cache information for squid:
        Request Hit Ratios:     5min: 46.8%, 60min: 47.8%
        Byte Hit Ratios:        5min: 14.7%, 60min: 14.5%
        Request Memory Hit Ratios:      5min: 17.4%, 60min: 21.3%
        Request Disk Hit Ratios:        5min: 13.4%, 60min: 11.9%
        Storage Swap size:      24968613 KB
        Storage Mem size:       1048296 KB
        Mean Object Size:       22.17 KB
        Requests given to unlinkd:      871980
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.19742  0.12106
        Cache Misses:          0.44492  0.30459
        Cache Hits:            0.07014  0.04277
        Near Hits:             0.30459  0.22004
        Not-Modified Replies:  0.05046  0.03427
        DNS Lookups:           0.04854  0.02809
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        71457.499 seconds
        CPU Time:       13602.398 seconds
        CPU Usage:      19.04%
        CPU Usage, 5 minute avg:        30.82%
        CPU Usage, 60 minute avg:       30.37%
        Process Data Segment Size via sbrk(): 1596379 KB
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 10096743
Memory usage for squid via mallinfo():
        Total space in arena:  1596379 KB
        Ordinary blocks:       1471660 KB 258638 blks
        Small blocks:               0 KB      0 blks
        Holding blocks:         10928 KB     10 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:  124718 KB
        Total in use:          1482588 KB 92%
        Total free:            124718 KB 8%
        Total size:            1607307 KB
Memory accounted for:
        Total accounted:       1287875 KB
        memPoolAlloc calls: 753300395
        memPoolFree calls: 746272417
File descriptor usage for squid:
        Maximum number of file descriptors:   32768
        Largest file desc currently in use:   5321
        Number of file desc currently in use: 4396
        Files queued for open:                   0
        Available number of file descriptors: 28372
        Reserved number of file descriptors:   100
        Store Disk files open:                  26
        IO loop method:                     poll
Internal Data Structures:
        1133385 StoreEntries
        143938 StoreEntries with MemObjects
        143638 Hot Object Cache Items
        1126459 on-disk objects

fugangyun

每次DOWN机都在messages里出现,

[ID 702911 user.alert] xmalloc: Unable to allocate 49152 bytes!
WARNING: High TCP connect timeout rate! System (port 8081) may be under a SYN flood attack!

chinaglwo

把系统的SYN flood 打开呢

linux可以的，solaris 不太清楚

xdb1010

可能是有人用CC攻击你的站点,耗死你服务器的资源.
提示SYN flood attack的话可能就是有SYN 洪水攻击.
设置操作系统本身效果不会很大的,如果有硬件防火墙效果就能好些.

fugangyun

咨询过SUN公司，出现

WARNING: High TCP connect timeout rate! System (port 8081) may be under a SYN flood attack!

是一种常规告警，并非真正的受到攻击

且这台服务器是在DMZ区，有CISCO PIX 535硬件防火墙

fugangyun

在squid 的FAQ找到如下信息:

xmalloc: Unable to allocate 4096 bytes!


Messages like "FATAL: xcalloc: Unable to allocate 4096 blocks of 1 bytes!" appear when Squid can't allocate more memory, and on most operating systems (inclusive BSD) there are only two possible reasons:

    * The machine is out of swap
    * The process' maximum data segment size has been reached

The first case is detected using the normal swap monitoring tools available on the platform (pstat on SunOS, perhaps pstat is used on BSD as well).

To tell if it is the second case, first rule out the first case and then monitor the size of the Squid process. If it dies at a certain size with plenty of swap left then the max data segment size is reached without no doubts.

The data segment size can be limited by two factors:

    * Kernel imposed maximum, which no user can go above
    * The size set with ulimit, which the user can control.

When squid starts it sets data and file ulimit's to the hard level. If you manually tune ulimit before starting Squid make sure that you set the hard limit and not only the soft limit (the default operation of ulimit is to only change the soft limit). root is allowed to raise the soft limit above the hard limit.

fugangyun

问题另类解决了!

SUN公司给了一个编译好的二进制squid2.5 for solaris 10的包给我，装好后正常

用户连接数同时达到55000，也不会进程僵死，速度还可以

zhangweizj

连接数太大而导致squid内存分配失败吗？，可惜不能看到已经编译好的squid的源代码。。，看看改了什么