squid cpu 99% 问题

suran007

大家好，我的系统是redhat as4 u5 x64,squid版本是2.6 stable 16,最近出现一些异常，时不时的squid会占用99% cpu，观察cache.log也没有什么异常，/var/log/message也没有问题，我检查syn连接数也不是很多，大家帮忙看看是什么原因导致的，我截取了当时的squid的状态信息

squidclient mgr:info信息如下：

[code]Squid Object Cache: Version 2.6.STABLE16
Start Time:     Tue, 12 Feb 2008 08:28:35 GMT
Current Time:   Tue, 19 Feb 2008 09:39:19 GMT
Connection information for squid:
        Number of clients accessing cache:      59027
        Number of HTTP requests received:       56468211
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   5563.0
        Average ICP messages per minute since start:    0.0
        Select loop called: 940948973 times, 0.647 ms avg
Cache information for squid:
        Request Hit Ratios:     5min: 93.1%, 60min: 91.5%
        Byte Hit Ratios:        5min: 90.1%, 60min: 89.0%
        Request Memory Hit Ratios:      5min: 17.2%, 60min: 15.9%
        Request Disk Hit Ratios:        5min: 17.8%, 60min: 19.4%
        Storage Swap size:      291632548 KB
        Storage Mem size:       524392 KB
        Mean Object Size:       57.54 KB
        Requests given to unlinkd:      0
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.37825  0.52331
        Cache Misses:          0.76407  0.89858
        Cache Hits:            0.27332  0.42149
        Near Hits:             0.58309  0.85130
        Not-Modified Replies:  0.04277  0.05331
        DNS Lookups:           0.00000  0.00000
        ICP Queries:           0.00000  0.00000
[\code]

squidclient mgr:utilization信息如下：
[code]Cache Utilisation:

Last 5 minutes:
sample_start_time = 1203413525.401003 (Tue, 19 Feb 2008 09:32:05 GMT)
sample_end_time = 1203413826.559752 (Tue, 19 Feb 2008 09:37:06 GMT)
client_http.requests = 148.171023/sec
client_http.hits = 137.724705/sec
client_http.errors = 0.000000/sec
client_http.kbytes_in = 69.405256/sec
client_http.kbytes_out = 4806.936557/sec
client_http.all_median_svc_time = 0.399283 seconds
client_http.miss_median_svc_time = 0.851301 seconds
client_http.nm_median_svc_time = 0.045193 seconds
client_http.nh_median_svc_time = 0.649683 seconds
client_http.hit_median_svc_time = 0.273318 seconds
server.all.requests = 65.148365/sec
server.all.errors = 0.000000/sec
server.all.kbytes_in = 476.210638/sec
server.all.kbytes_out = 39.716595/sec
server.http.requests = 65.148365/sec
server.http.errors = 0.000000/sec
server.http.kbytes_in = 476.210638/sec
server.http.kbytes_out = 39.716595/sec
server.ftp.requests = 0.000000/sec
server.ftp.errors = 0.000000/sec
server.ftp.kbytes_in = 0.000000/sec
server.ftp.kbytes_out = 0.000000/sec
server.other.requests = 0.000000/sec
server.other.errors = 0.000000/sec
server.other.kbytes_in = 0.000000/sec
server.other.kbytes_out = 0.000000/sec
icp.pkts_sent = 0.000000/sec
icp.pkts_recv = 0.000000/sec
icp.queries_sent = 0.000000/sec
icp.replies_sent = 0.000000/sec
icp.queries_recv = 0.000000/sec
icp.replies_recv = 0.000000/sec
icp.replies_queued = 0.000000/sec
icp.query_timeouts = 0.000000/sec
icp.kbytes_sent = 0.000000/sec
icp.kbytes_recv = 0.000000/sec
icp.q_kbytes_sent = 0.000000/sec
icp.r_kbytes_sent = 0.000000/sec
icp.q_kbytes_recv = 0.000000/sec
icp.r_kbytes_recv = 0.000000/sec
icp.query_median_svc_time = 0.000000 seconds
icp.reply_median_svc_time = 0.000000 seconds
dns.median_svc_time = 0.000000 seconds
unlink.requests = 0.000000/sec
page_faults = 0.000000/sec
select_loops = 329.268203/sec
select_fds = 1958.511921/sec
average_select_fd_period = 0.000511/fd
median_select_fds = 0.000000
swap.outs = 7.510989/sec
swap.ins = 116.636160/sec
swap.files_cleaned = 0.000000/sec
aborted_requests = 29.808199/sec
syscalls.polls = 329.268203/sec
syscalls.disk.opens = 69.544717/sec
syscalls.disk.closes = 139.096075/sec
syscalls.disk.reads = 920.298683/sec
syscalls.disk.writes = 134.679800/sec
syscalls.disk.seeks = 0.000000/sec
syscalls.disk.unlinks = 7.720181/sec
syscalls.sock.accepts = 91.895056/sec
syscalls.sock.sockets = 65.384121/sec
syscalls.sock.connects = 65.380800/sec
syscalls.sock.binds = 65.384121/sec
syscalls.sock.closes = 137.488949/sec
syscalls.sock.reads = 596.376498/sec
syscalls.sock.writes = 1380.624011/sec
syscalls.sock.recvfroms = 0.006641/sec
syscalls.sock.sendtos = 0.006641/sec
cpu_time = 279.568499 seconds
wall_time = 301.158749 seconds
cpu_usage = 92.830940%
[\code]

很不解为什么显示148个连接，命中137个，但是向后面realserver的请求还65个，而且aborted_requests = 29.808199/sec这个值也很高。大家帮忙看看是什么原因导致的。

suran007

请大侠们帮忙看看

ability

看看是不是IO负载太高了

suran007

原帖由 ability 于 2008-2-20 13:07 发表
看看是不是IO负载太高了

io不高，由于有台新上的机器，也出现这个cpu 99%问题

Dalamar

client_http.all_median_svc_time = 0.399283 seconds
client_http.miss_median_svc_time = 0.851301 seconds
client_http.nm_median_svc_time = 0.045193 seconds
client_http.nh_median_svc_time = 0.649683 seconds
client_http.hit_median_svc_time = 0.273318 seconds

这个些数据好像不太正常，squid是编译成epoll的么？

我这里的参考数据
sample_start_time = 1203575485.601540 (Thu, 21 Feb 2008 06:31:25 GMT)
sample_end_time = 1203575785.601714 (Thu, 21 Feb 2008 06:36:25 GMT)
client_http.requests = 807.622865/sec
client_http.hits = 773.612885/sec
client_http.errors = 0.000000/sec
client_http.kbytes_in = 444.193076/sec
client_http.kbytes_out = 4427.197432/sec
client_http.all_median_svc_time = 0.000000 seconds
client_http.miss_median_svc_time = 0.127833 seconds
client_http.nm_median_svc_time = 0.000000 seconds
client_http.nh_median_svc_time = 0.002856 seconds
client_http.hit_median_svc_time = 0.000000 seconds
server.all.requests = 34.153314/sec
server.all.errors = 0.000000/sec
server.all.kbytes_in = 874.422826/sec
server.all.kbytes_out = 19.109989/sec
server.http.requests = 34.153314/sec
server.http.errors = 0.000000/sec
server.http.kbytes_in = 874.422826/sec
server.http.kbytes_out = 19.109989/sec

suran007

原帖由 Dalamar 于 2008-2-21 14:38 发表
client_http.all_median_svc_time = 0.399283 seconds
client_http.miss_median_svc_time = 0.851301 seconds
client_http.nm_median_svc_time = 0.045193 seconds
client_http.nh_median_svc_time = 0.6496 ...

恩，squid是epoll的，上面数据是squid 占cpu 99%的时候截取的，所以squid的相关响应时间的值比较高

Dalamar

persistent_request_timeout 5 seconds
client_persistent_connections off
server_persistent_connections on


cache_swap_low 90
cache_swap_high 95

maximum_object_size 8192 KB
maximum_object_size_in_memory 1024 KB

ipcache_size 2048
ipcache_low 80
ipcache_high 95

这些参数尝试调整过么？

zouyi2005

你这个squid缓存使用的空间是用使用tmpfs 生成的内存空间，还是直接利用磁盘做的缓存空间？

Dalamar

Memory Hit Ratios 太低了，如果配置都正常的话，检查下各种命中率的情况。LZ是用的反向代理么？