求教，apache+tomcat+mod_jk 8009端口CLOSE_WAIT状态如何降低
本帖最后由 tianshanhanshui 于 2013-07-18 17:25 编辑大家好，有个问题请教
apache+tomcat+mod_jk8009端口出现大量CLOSE_WAIT状态
操作系统：
centos 6.2
apache 2.2
tomcat 7
tomcat-connectors-1.2.32
用 /bin/netstat -nat|grep CLOSE_WAIT ，如下图
tcp 1 0 127.0.0.1:17654 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17153 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17810 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17865 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17800 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17862 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17581 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17720 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17777 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17472 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17684 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17770 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17647 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17701 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17719 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17839 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:16734 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17637 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17384 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17648 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17659 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17884 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 127.0.0.1:17704 127.0.0.1:8009 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:4466 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:10984 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:14488 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:15399 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:15259 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:5395 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:10792 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:11712 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:8356 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:1156 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:5110 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:5386 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:64283 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:13236 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:3201 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:4768 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:57997 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:61811 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:5913 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:5569 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:1134 CLOSE_WAIT
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:13255 CLOSE_WAIT
操作系统相关内核参数调整如下
net.ipv4.tcp_keepalive_time=30
net.ipv4.tcp_keepalive_probes=2
net.ipv4.tcp_keepalive_intvl=2
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
workers.properties相关配置如下
connection_pool_timeout=600
socket_connect_timeout=30
tomcat的server.xml相关配置如下
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
connectionTimeout="30000">
访问某个jsp页面时，tcpdump抓包如下
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
16:57:21.470496 IP (tos 0x0, ttl 64, id 50782, offset 0, flags , proto TCP (6), length 52)
localhost.50634 > localhost.8009: Flags , cksum 0xec1d (correct), seq 2316770178, ack 2009568881, win 257, options , length 0
16:57:21.470516 IP (tos 0x0, ttl 64, id 0, offset 0, flags , proto TCP (6), length 40)
localhost.8009 > localhost.50634: Flags , cksum 0xbe91 (correct), seq 2009568881, win 0, length 0
16:57:21.471513 IP (tos 0x0, ttl 64, id 50311, offset 0, flags , proto TCP (6), length 60)
localhost.50639 > localhost.8009: Flags , cksum 0xe40d (correct), seq 2891312877, win 32792, options , length 0
16:57:21.471525 IP (tos 0x0, ttl 64, id 0, offset 0, flags , proto TCP (6), length 60)
localhost.8009 > localhost.50639: Flags , cksum 0x477f (correct), seq 435509117, ack 2891312878, win 32768, options , length 0
16:57:21.471536 IP (tos 0x0, ttl 64, id 50312, offset 0, flags , proto TCP (6), length 52)
localhost.50639 > localhost.8009: Flags [.], cksum 0x2fa3 (correct), seq 1, ack 1, win 257, options , length 0
16:57:21.472878 IP (tos 0x0, ttl 64, id 50313, offset 0, flags , proto TCP (6), length 435)
localhost.50639 > localhost.8009: Flags , cksum 0xffa7 (incorrect -> 0x2908), seq 1:384, ack 1, win 257, options , length 383
16:57:21.472887 IP (tos 0x0, ttl 64, id 32020, offset 0, flags , proto TCP (6), length 52)
localhost.8009 > localhost.50639: Flags [.], cksum 0x2e1a (correct), seq 1, ack 384, win 265, options , length 0
16:57:21.527597 IP (tos 0x0, ttl 64, id 32021, offset 0, flags , proto TCP (6), length 101)
localhost.8009 > localhost.50639: Flags , cksum 0xfe59 (incorrect -> 0xdfaa), seq 1:50, ack 384, win 265, options , length 49
16:57:21.527655 IP (tos 0x0, ttl 64, id 50314, offset 0, flags , proto TCP (6), length 52)
localhost.50639 > localhost.8009: Flags [.], cksum 0x2d83 (correct), seq 384, ack 50, win 257, options , length 0
16:57:21.527833 IP (tos 0x0, ttl 64, id 32022, offset 0, flags , proto TCP (6), length 125)
localhost.8009 > localhost.50639: Flags , cksum 0xfe71 (incorrect -> 0x9740), seq 50:123, ack 384, win 265, options , length 73
16:57:21.527877 IP (tos 0x0, ttl 64, id 50315, offset 0, flags , proto TCP (6), length 52)
localhost.50639 > localhost.8009: Flags [.], cksum 0x2d3a (correct), seq 384, ack 123, win 257, options , length 0
16:57:21.527991 IP (tos 0x0, ttl 64, id 32023, offset 0, flags , proto TCP (6), length 58)
localhost.8009 > localhost.50639: Flags , cksum 0xfe2e (incorrect -> 0xe6de), seq 123:129, ack 384, win 265, options , length 6
16:57:21.528016 IP (tos 0x0, ttl 64, id 50316, offset 0, flags , proto TCP (6), length 52)
localhost.50639 > localhost.8009: Flags [.], cksum 0x2d34 (correct), seq 384, ack 129, win 257, options , length 0
自己测试tcp状态的转换过程，（用浏览器访问一个jsp页面，页面完全打开后关闭浏览器）
浏览器没访问的时候
1 established)
1 ESTABLISHED
1 State
6 LISTEN
浏览器访问，并关闭浏览器，将出现以下几个过程依次为
1 established)
1 State
1 SYN_RECV
1 TIME_WAIT
4 ESTABLISHED
6 LISTEN
1 established)
1 FIN_WAIT2
1 State
1 TIME_WAIT
4 ESTABLISHED
6 LISTEN
1 established)
1 State
2 TIME_WAIT
3 ESTABLISHED
6 LISTEN
1 CLOSE_WAIT
1 established)
1 ESTABLISHED
1 State
6 LISTEN
close_wait最后出现，从浏览器访问jsp到，最终出现close_wait的时间为30秒，为tomcat配置的connectionTimeout="30000"
请问如何降低close_wait状态的数量 apache返回页面上有没有什么信息？ 回复 2# sohusina
CLOSE_WAIT，一直增长到几千个的时候，页面会报503错误！只能重启tomcat。
如果jsp页本身没有问题的话，有可能是apache或者tomcat配置文件问题。从tomcat报错的信息比较容易找出问题来。 回复 4# sohusina
你好，谢谢您的回复！
当CLOSE_WAIT持续增长，会耗尽内存，tomcat会得到 PerGen out of memory的提示，其他的就找不到有用的信息了。（日志不能贴出来，不好意思！）
但我也做了一些测试
改变tomcat 配置文件 connectionTimeout=“20000” 到connectionTimeout=“80000”
会减少下面这类CLOSE_WAIT.(apache为发起端，tomcat为接收端)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 1 0 127.0.0.1:2783 127.0.0.1:9009 CLOSE_WAIT
但会增加 (tomcat为发起端，apache为接收端)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 1 0 127.0.0.1:9009 127.0.0.1:58657 CLOSE_WAIT
我还没搞清楚 apache,jk,tomcat之间 为什么会产生 CLOSE_WAIT，这需要查一些文件，做一些调整试验。 试试tomcat中的catalina.sh 中加：
JAVA_OPTS="-Xms1024m -Xmx1024m" sohusina 发表于 2013-07-22 12:13 static/image/common/back.gif
试试tomcat中的catalina.sh 中加：
JAVA_OPTS="-Xms1024m -Xmx1024m"
谢谢，sohusina兄弟。
现在看来不是这个问题，就算加大PerGen区域的内存，也会有耗尽的时候，因为大量close_wait会耗尽系统内存。
应该还是apache或tomcat或者jk配置的问题。
我先测试一下参数，然后贴出结果，也希望遇到过这个问题的朋友提示一些解决方向或参数调整方向。 本帖最后由 tianshanhanshui 于 2013-07-24 18:03 编辑
网上查找及测试，有两组参数能降低8009的close_wait数量
法1
1、内核参数
#当keepalive起用的时候,TCP发送keepalive消息的频度
net.ipv4.tcp_keepalive_time=30
#TCP发送keepalive探测以确定该连接已经断开的次数
net.ipv4.tcp_keepalive_probes=2
TCP发送keepalive探测无应答时，重发间隔（单位：秒）
net.ipv4.tcp_keepalive_intvl=2
centos调整完后需要重新启动计算机才能生效
内核参数调整是对真个系统而言，使用的时候要慎重。
2、workers.properties
worker.tomcat1.socket_keepalive=1
1、2两条必须都被编辑，才能产生效果，只编辑其中一条，不起作用
法2
1、在apache配置文件里加入
#watchdog线程产生的时间间隔
JkWatchdogInterval 60 （单位是秒 ）
2、在workers.properties里加入
worker.tomcat1.connection_ping_interval=1
worker.tomcat1.ping_timeout=1000
worker.tomcat1.ping_mode=A
参考文件
http://tomcat.apache.org/connectors-doc/reference/workers.html
http://blog.chinaunix.net/uid-72212-id-2657630.html
页:
[1]