重负载MYSQL自动重启的问题

ginew

我是4.1.22的mysql+rhel 2.6.9-34.ELsmp，4G内存。
最近2个月我的mysql每天至少会重启1到2次，有时候是在访问高峰期，有时候却是在凌晨没什么负载的情况下重启。
以下是重启时.err报的错误：
--------------------------------------------------------------------------------------------------------------------------
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=16773120
max_used_connections=201
max_connections=200
threads_connected=34
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2620638 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x310cf398
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb, backtrace may not be correct.
Bogus stack limit or frame pointer, fp=0xb, stack_bottom=0x4860000, thread_stack=196608, aborting backtrace.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x18fdb6c8  is invalid pointer
thd->thread_id=13294
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
--------------------------------------------------------------------------------------------------------------------------
我调过key_buffer_size，read_buffer_size，sort_buffer_size等参数，调到很低也照样会死掉。
我也看过coredump出来的core文件，没看出每次死掉的时候有什么共同点，也看不到在执行什么语句。。
现在我怀疑是linux2.6的内核，不允许单个进程使用内存超过3G的原因。
以上这台出问题的mysql，是我7台作用相同的数据库中的一台，因为硬件条件比较好，所以分配给他的负载比较重

希望各位遇到过相同情况的，帮我看看，为什么这个数据库的占用内存一直会增大，就算在凌晨也不释放，或者是不是有别的原因导致它死掉？

stormgenius

reboot系统再观察一下。mysql有时就是这样

yejr

1. 贴配置
2. 贴mysqlreport的结果

ginew

REBOOT过系统，也调整过kernel.shmmax等参数，但无法解决此问题，MYSQL依旧不定时重启

配置如下：
------------------------------------------------------------------------------------------------------------------
back_log = 90
max_connections = 200
table_cache = 2048
max_allowed_packet = 256M
binlog_cache_size = 1M
max_heap_table_size = 64M
sort_buffer_size = 16M
join_buffer_size = 64M
thread_cache = 8
thread_concurrency = 8
query_cache_size = 64M
query_cache_limit = 2M
ft_min_word_len = 4
default_table_type = INNODB
thread_stack = 192K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 128M
long_query_time = 10
log_long_format
slave-skip-errors=all
server-id = 222
master-host = xxx.xxx.xxx.xxx
master-user = xxx
master-password = xxxxxx
log_slave_updates = OFF
key_buffer_size = 256M
read_buffer_size = 16M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_max_extra_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
skip-bdb
innodb_additional_mem_pool_size = 128M
innodb_buffer_pool_size = 1800M
innodb_data_home_dir = /data/mysql/data
innodb_data_file_path = ibdata01:2000M;ibdata02:2000M;ibdata03:2000M;ibdata04:2000M;ibdata05:2000M;ibdata06:2000M;ibdata07:20
00M;ibdata08:2000M;ibdata09:2000M;ibdata10:2000M;ibdata11:2000M;ibdata12:2000M;ibdata13:2000M;ibdata14:2000M;ibdata15:2000M;i
bdata16:2000M
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_fast_shutdown
innodb_log_buffer_size = 32M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_log_group_home_dir = /data/mysql/data
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
----------------------------------------------------------------------------------------------------------------------------

下面是mysqlreport的结果，因为数据库13：58刚自动重启过，现在是15：00，才运行了一个多小时

__ Key _________________________________________________________________
Buffer used    24.63M of 256.00M  %Used:   9.62
  Current      53.45M            %Usage:  20.88
Write hit      99.93%
Read hit       99.56%

__ Questions ___________________________________________________________
Total           3.00M   765.7/s
Slow              467     0.1/s  %Total:   0.02  %DMS:   0.04
DMS             1.21M   310.3/s           40.53

__ Table Locks _________________________________________________________
Waited              1     0.0/s  %Total:   0.00
Immediate       1.80M   460.4/s

__ Tables ______________________________________________________________
Open              629 of 2048    %Cache:  30.71
Opened            640     0.2/s

__ Connections _________________________________________________________
Max used          201 of  200      %Max: 100.50
Total           5.51k     1.4/s

__ Created Temp ________________________________________________________
Disk table     58.56k    15.0/s
Table         100.08k    25.6/s
File              102     0.0/s


我曾经调过sort_buffer_size到8M，join_buffer_size到16M，read_buffer_size到8M，innodb_buffer_pool_size到1G等，都无法解决此问题。。
另外，磁盘IO很高，iowait经常保持在50-80%

yejr

让mysql多运行一阵子，再贴mysqlreport结果