服务器出现严重问题了，大家帮忙！！

白狐狸

All mbuf clusters exhausted ,  Please see tuning(7)
我的服务器现在一启动就出现上面的错误信息，满屏满屏的滚，根本无法登陆，按alt + f3(f4、f5等)则可以登陆，从而导致服务器严重丢包（丢90%以上），大家帮忙看看是怎么回事啊？

quakelee

先把网线罢了看看

白狐狸

对了，我的服务器主要跑的是 apache + mysql，用来提供免费个人主页的，跑了几个月都很好，今天晚上突然出现不能访问的现象，机房人员重启服务器后就说出现上面的错误，郁闷啊，从没遇到这样的问题，幸好不是硬盘坏，心里总算塌实了点啊，各位英雄给指点指点啊

白狐狸

quakelee，我现在不在机房啊，离服务器一千多公里呢，我ping服务器，基本上丢包率在99%以上，根本无法远程登陆进去，真正体会到干着急是什么意境了
      

quakelee

你让机房的人先把网线拔了再重启看看

白狐狸

这是我在 google 上找到的相关内容：

RH Linux has a (really dumb) default of using UDP mounts with a 16K
read and write size.  A small percentage of packet loss will cause
many fragments to build up on the IP reassembly queues and run you out
of mbuf clusters.

There are 3 things you could do to fix this:

a) Use more reasonable NFS mount options on the linux boxes.  8K UDP
should help quite a bit, 32K TCP should totally eliminate the problem.

b) Drastically reduce net.inet.ip.maxfragpackets (make sure that
(maxfragpackets * (16384/1472)) < (maxmbufclusters - some_constant).
This will limit the amount of packets (and therefor mbufs) on the IP
reassembly queues.  This may could destroy your observed NFS write
performance from the linux side if you do not also implement (a)
above.

c) Eliminate all packet loss in your network between the client and
server.  Not likely,  as I suspect some of the packet loss might even
be inside the linux or freebsd boxes.

I suggest you do both (a) and (b).

白狐狸

跟NFS有什么关系呀，我服务器根本没开启这个什么NFS服务，是不是/var/log日志文件把整个/var分区给占满了呀

quakelee

我记得其实增加
maxusers就能增加mbuf的大小
你原来设置的是多少？
有可能有人进攻你的机器，当然是dos，所以先把网线拔了看看，如果不出这个了，最好能让人在本地修改一下内核配置

白狐狸

这需要重新编译内核吗？我再找找啊，刚刚在网上看到解决方法，但还是看不太明白啊

白狐狸

网上找的相关资料，研究中。。。。大家也帮我看看啊，急死了     
Content-Transfer-Encoding: quoted-printable

I killed my FreeBSD 4.8-PRE (2003-02-1 server today by exhausting the
available mbufs.  I'd seen warnings like "All mbuf clusters exhausted,
please see tuning(7)." in /var/log/messages, so I added
`kern.ipc.nmbclusters=3D"16384"' to /boot/loader.conf.local a while back.

This server has 768MB of memory, and I run at least two copies of PostgreSQL
(it has multiple jails) at any given time and use it to server NFS to a
couple of Linux clients.  Today I wanted to test Bochs on one of the Linux
machines, ran the bochsconf program, watched it run "createdisk" to make a
512MB disk image in my home directory, and saw the server panic within a few
seconds.  When it came back up, I bumped nmbclusters to 32768, rebooted, and
tried createdisk it.  This time I watched `netstat -m' as the dreaded "All
mbuf clusters exhausted..." message began to scroll up the screen.  This
time, the server stayed alive, but it clearly maxed out all available
buffers.  After createdisk was finished, the mbuf usage went back to normal:

    $ netstat -m
    389/36736/131072 mbufs in use (current/peak/max):
            389 mbufs allocated to data
    388/32768/32768 mbuf clusters in use (current/peak/max)
    74720 Kbytes allocated to network (33% of mb_map in use)
    345 requests for memory denied
    1 requests for memory delayed
    0 calls to protocol drain routines

My questions are these:

1) Is there a guideline for how many nmbclusters to allocate?  Short of
   running out of physical memory, is there such a thing as too many?

2) I've already allocated 64MB to network buffering.  This seems like a huge
   amount to me.  Is it?

3) Should running out of mbufs be expected to crash the server, or should I
   try to reproduce it and file a PR?

4) What could createdisk have been doing to spike usage that heavily?

5) Could Linux's NFS client implementation have aggravated the situation?