NFS读数据速度异常问题，请教

kindblue

现有一台sco openserver 5.0.6,使用其他服务器nfs share过来的卷，
从nfs卷读数据速度正常，大概10MB/s（百兆以太网），但是往nfs卷上写
数据速度非常的慢，大概只有几百KB/s，请问，应该这可能是哪方面引起的
问题？

如果使用ftp往这个nfs server share出来的卷传数据，速度是正常的，所以
问题还应该是nfs方面的，哪位对这方面问题有过经验或思路，请多帮忙

多谢！

CNL

在NFS服务器上增加nfsd守护进程可能会加快写的速度
详见：
http://wdb1.sco.com/kb/showta?taid=102447&qid=870605556&sid=908356439&pgnum=1

1. At the shell prompt, type:


                  ps -ef | grep <daemon>;


where <daemon>; is:

                 - "nfsd", if this is the NFS file server
                 - "biod", if this is the NFS client


For example:

                  ps -ef | grep nfsd


2. Determine which daemons need increasing or decreasing.
If all processes displayed have an accumulated time, you may want to increase the number of daemons that are started. If one or more processes show a zero time accumulated, then you may want to reduce the number of daemons started. An example of the output to "ps -ef | grep nfsd" is:


       root   381   379  0  Feb  7   ?        0:11 nfsd 4
       root   382   379  0  Feb  7   ?        0:10 nfsd 4
       root   383   379  0  Feb  7   ?        0:09 nfsd 4
       root   379     1  0  Feb  7   ?        0:09 nfsd 4
       root  9421  6121  1 17:55:35  p4       0:00 grep nfsd
                                              ^^^^

Notice that each of the nfsd daemons has time accumulated.
3. Make the changes as follows:

a. Stop NFS with:


                    nfs stop


b. Edit /etc/nfs and change the number on the line that starts the daemon (either nfsd or biod). For example, if the /etc/nfs file starts the nfsd daemon as:

                    nfsd 4


and the "ps -ef | grep nfsd" revealed that all processes had accumulated time, then change the line to:

                    nfsd 5


Note: For SCO NFS 2.0.0 on OpenServer Release 5, the syntax for the options are slightly different. Also, the nfsd daemons are started by inetd on an as-needed basis. For example, if the line that starts the nfsd daemon reads:

                    nfsd -u 4 -t 16 &


it means there can be a maximum of 4 nfsd daemons running that use the UDP protocol and 16 nfsd daemons that use the TCP protocol.
c. Restart NFS with:


                    nfs start


You can verify the number of daemons running by repeating step 1 again. Continue to monitor your system periodically to see if further tuning is necessary. See the SEE ALSO section for references to Streams, UDP and TCP tuning, along with more detailed information on nfsd and biod tuning.

Notes
(1) You cannot increase the number of nfsd daemons to an arbitrarily large value. As the number of nfsd daemons increases, context switching and scheduling overhead becomes more of a factor.
(2) Increasing the number of biod processes above the default only allows an NFS client to send more RPC requests at one time; it will not improve performance, and often makes it worse, if server request handling bandwidth is a constraint.

kindblue

[quote]原帖由 "CNL"][/quote 发表：

非常感谢，我试试看

kindblue

好像不行，没效果

CNL

如果ftp传输正常，应可排除网络连接以及硬盘写速度的原因
用nfsstat检查一下先看看NFS的状态有何异常
服务器端：# nfsstat -s

The fields displayed in the output indicate if the server is having problems receiving or removing packets from the NFS service socket.

SERVER RPC statistics:

calls: This is a count of the total number of NFS RPC calls made to the server.

badcalls: A count of the RPC requests that were rejected by the server by the RPC mechanism, before they were passed to the kernel. An RPC request can be rejected if there was an authentication failure, for example, the client did not present valid credentials or a client attempted a write operation on a file system that was exported as read-only.

nullrecv: This is incremented whenever an nfsd daemon was scheduled to run but found no packet on the NFS service socket queue. If this is greater than 0, it indicates that there may be too many instances of nfsd running on this server.

badlen/xdrcall: A count of RPC requests received by the server that were too short(badlen) or the XDR(external data representation) headers in the packet were incorrect (xdrcall). This points to an underlying network problem that is causing these packets to get corrupted.

SERVER NFS statistics:

This section details the activity at the NFS application level. It lists the different call types, each corresponding to a procedure within the NFS RPC service.

calls: same as above

badcalls: same as above

null: A null procedure is included in every RPC program. A successful return from a call to null shows that the server is alive and the network is operational. It shouldn't be greater than 1%.

getattr: Number of get file (and directory) attribute calls, such as modification time, size, owner, etc. This information is cached to avoid having to contact the server every time it is required. If this is too high (greater than 50%), check the attribute cache value on the client to make sure the cache window is not set to zero.

setattr: Number of set file attribute calls as a result of changing a remote file's attributes.

lookup: Number of lookup requests. A lookup request takes a filename and file handle for a directory and returns a file handle pointing to the file on the server.

readlink: Number of read-symbolic-link value requests. Not applicable under SCO NFS Release 1.1.*, since we do not support symbolic links in these releases.

read, write, create, remove, rename: Number of requests to read, write, create, remove and rename a file.

link: Number of requests to create a hard link on the server.

symlink: Number of requests to create symbolic links. SCO NFS 1.1.* does not support this and hence the count would be 0.

mkdir, rmdir, readdir: Number of requests to create, remove and read a directory of files.

fsstat: Number of requests to return information about the mounted file system.

客户端：# nfsstat -c

The client RPC statistics indicate the success rate of the client in reaching NFS servers. The individual fields and what they represent are as follows:

CLIENT RPC Statistics:

calls: Total number of client requests to NFS servers.

badcalls: Number of RPC requests that returned an error. If a request fails due to a timeout (the server did not respond in the given time interval) or an interruption, this value is incremented.

retrans: Number of requests that were retransmitted because no response was received from the NFS server within the timeout interval.

timeout: Number of calls that timed out waiting for a response from the server. If this is high, it could mean that the requests are not reaching the server.

badxid: The XID is a serial number that uniquely identifies the request. Multiple retransmissions of the same request are sent with the same XID. If a client receives a response with an XID for which there is no outstanding RPC request, then this value is incremented. This means that the client has already received a reply to that request. This points to the fact that the server is slow in responding to all NFS requests.


                   If this value is low, for example, close to 0, and timeout
                   is high, it indicates that the network is dropping packets.
                   The mount command uses a default setting of 8k for read
                   and write buffer sizes.  Reduce these values using the
                   rsize, wsize options.

                   If this value is almost equal to the number in the timeout
                   field, increase the value of the timeout option in the mount
                   command, to reduce excessive retransmissions.


wait: Number of requests that had to wait for an available client handle. If this number is high, the number of nfsclienthandles on the server should be increased. On SCO systems, this is set to 4, by default. To increase this, change the value of x in:

                nfsclnt x

                in the file /etc/nfs.  You will need to restart NFS to make
                this change effective.


newcred: Number of times the client authentication information had to be refreshed. Does not apply in the case of SCO NFS 1.1.*, since these releases do not support Secure RPC.
CLIENT NFS Statistics:

The client NFS statistics provide almost the same information as discussed in the server NFS statistics, with a few additions:

nclget: Number of times the client had to request for a new client handle for an NFS call.

nclsleep: Number of times the client had to wait because no client handle was available. Ideally, this should be zero.

你可把结果贴出，看看哪位在用NFS的做个对比帮你找一下原因。

kindblue

nfs put结果（速度异常）：
# nfsstat -c

Client rpc:
calls      badcalls   retrans    badxid     timeout    wait       newcred
89713      0          1          0          1          0          0
peekeers   badresps
0          0

Client nfs:
calls      badcalls   nclget     nclsleep
89701      0          89716      0
null       getattr    setattr    root       lookup     readlink   read
0  0%      39  0%     0  0%      0  0%      20  0%     0  0%      12801 14%
wrcache    write      create     remove     rename     link       symlink
0  0%      76815 85%  6  0%      4  0%      0  0%      0  0%      0  0%
mkdir      rmdir      readdir    fsstat
0  0%      0  0%      10  0%     6  0%


nfs get 结果（速度正常）：
# nfsstat -c

Client rpc:
calls      badcalls   retrans    badxid     timeout    wait       newcred
102517     0          1          0          1          0          0
peekeers   badresps
0          0

Client nfs:
calls      badcalls   nclget     nclsleep
102505     0          102520     0
null       getattr    setattr    root       lookup     readlink   read
0  0%      41  0%     0  0%      0  0%      21  0%     0  0%      25602 24%
wrcache    write      create     remove     rename     link       symlink
0  0%      76815 74%  6  0%      4  0%      0  0%      0  0%      0  0%
mkdir      rmdir      readdir    fsstat
0  0%      0  0%      10  0%     6  0%


用于测试的文件是一个100M的文件。

CNL

在NFS服务器上执行
# nfsstat -s
内容贴出来看

NFS的问题一般是硬件问题(网卡可用netstat -i或netstat -m，CPU用sar)，
尤其是服务端的硬件如CPU能力必须够用，如果负担过重，处理速度自然就慢的多；
如果NFS的应用软件设计的有问题也会造成异常。

下面是服务端NFS的nfsstat解释，你可参考一下自己找找原因

The server RPC fields indicate if the server is receiving packets that are
damaged in transit, or if there are problems removing the packets from the
NFS service socket. Example:

Server rpc:
calls      badcalls   nullrecv   badlen     xdrcall
4          0          0          0          0

calls -    Total number of NFS RPC calls made to the server, from all clients.
            RPC calls made for other services, such as NIS, are not included
            in this count.

badcalls - RPC requests that were rejected by the server's RPC mechanism,
            before the request was passed to the NFS service routines in
            the kernel.  An RPC call will be rejected if there is an
            authentication failure, where the calling client does not present
            valid credentials.  Similarly, badcalls is incremented if root
            on a client attempts to perform write or modification operations
            on a filesystem that is not exported with "root" permissions
            enabled.

nullrecv - This field is incremented whenever an nfsd daemon is scheduled
            to run but finds that there is no packet on the NFS service
            socket queue. If the server is running an excessive number
            of nfsd daemons, it is possible that there will be more runnable
            daemons than requests to drain from the NFS socket, therefore
            some daemons wake up but do not receive any data. The number
            of nfsd daemons that are started up are by default is 4. This
            number can be adjusted by editing the /etc/nfs start/stop script.
            If this number is >; 0, then the nfsd daemons should be reduced
            until it is not incremented.

badlen/xdrcall - The RPC request received by the server was too short (badlen)
            or the XDR headers in the packet are malformed (xdrcall). It is
            possible that packet was truncated or damaged by a network problem.
            On a local area network it is very rare for this problem to occur,
            a wide area network is more likely to cause this type of problem.


The server nfs statistics show the total number of NFS calls made to the
server and are broken down by the procedure of those calls handled. Each
call type corresponds to a procedure within the NFS RPC service.

Server nfs:
calls      badcalls
0          0

calls -    total number of calls to the server

badcalls - total number of bad calls to the server, see RPC service for
            a breakdown of badcalls.  These fields can indicate problems
            caused by having a user in too many groups, attempts to access
            exported filesystems as root, or an improper secure RPC
            configuration.

null       getattr    setattr    root       lookup     readlink   read
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%

null -    The null procedure is included in every RPC program for "ping'ing"

the RPC server and is designed to ensure that the network is operational and that the server host is alive. The "rpcinfo(NADM)" calls the null procedure to check RPC server health, and the "automounter" calls the null procedure of all NFS severs in parallel when multiple machines are listed for a single mount point. The automounter and rpcinfo should account for the total null calls.

root -    number of calls to nfsd for root permissions.

wrcache    write      create     remove     rename     link       symlink
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%

wrcache - number of calls to the read write cache, this indicates the
           amount of work biod is performing.

mkdir      rmdir      readdir    fsstat
0 0%       0 0%       0 0%       0 0%

The rest of the fields indicate an NFS request generated by UNIX commands.
The table below illustrates some examples of which UNIX commands cause the
RPC fields to be incremented.

UNIX CMD         NFS RPC Operation
----------------------------------------------------
cp                  read, write, setattr
find                  lookup, readdir, getattr
ln                  symlink
ls                  lookup, getattr, readdir, readlink
mv                  rename
mkdir           mkdir
rmdir           rmdir
rm              remove, (with -rf option, rmdir)

Fields to check for problem areas:

symlink >; 10%. Clients are making excessive use of symbolic links that are
on filesystems exported by the server. Replace the symbolic link with a
directory, and mount both the underlying filesystem and the link's target
on the client.

getattr >; 60%. Check for possible non-default attribute cache values on NFS
clients.  A very high percentage of "getattr" requests indicate that the
attribute cache window has been reduced or set to zero with the "noac"
mount option.

null >; 1%. The automounter has been configured to mount replicated filesystems,
but the timeout values for the mount are too short.  The null procedure calls
are made by the automounter to locate a server for the filesystem; too many
null calls indicate that the automounter is retrying the mount frequently.
Increase the mount timeout parameter (timeo) on the automounter command line.

kindblue

服务器端应该没有问题，网络里还有其他平台的nfs client，写的速度就很正常，所以真是觉得很奇怪，摸不着头绪

CNL

如果其他平台的nfs client向你这个nfs卷上写速度正常，那就要考虑你客户端的问题了
其他平台如果和你的客户端同机，那么看你的软件
如果不是，看你的机器配置，netstat -i/netstat -m/sar查看系统负载

你的nfsstat -c的结果我觉的看起来没有什么毛病

kindblue

奇怪的问题，是不是需要打什么patch？