ibm3850M2外挂EMC CX500，IO性能实在差的让人无法接受！

remen

我们去年新购IBM3850M2，安装REDHAT5,走SAN网连接EMC CX500,但发现在emc上拷贝数据奇慢无比，每秒只有6-8M的速度，在本地磁盘内部拷贝数据没问题在40-50M/S，本地盘和EMC之间拷贝数据也没问题，在50-60M/S，唯独在EMC LUN上拷贝数据奇慢无比！
请问都有哪些可能呢？
# time cp STAGD.dbf STAGD.dbf.bak
real 12m13.588s
user 0m0.598s
sys 0m22.529s
# du -sk STAGD.dbf
5125020 STAGD.dbf
$ sqlplus "/as sysdba"
SQL> select 5125020/1024/733.588 from dual;
5125020/1024/733.588
--------------------
6.82249757
可以看到每秒的速度只有6M左右，太惨了吧！

remen

回复 1# remen


    同样的lun分配给solaris机器，一切正常，io在50M/s，如下：
Solaris:
Emc 　emc
$ time cp system01.dbf system01.dbf.bak

real 0m19.65s
user 0m0.00s
sys 0m6.33s
$ du -sk system01.dbf
1024520 system01.dbf
$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Jul 1 22:18:37 2010

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select 1024520/1024/19.65 from dual;

1024520/1024/19.65
------------------
50.9164281

remen

但是IBM3850本地OS盘拷贝数据没问题，如下：
Os disk-Os disk
# du -sk BIEED.dbf
1025012 BIEED.dbf

# time cp BIEED.dbf BIEEDbk.dbf
real 0m23.942s
user 0m0.122s
sys 0m4.443s

# pwd
/biee_bak/BIEE

SQL> select 1025012/1024/23.942 from dual;

1025012/1024/23.942
-------------------
41.8088832
系统os盘41M/S

remen

IBM3850M2从OS盘拷贝数据到EMC也没问题
Os-emc

# du -sk *
1025012 BIEEDbk.dbf
1025012 BIEED.dbf
1025012 bosunx01.dbf
102512 orabpel.dbf
51260 redo02.log

# time cp STAGD.dbf /u4/STAGD.dbf

real 1m11.555s
user 0m0.543s
sys 0m21.227s

SQL> select 4617888/1024/71.555 from dual;

4617888/1024/71.555
-------------------
63.0236357
63m/s

remen

HBA卡信息如下：
#lspci 　v
。。。。。。。。。。。。
Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
Subsystem: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter
Flags: bus master, fast devsel, latency 0, IRQ 225
Memory at f6340000 (64-bit, non-prefetchable) [size=4K]
Memory at f6341000 (64-bit, non-prefetchable) [size=256]
I/O ports at 3500 [size=256]
[virtual] Expansion ROM at e2100000 [disabled] [size=256K]
Capabilities: [58] Power Management version 2
Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/4 Enable-
Capabilities: [44] Express Endpoint IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [12c] Power Budgeting
。。。。。。。。。。。。。。

remen

fdisk -l的输出
# fdisk -l

Disk /dev/sda: 145.9 GB, 145999527936 bytes
255 heads, 63 sectors/track, 17750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 17750 142472452+ 8e Linux LVM

Disk /dev/sdb: 405.8 GB, 405874409472 bytes
255 heads, 63 sectors/track, 49344 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 49344 396355648+ 83 Linux

Disk /dev/sdc: 52 MB, 52428800 bytes
2 heads, 50 sectors/track, 1024 cylinders
Units = cylinders of 100 * 512 = 51200 bytes

Device Boot Start End Blocks Id System
[root@redhat1 ~]#

remen

回复 6# remen


   

remen

这个问题真的要难道所有人？大家帮忙看看？现在就是没思路，大家给提个醒！

spook

你powerpath的版本换换看，EMC 上面的两个 lun 放到 Solaris如果没问题，就要考虑 powerpath 和驱动的问题……

还有就是队列深度，和文件Cache的问题

wolfheader

难倒我了{:2_176:}