CentOS4.4 + RHCS(DLM) + GFS + Oracle10gR2 RAC + VM

hgdxzhj

CentOS4.4 + RHCS(DLM) + GFS + Oracle10gR2 RAC + VMWare Server 1.0.1 安装
****************************************************************************
* CentOS4.4 + RHCS(DLM) + GFS + Oracle10gR2 RAC + VMWare Server 1.0.1 安装 *
****************************************************************************
一、测试环境
        主机：一台PC,AMD-64位的芯片，4G内存,安装CentOS-4.4-x86_64版本的操作系统
        在这个主机上面安装了2个虚拟机，全部安装CentOS-4.4-x86_64版本的操作系统，未进行内核定制，网上更新到最新
二、安装 VMWare Server 1.0.1 for linux
三、创建共享磁盘
        vmware-vdiskmanager -c -s 6Gb -a lsilogic -t 2 "/vmware/share/ohome.vmdk"   |用于 Shared Oracle Home
        vmware-vdiskmanager -c -s 10Gb -a lsilogic -t 2 "/vmware/share/odata.vmdk"  |用于 datafiles and indexes
        vmware-vdiskmanager -c -s 3Gb -a lsilogic -t 2 "/vmware/share/oundo1.vmdk"  |用于 node1 Redo logs and Undo tablespaces
        vmware-vdiskmanager -c -s 3Gb -a lsilogic -t 2 "/vmware/share/oundo2.vmdk"  |用于 node2 Redo logs and Undo tablespaces
        vmware-vdiskmanager -c -s 512Mb -a lsilogic -t 2 "/vmware/share/oraw.vmdk"  |用于 Oracle集群注册表文件和CRS表决磁盘
        2个虚拟机使用一个共享磁盘
        
四、安装虚拟机
        1. 在vmware console 创建 vmware guest OS, 取名 gfs-node01, 选择custome create-> Redhat Enterprise Linux 4 64-bit，其它都是默认.
           内存选择1G(>800MB你就看不到warning了), 硬盘大小选择12GB, 建立方式不选择 pre-allocated
        2. 创建好后vmware guest OS之后, 给guest 加上一块NIC(也就是网卡)
        3. 关掉vmware console， 在node1目录下面，打开gfs-node1.vmx, 在最后空白处添加以下内容
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.filename = "/vmware/share/ohome.vmdk"
scsi1:1.deviceType = "disk"
scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.filename = "/vmware/share/odata.vmdk"
scsi1:2.deviceType = "disk"
scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.filename = "/vmware/share/oundo1.vmdk"
scsi1:3.deviceType = "disk"
scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.filename = "/vmware/share/oundo2.vmdk"
scsi1:4.deviceType = "disk"
scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.filename = "/vmware/share/oundo3.vmdk"
scsi1:5.deviceType = "disk"
scsi1:6.present = "TRUE"
scsi1:6.mode = "independent-persistent"
scsi1:6.filename = "/vmware/share/oraw.vmdk"
scsi1:6.deviceType = "disk"
disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.DataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
        这段是对vmware使用共享硬盘的方式进行定义，大多数人都知道设置 disk.locking ="false" 却漏掉dataCache
        保存退出之后，重新打开你的vmware-console,你就可以看到vmware guest OS的配置中，都有这些硬盘出现了.
五、需要安装的包以及顺序
        可以用yum安装:
        1、升级CentOS4.4
                yum update
        2、安装csgfs
                yum install yumex
                cd /etc/yum.repos.d
                wget http://mirror.centos.org/centos/4/csgfs/CentOS-csgfs.repo
                yumex
        也可以手动rpm安装:
    包下载地址：http://mirror.centos.org/centos/4/csgfs/x86_64/RPMS/
        1、在所有节点上安装必须的软件包，软件包完整列表请参考GFS6.1用户手册
rgmanager                                — Manages cluster services and resources
system-config-cluster        — Contains the Cluster Configuration Tool, used to graphically configure the cluster and the display of the current status of the nodes, resources, fencing agents, and cluster services
ccsd                                        — Contains the cluster configuration services daemon (ccsd) and associated files
magma                                        — Contains an interface library for cluster lock management
magma-plugins                        — Contains plugins for the magma library
cman                                        — Contains the Cluster Manager (CMAN), which is used for managing cluster membership, messaging, and notification
cman-kernel                                — Contains required CMAN kernel modules
dlm                                                — Contains distributed lock management (DLM) library
dlm-kernel                                — Contains required DLM kernel modules
fence                                        — The cluster I/O fencing system that allows cluster nodes to connect to a variety of network power switches, fibre channel switches, and integrated power management interfaces
iddev                                        — Contains libraries used to identify the file system (or volume manager) in which a device is formatted Also, you can optionally install Red Hat GFS on your Red Hat Cluster Suite. Red Hat GFS consists of the following RPMs:
GFS                                                — The Red Hat GFS module
GFS-kernel                                — The Red Hat GFS kernel module
lvm2-cluster                        — Cluster extensions for the logical volume manager
GFS-kernheaders                        — GFS kernel header files
        2、安装软件和顺序
安装脚本，install.sh
#!/bin/bash
rpm -ivh kernel-smp-2.6.9-42.EL.x86_64.rpm
rpm -ivh kernel-smp-devel-2.6.9-42.EL.x86_64.rpm
rpm -ivh perl-Net-Telnet-3.03-3.noarch.rpm
rpm -ivh magma-1.0.6-0.x86_64.rpm
rpm -ivh magma-devel-1.0.6-0.x86_64.rpm
rpm -ivh ccs-1.0.7-0.x86_64.rpm
rpm -ivh ccs-devel-1.0.7-0.x86_64.rpm
rpm -ivh cman-kernel-2.6.9-45.4.centos4.x86_64.rpm
rpm -ivh cman-kernheaders-2.6.9-45.4.centos4.x86_64.rpm
rpm -ivh cman-1.0.11-0.x86_64.rpm
rpm -ivh cman-devel-1.0.11-0.x86_64.rpm
rpm -ivh dlm-kernel-2.6.9-42.12.centos4.x86_64.rpm
rpm -ivh dlm-kernheaders-2.6.9-42.12.centos4.x86_64.rpm
rpm -ivh dlm-1.0.1-1.x86_64.rpm
rpm -ivh dlm-devel-1.0.1-1.x86_64.rpm
rpm -ivh fence-1.32.25-1.x86_64.rpm
rpm -ivh GFS-6.1.6-1.x86_64.rpm
rpm -ivh GFS-kernel-2.6.9-58.2.centos4.x86_64.rpm
rpm -ivh GFS-kernheaders-2.6.9-58.2.centos4.x86_64.rpm
rpm -ivh iddev-2.0.0-3.x86_64.rpm
rpm -ivh iddev-devel-2.0.0-3.x86_64.rpm
rpm -ivh magma-plugins-1.0.9-0.x86_64.rpm
rpm -ivh rgmanager-1.9.53-0.x86_64.rpm
rpm -ivh system-config-cluster-1.0.25-1.0.noarch.rpm
rpm -ivh ipvsadm-1.24-6.x86_64.rpm
rpm ivh piranha-0.8.2-1.x86_64.rpm --nodeps
注意：有些包有依赖关系，使用nodeps开关进行安装即可
        3、修改各个节点上的/etc/hosts文件（每个节点都一样）
        如下：
        [root@gfs-node1 etc]# cat hosts
        # Do not remove the following line, or various programs
        # that require network functionality will fail.
        127.0.0.1        localhost.localdomain localhost
                192.168.154.211 gfs-node1
                192.168.154.212 gfs-node2
        192.168.10.1    node1-prv
        192.168.10.2    node2-prv
                192.168.154.201 node1-vip
                192.168.154.202 node2-vip
        注意：主机名、cluster主机名、ocs中的pub节点名最好相同。
六、运行system-config-cluster进行配置
增加2个节点，节点的权置全部设置为1，即Quorum值设置为1
三个节点的名称为:
gfs-node1
gfs-node2
修改cluster.conf文件，如下：
[root@gfs-node1 ~]# cat /etc/cluster/cluster.conf
        
        
               
                        
                                
                                       
                                
                        
               
               
                        
                                
                                       
                                       
                        
               
        
               
        
        
               
        
        
        
               
                        
                                
                                
                                
                        
               
        
[注意] Use fence_bladecenter.  This will require that you have telnet enabled on
        your management module (may require a firmware update)
使用scp命令把这个配置文件copy到node2节点上
七、 在01/02节点上启动dlm,ccsd,fence等服务  
        可能在安装配置完上述步骤后，下面这些服务已经起来了。
        在2个节点上加载dlm模块  
        [root@gfs-node1 cluster]# modprobe lock_dlm
        [root@gfs-node2 cluster]# modprobe lock_dlm
        5.2、启动ccsd服务  
        [root@gfs-node1 cluster]# ccsd
        [root@gfs-node2 cluster]# ccsd
        5.3、启动集群管理器（cman）  
        root@gfs-node1 # /sbin/cman_tool join  
        root@gfs-node2 # /sbin/cman_tool join  
        5.4、测试ccsd服务  
        (注意：ccsd的测试要等cman启动完成后，然后才可以进行下面的测试
        [root@gfs-node1 cluster]# ccs_test connect
        [root@gfs-node2 cluster]# ccs_test connect
        # ccs_test connect 各个节点的返回如下：
        node 1：
        [root@gfs-node1 cluster]# ccs_test connect
        Connect successful.
        Connection descriptor = 0
        node 2：
        [root@gfs-node2 cluster]# ccs_test connect
        Connect successful.
        Connection descriptor = 30
        5.5、查看节点状态
        cat /proc/cluster/nodes,应该返回  
        [root@gfs-node1 cluster]# cat /proc/cluster/nodes
        Node  Votes Exp Sts  Name
          1    1    3   M   gfs-node1
          2    1    3   M   gfs-node2
        [root@gfs-node1 cluster]#
八、加入fence域：  
[root@gfs-node1 cluster]# /sbin/fence_tool join
[root@gfs-node2 cluster]# /sbin/fence_tool join
九、查看集群状态
Node 1:
[root@gfs-node1 cluster]# cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: alpha_cluster
Cluster ID: 50356
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 2   
Active subsystems: 1
Node name: gfs-node1
Node ID: 1
Node addresses: 192.168.10.1
Node 2
[root@gfs-node2 cluster]# cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: alpha_cluster
Cluster ID: 50356
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 2   
Active subsystems: 1
Node name: gfs-node2
Node ID: 2
Node addresses: 192.168.10.2
十、在node-1上进行分区
        #dmesg |grep scsi察看scsi设备，如下：
    [root@gfs-node1 ~]# dmesg | grep scsi
        scsi0 : ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=169
        Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
        [root@gfs-node1 ~]# pvcreate /dev/sdb
                Physical volume "/dev/sdb" successfully created
        [root@gfs-node1 ~]# pvcreate /dev/sdc
                Physical volume "/dev/sdc" successfully created
        [root@gfs-node1 ~]# pvcreate /dev/sdd
                Physical volume "/dev/sdd" successfully created
        [root@gfs-node1 ~]# pvcreate /dev/sde
                Physical volume "/dev/sde" successfully created
        
        [root@gfs-node1 ~]# system-config-lvm
                physical extent size 改为 128k
                sdb -> common -> ohome
                sdc -> oradata -> datafiles
                sdd -> redo1 -> log1
                sde -> redo2 -> log2
[注意] 在pvcreate之前也可以先用fdisk分区
十一、创建GFS文件系统
        [root@gfs-node1 ~]# mkfs.gfs -J 32 -j 4 -p lock_dlm -t alpha_cluster:log1 /dev/redo1/log1
        [root@gfs-node1 ~]# mkfs.gfs -J 32 -j 4 -p lock_dlm -t alpha_cluster:log2 /dev/redo2/log2
        [root@gfs-node1 ~]# mkfs.gfs -J 32 -j 4 -p lock_dlm -t alpha_cluster

home /dev/common/ohome
        [root@gfs-node1 ~]# mkfs.gfs -J 32 -j 4 -p lock_dlm -t alpha_cluster:datafiles /dev/oradata/datafiles
        查看：
        dmesg | grep scsi
        lvscan
        修改 /etc/fstab 文件：在文件末尾添加
        /dev/common/ohome       /dbms/ohome     gfs _netdev 0 0
        /dev/oradata/datafiles  /dbms/oradata   gfs _netdev 0 0
        /dev/redo1/log1         /dbms/log1      gfs _netdev 0 0
        /dev/redo2/log2         /dbms/log2      gfs _netdev 0 0
        The _netdev option is also useful as it insures the filesystems are un-mounted before cluster services shutdown.
        2个节点都要修改 fstab 文件. /dbms 及其子目录都要手工创建好。
十二、创建RAW分区
        The certified version of Oracle 10g on GFS requires that the two clusterware files be located on shared raw partitions and
        be visible by all RAC nodes in the cluster.
        [root@gfs-node1 ~]# fdisk /dev/sdg
        创建2个256M的raw device
        If the other nodes were already up and running while you created these partitions, these other nodes must re-read the partition
        table from disk:
        [root@gfs-node2 ~]# blockdev --rereadpt /dev/sdg
        Make sure the service rawdevices is enabled on all three RAC nodes for the run level that will be used. This example enables
it for both run levels. Run:
rac1 # chkconfig –level 35 rawdevices on
The mapping occurs in the files /etc/sysconfig/rawdevices
# raw device bindings
# format:   
#  
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/sdg1
/dev/raw/raw2 /dev/sdg2
The permissions of these files must always be owned by the oracle user used to install the software (oracle). A 10
second delay is needed to insure that the rawdevices service has a chance to configure the /dev/raw directory. Add
these lines to the /etc/rc.local file. This file is symbolically linked to /etc/rc?.d/S99local.
echo "Sleep a bit first and then set the permissions on raw"
sleep 10
chown oracle:dba /dev/raw/raw1
chown oracle:dba /dev/raw/raw2
十二、修改 /etc/sysctl.conf
kernel.shmmax = 4047483648
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 1024 65000
fs.file-max = 65536
##
This is for Oracle RAC core GCS services
#
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
十三、Create the oracle user
[root@gfs-node1 ~]# groupadd oinstall
[root@gfs-node1 ~]# groupadd dba
[root@gfs-node1 ~]# useradd oracle -g oinstall -G dba
配置 /etc/sudoers 文件，so that oracle admin users can safely execute root commands:
# User alias specification
User_Alias SYSAD=oracle, oinstall
User_Alias USERADM=oracle, oinstall
# User privilege specification
SYSAD ALL=(ALL) ALL
USERADM ALL=(root) NOPASSWD:/usr/local/etc/yanis.client
root ALL=(ALL) ALL
每个节点都要做以上工作。
十四、oracle用户 Create_a_clean_ssh_connection_environment
        1、在每个节点上执行 ssh-keygen –t dsa 按回车直到执行完毕
        2、在node1 collect up all the ~/.ssh/id_dsa.pub files into one ~/.ssh/authorized_keys file and distribute this to the other three nodes:
                ssh gfs-node1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
                ssh gfs-node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
                scp ~/.ssh/authorized_keys gfs-node2:~/.ssh
        3、执行一遍以下命令：
                 [oracle@gfs-node1 ~]$ ssh gfs-node1 date
                 [oracle@gfs-node1 ~]$ ssh node1-prv date
                 [oracle@gfs-node1 ~]$ ssh gfs-node2 date
                 [oracle@gfs-node1 ~]$ ssh node2-prv date
                node2节点同样做一遍
之后就是安装 oracle10g 了，注意选取 cluster 方式安装。

本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u/30029/showart_256751.html