RedHat5.4+ASM+oracle11g RAC安裝

1、IP地址規劃

node1：

public-ip   192.168.2.208     eth0

virtual-ip   192.168.2.210     eth0:1

priv-ip       10.0.1.55              eth1

node2：

public-ip   192.168.2.209     eth0

virtual-ip   192.168.2.211     eth0:1

priv-ip       10.0.1.56              eth1

SCAN-IP:192.168.2.212

2、配置主機IP地址，修改hosts文件，DNS Server配置
-----hosts文件，兩個節點一樣-----
[root@db1]#vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost
::1             localhost6.localdomain6 localhost6
#Public Network
192.168.2.208 db1
192.168.2.209 db2
#Private Interconnect
10.0.1.55 db1-priv
10.0.1.56 db2-priv
#Public Virtual ip
192.168.2.210 db1-vip
192.168.2.211 db2-vip
192.168.2.98 rac-scan.flawless.com   rac-scan


-----網卡，分別配置兩個節點-----
[root@db1]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.2.208
NETMASK=255.255.255.0
GATEWAY=192.168.2.254


[root@db1]# vi /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth1
ONBOOT=yes
IPADDR=10.0.1.55
NETMASK=255.255.255.0


-----DNS服務器配置-----
這裏選用192.168.2.98（linux）作爲DNS服務器，配置方法另見blog:    http://blog.csdn.net/flawless_521/article/details/27558529


3、創建數據庫用戶
[root@db1 etc]# groupadd -g 1000 oinstall
[root@db1 etc]# groupadd -g 1200 asmadmin
[root@db1 etc]# groupadd -g 1201 asmdba
[root@db1 etc]# groupadd -g 1202 asmoper
[root@db1 etc]# useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid
[root@db1 etc]# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper) context=root:system_r:unconfined_t:SystemLow-SystemHigh


[root@db1 etc]# groupadd -g 1300 dba
[root@db1 etc]# groupadd -g 1301 oper
[root@db1 etc]# useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
[root@db1 etc]# id oracle
uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper) context=root:system_r:unconfined_t:SystemLow-SystemHig


[root@db1 etc]passwd oracle
[root@db1 etc]passwd grid


驗證nobody用戶存在：id nobody，如果輸出了用戶的信息，則用戶存在，如果用戶不存在，則使用以下語句創建用戶：
[root@db1 etc]#/usr/sbin/useradd nobody


4、在所有節點創建安裝目錄
[root@db2 ~]# mkdir -p /u01/app/grid
[root@db2 ~]# mkdir -p /u01/app/11.2.0/grid
[root@db2 ~]# chown -R grid:oinstall /u01


[root@db2 ~]# mkdir -p /u01/app/oracle
[root@db2 ~]# chown -R oracle:oinstall /u01/app/oracle
[root@db2 ~]# chmod -R 755 /u01


/u01/app/oracle 是oracle的ORACLE_BASE目錄
/u01/app/grid 是grid的ORACLE_BASE目錄
/u01/app/11.2.0/grid是grid的ORACLE_HOME目錄
說明：grid在安裝的時候ORACLE_HOME不能是ORACLE_BASE的子目錄，所以要新建立一個目錄存放grid的ORACLE_HOME


5、修改oracle和grid的.bash_profile文件
su - grid
vi .bash_profile


export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp




su - oracle
vi .bash_profile


ORACLE_SID=racdb1; export ORACLE_SID
ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
ORACLE_PATH=/u01/app/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp



6、修改系統參數
vi /etc/security/limits.conf
尾部增加以下文件
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536


vi /etc/pam.d/login
尾部增加以下文件
session    required     pam_limits.so


vi /etc/profile
if [ /$USER = "oracle" ] || [ /$USER = "grid" ]; then
    if [ /$SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
    else
        ulimit -u 16384 -n 65536
    fi
    umask 022
fi


vi /etc/sysctl.conf


net.core.rmem_default    = 262144
net.core.rmem_max                = 262144
net.core.wmem_default    = 262144
net.core.wmem_max                = 262144
kernel.shmall            = 131072000
kernel.shmmax            = 536870912
kernel.shmmni            = 4096
kernel.sem               = 250 32000 100 128
fs.file-max                   = 65536
net.ipv4.ip_local_port_range = 1024 65500
fs.aio-max-nr=1048576


sysctl -p 使參數生效

7、同步時間
使用集羣時間同步服務在集羣中提供同步服務，需要卸載網絡時間協議 (NTP) 及其配置。
要停用 NTP 服務，必須停止當前的 ntpd 服務，從初始化序列中禁用該服務，並刪除 ntp.conf 文件。要在 Oracle Enterprise Linux 上完成這些步驟，以 root 用戶身份在兩個 Oracle RAC 節點上運行以下命令：
[root@db1 ~]# /sbin/service ntpd stop
[root@db1 ~]# chkconfig ntpd off
[root@db1 ~]# mv /etc/ntp.conf /etc/ntp.conf.original
 
還要刪除以下文件：
[root@db1 ~]# rm /var/run/ntpd.pid
此文件保存了 NTP 後臺程序的 pid。


8、配置用戶等效性
這裏注意原來10g安裝rac只需要配置oracle用戶等效性就可以了，11g的rac還需要配置grid用戶的等效性，grid用戶配置與oracle用戶配置同理，這裏只給出oracle用戶等效性配置


在db1:
[oracle@db1 ~]$ mkdir ~/.ssh
[oracle@db1 ~]$ chmod 700 ~/.ssh
[oracle@db1 ~]$ ssh-keygen -t rsa
[oracle@db1 ~]$ ssh-keygen -t dsa
 
在db2：
[oracle@db2 ~]$ mkdir ~/.ssh
[oracle@db2 ~]$ chmod 700 ~/.ssh
[oracle@db2 ~]$ ssh-keygen -t rsa
[oracle@db2 ~]$ ssh-keygen -t dsa


切換回db1，接着執行：
[oracle@db1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[oracle@db1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
 
提示：下列命令會提示你輸入bjgtj2 的oracle 密碼，按照提示輸入即可，如果失敗可重新嘗試執行命令。
db1 節點：
[oracle@db1 ~]$ scp ~/.ssh/authorized_keys db2:~/.ssh/authorized_keys
 
db2節點：
[oracle@db2 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[oracle@db2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[oracle@db2 ~]$  scp ~/.ssh/authorized_keys db1:~/.ssh/authorized_keys
 
確保2個node都有相互的結點信息。兩機相互執行。
[oracle@db1 ~]$ ssh db1 date
[oracle@db1~]$ ssh db2 date
[oracle@db1 ~]$ ssh db1-priv date
[oracle@db1 ~]$ ssh db2-priv date
 
切換至db2 執行
[oracle@db2 ~]$ ssh db1 date
[oracle@db2~]$ ssh db2 date
[oracle@db2 ~]$ ssh db1-priv date
[oracle@db2 ~]$ ssh db2-priv date


9、添加共享磁盤，安裝ASM並創建ASM磁盤
這裏用vmware新添加了一塊50GB的共享磁盤，將磁盤分區成sb1 （1GB存放ocr與votedisk）、sdb2（30GB存放數據文件） 、sdb3（19GB存放FRA數據），以下只給出分區sdb1的過程，大小爲1024m，其他的分區同下：
三個磁盤組名字：OCRVOTEDG、DATAFILEDG、FRADG


[root@db1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 7832.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)


Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)


Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-7832, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-7832, default 7832): +1024M  


Command (m for help): p


Disk /dev/sdb: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1          62      497983+  83  Linux


Command (m for help): w
The partition table has been altered!


Calling ioctl() to re-read partition table.
Syncing disks.


---安裝ASM包（兩個節點都執行）
uname -a
Linux db2 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

安裝以下三個包，按順序安裝：
rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm
rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm

安裝完後檢查安裝是否成功：rpm -qa|grep asm


---配置ASM
[root@db1]# oracleasm configure -i（兩個節點都執行）
Configuring the Oracle ASM library driver.


This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.


Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done


---加載oracleasm內核模塊
[root@db1 ASM_RHL-2.6.8-164.el5]# oracleasm init（兩個節點都執行）
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm


---創建asm磁盤
[root@db1]# oracleasm createdisk OCRVOTEDG /dev/sdb1
Writing disk header: done
Instantiating disk: done

[root@db1]# oracleasm createdisk DATAFILEDG /dev/sdb2
Writing disk header: done
Instantiating disk: done

[root@db1]# oracleasm createdisk FRADG /dev/sdb3
Writing disk header: done
Instantiating disk: done




在另一個節點執行掃描
[root@db2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCRVOTEDG"
Instantiating disk "DATAFILEDG"
Instantiating disk "FRADG"


[root@db2 ~]# oracleasm listdisks
DATAFILEDG
FRADG
OCRVOTEDG


10、安裝CLuster軟件
11g中用grid用戶管理cluster，故用grid用戶安裝cluster軟件，安裝grid軟件之前可以先檢查安裝環境，進入到grid的解壓目錄裏，找到runcluvfy.sh，運行以下命令：
./runcluvfy.sh stage -pre crsinst -n db1,db2 -fixup -verbose    如果檢查有failed的則修復，直到沒有failed的進行下一步安裝，依次選擇的順序如下：


install and  Configure Grid infrastructure for a Cluster
Advanced Installation
語言默認用English
將Configure GNS的勾去掉，輸入Cluster Name和SCAN Name，這兩處輸入的名字可以按照你之前配置的/etc/hosts文件最後添加的內容 192.168.2.212 rac-scan.flawless.com   rac-scan
Cluster Name  rac-scan
SCAN Name   rac-scan.flawless.com
點擊add增加節點2的信息，然後點擊SSH Connectivity，輸入密碼後點擊Setup，驗證成功則進入下一步
網卡eth0和eth1這塊直接下一步
存儲方式選擇ASM
輸入Disk Group Name  : OCRVOTEDG，Redundancy選擇External，這裏先給ocr和votedisk選擇ORCL：OCRVOTEDG然後下一步
Use same password for  these account，輸入密碼
Do not use intellgent Platform Management Interface(IPM)
下面一直下一步就可以了，然後只需要等待，最後會彈出需要執行的腳本，分別用root用戶在兩個節點執行腳本就可以了，如果腳本執行中有報錯，那麼需要解決掉，我這裏也遇到了一次錯誤，但是我直接把目錄刪了，修改了一些東西后重新運行了runInstall，安裝完成後，分別在兩個節點切換到grid用戶運行crs_stat -t，驗證安裝情況，同時運行命令crsctl check ctss檢查時間是否同步，正常情況下應輸出以下內容：
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0


11、安裝oracle軟件
切換到oracle用戶下，安裝oracle軟件，只需要在第一個節點安裝，注意一個地方將兩個節點都選上select all，安裝完成後用root用戶按照提示，分別在兩個節點執行腳本


12、創建其他ASM磁盤組
安裝grid的時候就創建了ASM實例，但是隻創建了一個CRS組來安裝ocr和votedisk，現在繼續創建DATAFIELDG和FRADG
用grid用戶直接運行asmca，直接點擊create創建這兩個磁盤組，創建完之後直接點擊Exit退出窗口即可。


13、創建實例
直接在第一個節點運行dbca，與安裝單實例數據庫基本相同。