Oracle RAC是Oracle Real Application Cluster的簡寫，官方中文文檔一般翻譯爲“真正應用集羣”，它一般有兩臺或者兩臺以上同構計算機及共享存儲設備構成，可提供強大的數據庫處理能力，現在是Oracle 10g Grid應用的重要組成部分。RAC（Race Game）指競速類遊戲。--百度。
一些圖片在很久之前就截了下載，最近在虛擬機裏面跑一下測試，現在再整理一下，發佈上來，圖片很多。在整理的過程中發現，自己單獨做一篇都是簡單，但是記錄成文檔，再發布上來真的就不容易了，感覺像一下子做了十次一樣，每個錯誤都想記錄下來，但是又很多細節無法一一詳記。希望大家能知道老夫的不容易啊，呵呵。有一些記錄的不好的話大家多多見諒，互相學習啊。
部分測試環境
硬件：兩個虛擬機內存2g，swap，結果節點一出現死機，兩個節點內存差不多佔滿，建議內存3g+。
系統：rhel5.5,oracle linux5U8,centos6.6，oracle linux 6.7，結果，除了redhat5.x以及oracle linux5 比較好之外，其他版本內核和模塊原因不能繼續。
...
...
網絡規劃
rac1
eth0 10.1.2.140 rac1
10.1.2.141 rac1-vip
eth1 192.168.0.140 rac1-priv
rac2
eth0 10.1.2.142 rac2
10.1.2.143 rac2-vip
eth1 192.168.0.142 rac2-priv
/etc/hosts
#eth0 public
10.1.2.140 rac1.kong.com rac1
10.1.2.142 rac2.kong.com rac2
#virtual
10.1.2.141 rac1-vip.kong.com rac1-vip
10.1.2.143 rac2-vip.kong.com rac2-vip
#eth1 private
192.168.0.140 rac1-priv.kong.com rac1-priv
192.168.0.142 rac2-priv.kong.com rac2-priv
#scan
10.1.2.144 rac-scan.kong.com rac-scan
存儲
10g-->分三分區,i
勾選立即分配，單個文件存儲提高性能
 |
高級修改->scsi通道改爲1：X，獨立永久。
 |
節點2添加現有磁盤 修改和節點1相同配置
修改虛擬機文件“虛擬機名稱.vmx”文件
#shared disks configure
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.maxUnsyncedWrites = "0"
disk.EnableUUID = "TRUE" #否則UUID出不來
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"
scsi1:0.present = "TRUE"
scsi1:0.mode = "independent-persistent"
scsi1:0.fileName = "E:\share\ocr_vote.vmdk"
scsi1:0.deviceType = "disk"
scsi1:0.redo = ""
－－－－－－－－－－－－－
scsi1.virtualDev = "lsilogic"
scsi1.present = "TRUE"
scsi1.sharedBus = "VIRTUAL"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "D:\Program Files (x86)\VMware\Shared Virtual Mchine\ShareStore\vmShareStore.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "disk"
scsi0:1.present = "FALSE"
floppy0.present = "FALSE"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.maxUnsyncedWrites = "0"
disk.EnableUUID = "TRUE"
－－－－－－－－－－－－－
現在rac1添加共享存儲，再手動rac2添加共享存儲，再添加上面的代碼。
依賴關係
使用腳本批量安裝
vi yumget.sh
#!bin/bash
#start install required packages
echo "start installing required packages"
yum -y install binutils-*
yum -y install compat-libstdc++-*
yum -y install compat-libstdc++-*
yum -y install elfutils-libelf-*
yum -y install elfutils-libelf-devel-*
yum -y install gcc-*
yum -y install gcc-c++-*
yum -y install glibc-*
yum -y install glibc-*
yum -y install glibc-common-*
yum -y install glibc-devel-*
yum -y install glibc-devel-*
yum -y install glibc-headers-*
yum -y install ksh-*
yum -y install libaio-*
yum -y install libaio-*
yum -y install libaio-devel-*
yum -y install libaio-devel-*
yum -y install libgcc-*
yum -y install libgcc-*
yum -y install libstdc++-*
yum -y install libstdc++-*
yum -y install libstdc++-devel*
yum -y install make-*
yum -y install sysstat-*
yum -y install unixODBC-*
運行sh yumget.sh
勾選客機時間與主機同步相當於ntp，前提
安裝工作（兩個節點同時進行）
1.修改主機名
vi /etc/sysconfig/network
HOSTNAME=rac1.xl.com
2.創建用戶及相關
用戶規劃
grid：
ORACLE_HOME =/oracle/app/grid_home
ORACLE_BASE=/oracle/app/grid
oracle：
ORACLE_BASE=/oracle/app/oracle
ORACLE_HOME=/oracle/app/oracle/db
用戶創建
/usr/sbin/groupadd -g 505 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/groupadd -g 503 oper
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/groupadd -g 507 asmoper
/usr/sbin/useradd -u 505 -g oinstall -G asmadmin,dba,asmdba,asmoper grid
/usr/sbin/useradd -u 506 -g oinstall -G dba,asmdba,oper oracle
用戶密碼
echo -n 123456|passwd --stdin grid
echo -n 123456|passwd --stdin oracle
本地文件
mkdir -p /oracle/app/grid
chown grid.oinstall /oracle/app/grid
chmod 775 /oracle/app/grid
mkdir -p /oracle/app/grid_home
chown grid.oinstall /oracle/app/grid_home
chmod 775 /oracle/app/grid_home
mkdir -p /oracle/app/oracle
chown oracle.oinstall /oracle/app/oracle
chmod 775 /oracle/app/oracle
mkdir -p /oracle/app/oracle/db
chown oracle.oinstall /oracle/app/oracle/db
chmod 775 /oracle/app/oracle/db
chown grid.oinstall /oracle/app
chmod 775 /oracle/app
3.修改系統，用戶參數
cp /etc/sysctl.conf /etc/sysctl.conf_bak
vi /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 4294967295
kernel.shmall = 268435456
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
sysctl -p
vi /etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
4.配置用戶環境變量
4.1 grid用戶環境變量，節點不同ORACLE_SID->+ASM1+1
ORACLE_BASE=/oracle/app/grid; export ORACLE_BASE
ORACLE_HOME=/oracle/app/grid_home; export ORACLE_HOME
ORACLE_SID=+ASM1; export ORACLE_SID
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin
export PATH
4.2 oracle用戶環境變量，節點不同ORACLE_SID->rac1+1
ORACLE_BASE=/oracle/app/oracle; export ORACLE_BASE
ORACLE_HOME=/oracle/app/oracle/db; export ORACLE_HOME
ORACLE_SID=rac1; export ORACLE_SID
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin
export PATH
5. 用戶等效性
略過，後面會自動完成
6.關閉防火牆senlinux
臨時關閉
service iptables stop;
setenforce 0;
永久關閉
chkconfig iptables off;
vi /etc/selinux/config
SELINUX=disabled
7.安裝ASM驅動
安裝網站“http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html”
如果是redhat5的需要下載三個文件 區分32與64位
•
oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.i686.rpm
•
oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.i686.rpm
•
oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm
•
oracleasm-2.6.18-194.el5-debuginfo-2.0.5-1.el5.i686.rpm
•
oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm
如果是redhat6的只有64位的lib下載（或許可以下載oracle linux6 的32位）
•
oracleasm-support-2.1.8-1.el6.x86_64.rpm
•
oracleasmlib-2.0.4-1.el6.x86_64.rpm
如果是oracle linux6的有32與64位
•
oracleasm-support-2.1.8-1.el6.x86_64.rpm
•
oracleasmlib-2.0.4-1.el6.i686.rpm
如果是oracle linux7（redhat7以上只有64位系統）
•
oracleasm-support-2.1.8-1.el6.x86_64.rpm
•
oracleasmlib-2.0.8-2.el7.x86_64.rpm
[root@rac1 note]# uname -a
Linux rac1.kong.com 2.6.18-194.el5PAE #1 SMP Tue Mar 16 22:00:21 EDT 2010 i686 i686 i386 GNU/Linux
系統是有2.6.18-194.el5PAE 有PAE的所以要注意
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release (如果報驗證問題時執行）
[root@rac1 note]# rpm -ivh oracleasm-support-2.1.8-1.el5.i386.rpm
warning: oracleasm-support-2.1.8-1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]
[root@rac1 note]# rpm -ivh oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm
warning: oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasm-2.6.18-194.el########################################### [100%]
[root@rac1 note]# rpm -ivh oracleasmlib-2.0.4-1.el5.i386.rpm
warning: oracleasmlib-2.0.4-1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]
7.1配置asm
A
[root@rac2 note]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
B
[root@rac2 note]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm
8.分區（在節點1做）
[root@rac1 note]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xadbadc79.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): p
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1958, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1958, default 1958): +3G
Command (m for help): p
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79
Device Boot Start End Blocks Id System
/dev/sdb1 1 393 3156741 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
2
Invalid partition number for type `2'
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (394-1958, default 394):
Using default value 394
Last cylinder, +cylinders or +size{K,M,G} (394-1958, default 1958): +7G
Command (m for help): p
Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79
Device Boot Start End Blocks Id System
/dev/sdb1 1 393 3156741 83 Linux
/dev/sdb2 394 1308 7349737+ 83 Linux
Command (m for help): w
基本是3g，7g兩個分區
8.1節點一創建並發現ASM分區
[root@rac1 note]# oracleasm createdisk OCR_VOTE /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@rac1 note]# oracleasm createdisk DATA /dev/sdb2
Writing disk header: done
Instantiating disk: done
[root@rac1 note]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac1 note]# oracleasm listdisks
DATA
OCR_VOTE
[root@rac1 note]#
節點二隻需要發現ASM分區
[root@rac2 note]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCR_VOTE"
Instantiating disk "DATA"
[root@rac2 note]# oracleasm listdisks
DATA
OCR_VOTE
[root@rac2 note]# sync
9安裝grid（單節點執行）
切換grid用戶
運行runInstall
修改SCAN NAME爲你hosts文件裏面的對應的scan名
添加節點，大家可以看到hostname和virtual ip name在hosts文件是對應啊
其他的能默認的都默認
解決依賴關係
執行yumget.sh之後還有不能安裝的包就可以忽略有些64位不能識別
安裝ing
在65%的時候，rac2上面會自動出現一個grid用戶的ractrans 傳輸進程，如果沒有就檢查iptables和selinux有沒有關閉。
在這裏就用root用戶在兩個節點上面運行這兩個腳本，運行完就點ok。
最後回報一個“[INS-20802] Oracle Cluster Vertificate utility...."的錯，不用管點擊ok即可，點next，完成安裝
安裝完成後使用crs_stat -t去查看資源狀態，發現gsd/oc4j 都是在offline狀態！ 這個是11.2.0.1的默認現象
gsd用來支持oem等交互式工具，oc4j是oracle應用服務的一些組件，如biee
[grid@rac1 ~]$ /oracle/app/grid_home/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.eons ora.eons.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
ping 節點的vip看看能不能ping通
這裏vm可以先打個點做個快照，但是由於vm的共享存儲集羣是用獨立\立即\寫入的，所以不受快照影響，寫快照要想關機。
關機要先關crs
關閉crs
[root@rac1 ~]# /oracle/app/grid_home/bin/crs_stop -all 或
[root@rac1 ~]# /oracle/app/grid_home/bin/crsctl stop crs
開啓crs
[root@rac1 ~]# /oracle/app/grid_home/bin/crs_start -all 或
[root@rac1 ~]# /oracle/app/grid_home/bin/crsctl start crs
oracle rac 默認會開機自啓動，如需維護時可使用以下命令：
關閉：
crsctl stop cluster 停止本節點集羣服務
crsctl stop cluster –all 停止所有節點服務
開啓
crsctl start cluster 開啓本節點集羣服務
crsctl stop cluster –all 開啓所有節點服務
注：以上命令需以 root 用戶執行
9.2 錯誤
運行root.sh報錯，是在紅帽6的一個bug
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /oracle/app/grid_home/crs/install/rootcrs.pl line 443.
卸載配置
[root@rac1 Desktop]# /oracle/app/grid_home/crs/install/roothas.pl -delete -force
2015-08-04 01:23:49: Checking for super user privileges
2015-08-04 01:23:49: User has super user privileges
2015-08-04 01:23:49: Parsing the host name
Using configuration parameter file: /oracle/app/grid_home/crs/install/crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
/oracle/app/grid_home/bin/acfsdriverstate: line 51: /lib/acfstoolsdriver.sh: No such file or directory
/oracle/app/grid_home/bin/acfsdriverstate: line 51: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory
Successfully deconfigured Oracle Restart stack
chmod a+wr /var/tmp/.oracle/npohasd
[root@rac1 install]# /bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1 count=1024
/oracle/app/grid_home/crs/install/roothas.pl -deconfig -force -verbose
ADVM/ACFS is not supported on redhat-release-server-6Server-6.7.0.3.0.1.el6.x86_64
[root@rac1 install]# more /tmp/.linux_release
redhat-release-server-6Server-6.7.0.3.0.1.el6.x86_64
[root@rac1 install]# echo "redhat-release-5Server-5" > /tmp/.linux_release
[root@rac1 install]# more /tmp/.linux_release
redhat-release-5Server-5
[root@rac1 install]# /usr/bin/chattr +i /tmp/.linux_release
[root@rac1 install]#
acfsroot: ACFS-9301: ADVM/ACFS installation can not proceed:
acfsroot: ACFS-9302: No installation files found at /oracle/app/grid_home/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.x86_64-x86_64/bin.
root@rac1 bin]# gedit /oracle/app/grid_home/lib/osds_acfslib.pm
1. [root@rac1 lib]# vi osds_acfslib.pm
2. if ((defined($release)) && # Redhat or OEL if defined
3. (($release =~ /^redhat-release/) || # straight RH
4. ($release =~ /^enterprise-release/) || # Oracle Enterprise Linux
5. ($release =~ /^oraclelinux-release/))) # Oracle Linux
6. {
7.
8. 將上面的代碼片段修改如下:
9. if ((defined($release)) && # Redhat or OEL if defined
10. (($release =~ /^redhat-release/) || # straight RH
11. ($release =~ /^enterprise-release/) || # Oracle Enterprise Linux
12. ($release =~ /^centos-release/) || # CentOS hack
13. ($release =~ /^oraclelinux-release/))) # Oracle Linux
14. {
Oracle 11g R2+RAC+ASM+redhat安裝詳解2
卸載Oracle 11g R2 RAC