在VMware Workstation中一步一步搭建Oracle 21c rac集羣環境

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html

一、rac安裝規劃

官網安裝過程：https://docs.oracle.com/en/database/oracle/oracle-database/21/rilin/index.html

1.1、所需軟件介紹

Oracle RAC不支持異構平臺。在同一個集羣中，可以支持具有速度和規模不同的機器，但所有節點必須運行在相同的操作系統。Oracle RAC不支持具有不同的芯片架構的機器。

序號	類型	內容	MD5
1	數據庫	LINUX.X64_213000_db_home.zip	8ac915a800800ddf16a382506d3953db
2	集羣軟件	LINUX.X64_213000_grid_home.zip	b3fbdb7621ad82cbd4f40943effdd1be
3	操作系統	CentOS-7.8-x86_64-DVD-2003.iso	16673979023254EA09CC0B57853A7BBD
4	虛擬機軟件	VMware Workstation Pro 16.0.0 build-16894299
5	Xmanager Enterprise 4	Xmanager Enterprise 4，用於打開圖形界面
6	SecureCRTPortable.exe	Version 6.6.1 (build 289) ，帶有SecureCRT和SecureFX，用於SSH連接和FTP上傳

備註：

1、OS內存至少8G

2、安裝OS和數據庫前需要對下載好的安裝包進行MD5值校驗，檢查其完整性

• 21c數據庫軟件下載可以參考：https://www.xmmup.com/dbbao76zaidockerzhongzhixu2bujikeyongyouoracle-21chuanjing.html

1.2、IP地址規劃

從Oracle 11g開始，一共至少7個IP地址，2塊網卡，其中public、vip和scan都在同一個網段，private在另一個網段，主機名不要包含下橫線，如：RAC_01是不允許的；通過執行ifconfig -a檢查2個節點的網卡名稱必須一致。另外，在安裝之前，公網、私網共4個IP可以ping通，其它3個不能ping通才是正常的。

從18c開始，scan建議至少3個。

節點/主機名	IP Address	Interface Name	地址類型	註冊位置	虛擬網卡適配器	操作系統網卡
raclhr-21c-n1	192.168.59.62	raclhr-21c-n1	Public	/etc/hosts	VMnet8（nat模式）	ens33
raclhr-21c-n1	192.168.59.64	raclhr-21c-n1-vip	Virtual	/etc/hosts	VMnet8（nat模式）	ens33
raclhr-21c-n1	192.168.2.62	raclhr-21c-n1-priv	Private	/etc/hosts	VMnet2（僅主機模式）	ens34
raclhr-21c-n2	192.168.59.63	raclhr-21c-n2	Public	/etc/hosts	VMnet8（nat模式）	ens33
raclhr-21c-n2	192.168.59.65	raclhr-21c-n2-vip	Virtual	/etc/hosts	VMnet8（nat模式）	ens33
raclhr-21c-n2	192.168.2.63	raclhr-21c-n2-priv	Private	/etc/hosts	VMnet2（僅主機模式）	ens34
	192.168.59.66 192.168.59.67 192.168.59.68	raclhr-21c-scan	SCAN	/etc/hosts	VMnet8（nat模式）	ens33

1.3、操作系統本地磁盤分區規劃

除了/boot分區外，其它分區均採用邏輯卷的方式，這樣有利於文件系統的擴展。

序號	分區名稱	大小	邏輯卷	用途說明
1	/u01	50G	/dev/mapper/vg_orasoft-lv_orasoft_u01	oracle和grid的安裝目錄
2	/soft	20G	/dev/mapper/VG_OS-lv_VG_OS_soft	存儲軟件，至少10G

1.4、共享存儲與ASM磁盤組規劃

序號	ASM磁盤名稱	磁盤組名稱	冗餘方式	大小	用途	備註
1	/dev/asm-diskd /dev/asm-diske /dev/asm-diskf	OCR	Normal	3G	OCR+VOTINGDISK	最少1G
2	/dev/asm-diskg /dev/asm-diskh /dev/asm-diski	MGMT	External	30G	MGMT	最少30G，可以選擇不安裝GIMR組件
3	/dev/asm-diskj /dev/asm-diskk /dev/asm-disl	DATA	External	45G	存儲數據庫數據文件
4	/dev/asm-diskm /dev/asm-diskn /dev/asm-disko	FRA	External	30G	快速恢復區

二、操作系統配置

若無特殊說明，那麼以下操作在2個節點均需要執行。

2.1、準備OS

安裝步驟略，也可以下載小麥苗已經安裝好的虛擬機環境：https://www.xmmup.com/ziliaofenxiang.html

安裝一臺虛擬機，然後複製改名，如下：

用虛擬機軟件打開：

2.2、添加網卡

首先，需要添加網卡適配器：

VMnet2如下：

VMnet8如下：

網關爲：192.168.59.2，如下所示：

先刪除再添加，添加2塊網卡，VMnet8爲公有網卡，VMnet2爲私有網卡，如下所示：

打開後，選擇我已複製該虛擬機。

2.3、修改主機名

修改2個節點的主機名爲raclhr-21c-n1和raclhr-21c-n2：

hostnamectl set-hostname raclhr-21c-n1
hostnamectl set-hostname raclhr-21c-n2

2.4、配置靜態IP地址

在2個節點上分別配置靜態IP地址。

在節點2上配置IP的時候注意將IP地址（IPADDR）修改掉。需要確保2個節點上的網卡MAC地址不一樣，否則節點間不能通信。

2.4.1、修改mac地址

使用ifconfig或ip link show查看mac地址，修改文件/etc/udev/rules.d/70-persistent-ipoib.rules

cat > /etc/udev/rules.d/70-persistent-ipoib.rules <<"EOF"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:25:83:db", ATTR{type}=="1", KERNEL=="eth*", NAME="ens33"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:25:83:e5", ATTR{type}=="1", KERNEL=="eth*", NAME="ens34"
EOF

2.4.2、配置靜態IP地址

注意修改IPADDR和HWADDR地址。

1、配置公網：/etc/sysconfig/network-scripts/ifcfg-ens33

cat > /etc/sysconfig/network-scripts/ifcfg-ens33 <<"EOF"
DEVICE=ens33
NAME=ens33
IPADDR=192.168.59.62
NETMASK=255.255.255.0
GATEWAY=192.168.59.2
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:25:83:db
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF

2、配置私網：/etc/sysconfig/network-scripts/ifcfg-ens34 。注意第二塊網卡不能配置網關，否則系統默認網關就變成了第二塊網卡的網關，從而導致系統不能上外網。

cat > /etc/sysconfig/network-scripts/ifcfg-ens34 <<"EOF"
DEVICE=ens34
NAME=ens34
IPADDR=192.168.2.62
NETMASK=255.255.255.0
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:25:83:e5
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF

該部分內容做完後，可以重啓一次OS。

2.5、關閉防火牆

systemctl disable firewalld
systemctl stop firewalld
systemctl status firewalld

systemctl list-unit-files | grep fire

2.6、禁用selinux

手工修改/etc/selinux/config的SELINUX=disabled，或使用下面命令：

sed -i '/^SELINUX=.*/ s//SELINUX=disabled/' /etc/selinux/config

重啓才能生效，校驗：

[root@raclhr-21c-n1 ~]# getenforce 
Disabled

2.7、修改/etc/hosts文件

增加如下的內容：

#Public IP
192.168.59.62            raclhr-21c-n1
192.168.59.63            raclhr-21c-n2

#Private IP
192.168.2.62             raclhr-21c-n1-priv
192.168.2.63             raclhr-21c-n2-priv

#Virtual IP
192.168.59.64            raclhr-21c-n1-vip
192.168.59.65            raclhr-21c-n2-vip

#Scan IP
192.168.59.66            raclhr-21c-scan
192.168.59.67            raclhr-21c-scan
192.168.59.68            raclhr-21c-scan

注意：需要保留127.0.0.1 localhost這行。

2.8、添加組和用戶

groupadd -g 54321 oinstall  
groupadd -g 54322 dba  
groupadd -g 54323 oper  
groupadd -g 54324 backupdba  
groupadd -g 54325 dgdba  
groupadd -g 54326 kmdba  
groupadd -g 54327 asmdba  
groupadd -g 54328 asmoper  
groupadd -g 54329 asmadmin  
groupadd -g 54330 racdba
  
useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,oper oracle  
useradd -u 54322 -g oinstall -G asmadmin,asmdba,asmoper,dba,racdba grid  

echo lhr | passwd --stdin oracle
echo lhr | passwd --stdin grid

2.9、創建安裝目錄

2.9.1、掛載/u01目錄

首先，在2個節點都各自添加一塊200g大小的磁盤：

然後做卷組，200g磁盤大約分10個PE，每個PE大約20g：

內容太多，放不下，部分內容有所刪減，全文可以參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html

分區完成後，開始創建邏輯卷，掛載分區等操作：

pvcreate /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc5 /dev/sdc6 /dev/sdc7 /dev/sdc8 /dev/sdc9 /dev/sdc10 /dev/sdc11
vgcreate vg_oracle /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc5 /dev/sdc6 /dev/sdc7 /dev/sdc8 /dev/sdc9 /dev/sdc10 /dev/sdc11
lvcreate -n lv_orasoft_u01 -L 60G vg_oracle
mkfs.ext4 /dev/vg_oracle/lv_orasoft_u01
mkdir /u01
mount /dev/vg_oracle/lv_orasoft_u01 /u01

echo "/dev/vg_oracle/lv_orasoft_u01 /u01  ext4 defaults 0 0" >> /etc/fstab

查詢：

[root@raclhr-21c-n1 ~]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                              3.9G     0  3.9G   0% /dev
tmpfs                                 3.9G     0  3.9G   0% /dev/shm
tmpfs                                 3.9G   13M  3.9G   1% /run
tmpfs                                 3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/centos_lhrdocker-root      50G  5.0G   42G  11% /
/dev/sda1                             976M  143M  767M  16% /boot
/dev/mapper/centos_lhrdocker-home     9.8G   41M  9.2G   1% /home
/dev/mapper/vg_docker-lv_docker       788G   73M  748G   1% /var/lib/docker
tmpfs                                 797M   12K  797M   1% /run/user/42
tmpfs                                 797M     0  797M   0% /run/user/0
/dev/mapper/vg_oracle-lv_orasoft_u01   59G   53M   56G   1% /u01

2.9.2、創建目錄

mkdir -p /u01/app/21.3.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
mkdir -p /u01/app/oracle/product/21.3.0/dbhome_1
chown -R grid:oinstall /u01
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

2.10、配置grid和oracle用戶的環境變量文件

oracle用戶：

cat >> /home/oracle/.bash_profile <<"EOF"

umask 022
export ORACLE_SID=rac21c1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/21.3.0/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export TMP=/tmp
export TMPDIR=$TMP
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH


export EDITOR=vi
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORACLE_PATH=.:$ORACLE_BASE/dba_scripts/sql:$ORACLE_HOME/rdbms/admin
export SQLPATH=$ORACLE_HOME/sqlplus/admin


#export NLS_LANG="SIMPLIFIED CHINESE_CHINA.ZHS16GBK" --AL32UTF8 SELECT userenv('LANGUAGE') db_NLS_LANG FROM DUAL;
export NLS_LANG="AMERICAN_CHINA.ZHS16GBK"

alias sqlplus='rlwrap sqlplus'
alias rman='rlwrap rman'
alias asmcmd='rlwrap asmcmd'
alias dgmgrl='rlwrap dgmgrl'
alias sas='sqlplus / as sysdba'

EOF



cat >> /home/grid/.bash_profile <<"EOF"

umask 022
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/21.3.0/grid
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export PATH=$ORACLE_HOME/bin:$PATH
alias sqlplus='rlwrap sqlplus' 
alias asmcmd='rlwrap asmcmd'
alias dgmgrl='rlwrap dgmgrl'
alias sas='sqlplus / as sysdba'

EOF

注意：另外一臺數據庫實例名須做相應修改：

Oracle：export ORACLE_SID=rac21c2
grid：export ORACLE_SID=+ASM2

2.11、配置root用戶的環境變量

cat >> /etc/profile <<"EOF"

export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/21.3.0/grid
export GRID_BASE=$ORACLE_BASE
export GRID_HOME=$ORACLE_HOME
export PATH=$PATH:$ORACLE_HOME/bin

EOF

2.12、安裝軟件依賴包

安裝一些常用的包：

yum install -y openssh-clients openssh-server initscripts  net-tools telnet which wget \
    passwd e4fsprogs lrzsz sudo unzip lvm2 tree traceroute lsof file tar systemd \
    bridge-utils mlocate mailx strace less mmv stress
yum install -y dos2unix rlwrap xdpyinfo xorg-x11-apps nmap numactl numactl-devel \
    iproute rsyslog bash-completion tmux sysbench vim redhat-lsb smartmontools xinetd \
    gcc  make  sysstat ksh binutils socat cmake automake autoconf bzr bison libtool deltarpm \
    rsync libev pv subversion nload gnuplot jq oniguruma yum-fastestmirror net-snmp net-snmp-utils \
    nfs-utils rpcbind  postfix dovecot bind-utils bind bind-chroot dnsmasq haproxy keepalived bzr \
    fio bzip2 ntp flex lftp
yum install -y ncurses-devel libgcrypt-devel libaio libaio-devel \
    perl perl-Env perl-devel  perl-Time-HiRes perl-DBD-MySQL perl-ExtUtils-MakeMaker perl-TermReadKey \
    perl-Config-Tiny perl-Email-Date-Format perl-Log-Dispatch perl-Mail-Sender perl-Mail-Sendmail \
    perl-MIME-Lite perl-Parallel-ForkManager perl-Digest-MD5 perl-ExtUtils-CBuilder perl-IO-Socket-SSL \
    perl-JSON openssl-devel libverto-devel libsepol-devel libselinux-devel libkadm5 keyutils-libs-devel \
    krb5-devel libcom_err-devel cyrus-sasl* perl-DBD-Pg perf slang perl-DBI  perl-CPAN \
    perl-ExtUtils-eBuilder cpan perl-tests
yum install -y  compat-libstdc++-33 gcc-c++  glibc glibc-common glibc.i686  glibc-devel glibc-devel.i686 \
    libgcc libgcc.i686 libstdc++   libstdc++-devel  libaio.i686   libaio-devel.i686 \
    libXext  libXext.i686  libXtst  libXtst.i686  libX11  libX11.i686  libXau  libXau.i686  \
    libxcb  libxcb.i686  libXi  libXi.i686   unixODBC  unixODBC-devel  zlib-devel  zlib-devel.i686 \
    compat-libcap1 libXp libXp-devel libXp.i686 elfutils-libelf elfutils-libelf-devel compat-db \
    gnome-libs pdksh xscreensaver fontconfig-devel libXrender-devel

 yum remove PackageKit -y

檢查：

rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libcap1 \
compat-libstdc++-33 \
gcc \
gcc-c++ \
glibc \
glibc-devel \
ksh \
libgcc \
libstdc++ \
libstdc++-devel \
libaio \
libaio-devel \
libXext \
libXtst \
libX11 \
libXau \
libxcb \
libXi \
make \
elfutils-libelf-devel \
sysstat  | grep "not installed"

2.13、配置內核參數

2.13.1、修改/etc/sysctl.conf文件

cat >> /etc/sysctl.conf <<"EOF"
vm.swappiness = 1
vm.dirty_background_ratio = 3
vm.dirty_ratio = 80
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
kernel.shmmni = 4096  
kernel.shmall = 1073741824  
kernel.shmmax = 4398046511104
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500 
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576 
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.panic_on_oops = 1
kernel.watchdog_thresh=30
EOF

生效：

/sbin/sysctl -p

2.13.2、修改/etc/security/limits.conf文件

echo "grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
grid soft nproc 2047
grid hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
oracle soft nproc 2047
oracle hard nproc 16384
oracle hard memlock  8145728
oracle soft memlock  8145728
root soft nproc 2047 " >> /etc/security/limits.conf

2.13.3、修改/etc/pam.d/login文件

echo "session    required     pam_limits.so" >> /etc/pam.d/login

2.13.4、修改/etc/profile文件

cat >> /etc/profile << "EOF"
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
              ulimit -p 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
        umask 022
fi
EOF

2.14、關閉NTP和chrony服務

可以採用操作系統的NTP服務，也可以使用Oracle自帶的服務ctss，如果ntp沒有啓用，那麼Oracle會自動啓用自己的ctssd進程。
從oracle 11gR2 RAC開始使用Cluster Time Synchronization Service(CTSS)同步各節點的時間,當安裝程序發現NTP協議處於非活動狀態時，安裝集羣時間同步服務將以活動模式（active）自動進行安裝並同步所有節點的時間。如果發現配置了NTP，則以觀察者模式（observer mode）啓動集羣時間同步服務，Oracle Clusterware不會在集羣中進行活動的時間同步。

systemctl stop ntpd 
systemctl disable ntpd.service
mv /etc/ntp.conf /etc/ntp.conf.bak

Chrony是一個開源的自由軟件，它能幫助你保持系統時鐘與時鐘服務器（NTP）同步，因此讓你的時間保持精確。它由兩個程序組成，分別是chronyd和chronyc。chronyd是一個後臺運行的守護進程，用於調整內核中運行的系統時鐘和時鐘服務器同步。它確定計算機增減時間的比率，並對此進行補償。chronyc提供了一個用戶界面，用於監控性能並進行多樣化的配置。它可以在chronyd實例控制的計算機上工作，也可以在一臺不同的遠程計算機上工作。

systemctl disable chronyd
systemctl stop chronyd
mv /etc/chrony.conf /etc/chrony.conf_bak

2.15、關閉avahi-daemon

systemctl stop avahi-daemon 
systemctl disable avahi-daemon 

Avahi允許程序在不需要進行手動網絡配置的情況 下，在一個本地網絡中發佈和獲知各種服務和主機。例如，當某用戶把他的計算機接入到某個局域網時，如果他的機器運行有Avahi服務，則Avahi程式自動廣播，從而發現網絡中可用的打印機、共享文件和可相互聊天的其他用戶。這有點象他正在接收局域網中的各種網絡廣告一樣。

Linux下系統實際啓動的進程名，是avahi-daemon。

2.16、將NOZEROCONF=yes添加到/etc/sysconfig/network文件中

echo 'NOZEROCONF=yes' >> /etc/sysconfig/network

2.17、禁用透明大頁

參考：https://www.xmmup.com/linux-biaozhundayehetoumingdaye.html

cat >> /etc/rc.local <<"EOF"
 
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
 
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
 
EOF

chmod +x /etc/rc.d/rc.local
sh /etc/rc.local

cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/enabled

建議配置大頁功能，參考：https://www.xmmup.com/oracleshujukupeizhidaye.html

2.18、停止不用的服務

systemctl list-unit-files | grep enable

systemctl stop autofs
systemctl stop nfslock
systemctl stop rpcidmapd
systemctl stop rpcgssd
systemctl stop ntpd
systemctl stop bluetooth.service
systemctl stop cups.path 
systemctl stop cups.socket
systemctl stop postfix.service
systemctl stop rpcbind.service
systemctl stop rpcbind.socket
systemctl stop NetworkManager-dispatcher.service
systemctl stop dbus-org.freedesktop.NetworkManager.service
systemctl stop abrt-ccpp.service
systemctl stop cups.service
systemctl stop libvirtd

systemctl disable autofs
systemctl disable nfslock
systemctl disable rpcidmapd
systemctl disable rpcgssd
systemctl disable ntpd
systemctl disable bluetooth.service
systemctl disable cups.path 
systemctl disable cups.socket
systemctl disable postfix.service
systemctl disable rpcbind.service
systemctl disable rpcbind.socket
systemctl disable NetworkManager-dispatcher.service
systemctl disable dbus-org.freedesktop.NetworkManager.service
systemctl disable abrt-ccpp.service
systemctl disable cups.service
systemctl disable  libvirtd

2.19、提升開關機速度

在centos在關機或開機時一直等待，等待時間久，卡住時間久，提示A stop job is running for ……

解決：

cat >> /etc/systemd/system.conf << "EOF"
DefaultTimeoutStartSec=5s
DefaultTimeoutStopSec=5s
EOF

systemctl daemon-reload

三、配置共享存儲

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html/3

3.1、服務端配置

3.1.1、創建LVM

3.1.2、使用yum安裝targetd和targetcli

3.1.3、使用targetcli創建設備

3.1.4、使用targetcli創建iqn和LUN

3.1.5、使用targetcli創建acls

3.1.6、配置target監聽IP和端口

3.1.7、保存配置

3.2、客戶端配置

3.2.1、安裝服務並啓動

3.2.2、使用iscsiadm發現可用存儲設備並登陸連接

3.2.3、創建並配置udev rules文件

四、數據庫軟件包準備

4.1、上傳安裝軟件

打開SecureFX軟件：

複製粘貼數據庫文件到/soft目錄下並等待上傳完成。

注意，對安裝包需要進行MD5值校驗：

[root@raclhr-21c-n1 ~]# cd /soft
[root@raclhr-21c-n1 soft]# ll
total 5401812
-rw-r--r-- 1 root root 3109225519 Aug 16 09:58 LINUX.X64_213000_db_home.zip
-rw-r--r-- 1 root root 2422217613 Aug 16 09:57 LINUX.X64_213000_grid_home.zip
[root@raclhr-21c-n1 soft]# md5sum LINUX.X64_213000_grid_home.zip 
b3fbdb7621ad82cbd4f40943effdd1be  LINUX.X64_213000_grid_home.zip
[root@raclhr-21c-n1 soft]# md5sum LINUX.X64_213000_db_home.zip 
8ac915a800800ddf16a382506d3953db  LINUX.X64_213000_db_home.zip

4.2、解壓軟件

grid用戶：

unzip LINUX.X64_213000_grid_home.zip  -d /u01/app/21.3.0/grid

oracle用戶：

unzip LINUX.X64_213000_db_home.zip -d /u01/app/oracle/product/21.3.0/dbhome_1

只在節點1上進行解壓即可。

👉 注意：

1、不要同時進行解壓2個文件，否則可能有未預料到的錯誤

2、需要搜索一下解壓是否有報錯的地方，搜索關鍵詞“error”

注意：18c解壓後的文件所在目錄就是grid home。所以解壓的時候，就要把文件解壓到之前定的GRID_HOME下。在12c R2之前是安裝的時候，軟件會自動複製過去。

Starting with Oracle Database 18c, the Oracle Database software is available as an image file for download and installation. Extract the image software into the directory where you want your Oracle home to be located, and then run the runInstaller script to start the Oracle Database installation.

Starting with Oracle Database 18c, installation and configuration of Oracle Database software is simplified with Image-Based installation.

五、安裝前預檢查

5.1、安裝補丁包(cvuqdisk)

在安裝RAC之前，經常會需要運行集羣驗證工具CVU（Cluster Verification Utility），該工具執行系統檢查，確認當前的配置是否滿足要求。

首先判斷是否安裝了cvuqdisk包：

rpm -qa cvuqdisk

如果沒有安裝，那麼在2個節點上都執行如下命令進行安裝該包：

export CVUQDISK_GRP=oinstall
cd /u01/app/21.3.0/grid/cv/rpm
rpm -ivh  cvuqdisk-1.0.10-1.rpm

傳輸到第2個節點上進行安裝：

scp cvuqdisk-1.0.10-1.rpm root@raclhr-21c-n2:/soft

-- 節點2安裝cvuqdisk包
export CVUQDISK_GRP=oinstall
rpm -ivh  /soft/cvuqdisk-1.0.10-1.rpm

5.2、配SSH互信，建立ssh等效性

sshUserSetup.sh在GI安裝介質解壓縮後的sshsetup目錄下。下面兩條命令在節點1上執行即可，在root用戶下執行：

/u01/app/21.3.0/grid/oui/prov/resources/scripts/sshUserSetup.sh -user grid  -hosts "raclhr-21c-n1 raclhr-21c-n2" -advanced exverify -confirm
/u01/app/21.3.0/grid/oui/prov/resources/scripts/sshUserSetup.sh -user oracle  -hosts "raclhr-21c-n1 raclhr-21c-n2" -advanced exverify -confirm

輸入yes及密碼一路回車即可。

以oracle和grid用戶在2個節點上都測試兩節點連通性：

ssh raclhr-21c-n1 date
ssh raclhr-21c-n2 date
ssh raclhr-21c-n1-priv date
ssh raclhr-21c-n2-priv date

第二次執行時不再提示輸入口令，並且可以成功執行命令，則表示SSH對等性配置成功。

5.3、cluster硬件檢測--安裝前預檢查配置信息

Use Cluster Verification Utility (cvu)

Before installing Oracle Clusterware, use CVU to ensure that your cluster is prepared for an installation:

Oracle provides CVU to perform system checks in preparation for an installation, patch updates, or other system changes. In addition, CVU can generate fixup scripts that can change many kernel parameters to at lease the minimum settings required for a successful installation.

Using CVU can help system administrators, storage administrators, and DBA to ensure that everyone has completed the system configuration and preinstallation steps.

./runcluvfy.sh -help

./runcluvfy.sh stage -pre crsinst -n rac1,rac2 –fixup -verbose

Install the operating system package cvuqdisk to both Oracle RAC nodes. Without cvuqdisk, Cluster Verification Utility cannot discover shared disks, and you will receive the error message "Package cvuqdisk not installed" when the Cluster Verification Utility is run (either manually or at the end of the Oracle grid infrastructure installation). Use the cvuqdisk RPM for your hardware architecture (for example, x86_64 or i386). The cvuqdisk RPM can be found on the Oracle grid infrastructure installation media in the rpm directory. For the purpose of this article, the Oracle grid infrastructure media was extracted to the /home/grid/software/oracle/grid directory on racnode1 as the grid user.

在安裝GRID之前，建議先利用CVU(Cluster Verification Utility)檢查CRS的安裝前環境。以grid用戶運行：

su - grid
export CVUQDISK_GRP=oinstall
export CV_NODE_ALL=raclhr-21c-n1,raclhr-21c-n2
/u01/app/21.3.0/grid/runcluvfy.sh stage -pre crsinst -allnodes -fixup -verbose -method root

grid安裝完成後還可以進行如下校驗：

$ORACLE_HOME/bin/cluvfy stage  -pre crsinst -n all  -verbose -fixup

未檢測通過的顯示爲failed，有的failed可以根據提供的腳本進行修復。有的需要根據情況進行修復，有的failed也可以忽略。

• 報錯一：

/dev/shm mounted as temporary file system ...FAILED
raclhr-21c-n2: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
raclhr-21c-n1: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

參考：http://blog.itpub.net/26736162/viewspace-2214381

CentOS7和RHEL7在 /etc/fstab中不包含/dev/shm ，可以手動加進去，或者忽略都可以。

• 報錯二：

Systemd login manager IPC parameter ...FAILED
raclhr-21c-n2: PRVE-10233 : Systemd login manager parameter 'RemoveIPC' entry
               does not exist or is commented out in the configuration file
               "/etc/systemd/logind.conf" on node "raclhr-21c-n2".
               [Expected="no"]

raclhr-21c-n1: PRVE-10233 : Systemd login manager parameter 'RemoveIPC' entry
               does not exist or is commented out in the configuration file
               "/etc/systemd/logind.conf" on node "raclhr-21c-n1".
               [Expected="no"]

參考：http://blog.itpub.net/29371470/viewspace-2125673/

解決：

echo "RemoveIPC=no" >> /etc/systemd/logind.conf
systemctl daemon-reload
systemctl restart systemd-logind

• 報錯三：

Network Time Protocol (NTP) ...FAILED
raclhr-21c-n2: PRVG-1017 : NTP configuration file "/etc/ntp.conf" is present on
               nodes "raclhr-21c-n2,raclhr-21c-n1" on which NTP daemon or
               service was not running

raclhr-21c-n1: PRVG-1017 : NTP configuration file "/etc/ntp.conf" is present on
               nodes "raclhr-21c-n2,raclhr-21c-n1" on which NTP daemon or
               service was not running

我們使用ctssd來同步集羣的時間，所以ntp必須關閉。

六、圖形界面安裝集羣和db

安裝之前重啓一次OS，並檢查網絡和共享盤是否正確。

首先，打開Xmanager - Passive，如下：

6.1、安裝grid

注意：以grid用戶登錄，然後運行下面這個腳本，和之前版本的grid安裝有所不同：

[grid@raclhr-21c-n1 ~]$ export DISPLAY=192.168.59.1:0.0
[grid@raclhr-21c-n1 ~]$ /u01/app/21.3.0/grid/gridSetup.sh
Launching Oracle Grid Infrastructure Setup Wizard...

點擊add添加節點2後，點擊next

默認選中了不安裝GIMR，我們也暫時不安裝MIMR庫：

修改磁盤路徑爲“/dev/asm*”就可以找到ASM磁盤了。

忽悠所有錯誤繼續安裝

分別在節點1和節點2上執行：

/u01/app/oraInventory/orainstRoot.sh
/u01/app/21.3.0/grid/root.sh

節點1運行：

[root@raclhr-21c-n1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@raclhr-21c-n1 ~]# /u01/app/21.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/21.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/21.3.0/grid/crs/install/crsconfig_params
2021-08-20 08:38:46: Got permissions of file /u01/app/grid/crsdata/raclhr-21c-n1/crsconfig: 0775
2021-08-20 08:38:46: Got permissions of file /u01/app/grid/crsdata: 0775
2021-08-20 08:38:46: Got permissions of file /u01/app/grid/crsdata/raclhr-21c-n1: 0775
The log of current session can be found at:
  /u01/app/grid/crsdata/raclhr-21c-n1/crsconfig/rootcrs_raclhr-21c-n1_2021-08-20_08-38-46AM.log
2021/08/20 08:38:56 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/20 08:38:56 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/20 08:38:56 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/20 08:38:58 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/20 08:39:00 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
Redirecting to /bin/systemctl restart rsyslog.service
2021/08/20 08:39:00 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/20 08:39:00 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/20 08:39:13 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/20 08:39:17 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/20 08:39:35 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/20 08:39:35 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/20 08:39:40 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/20 08:39:41 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/08/20 08:39:58 CLSRSC-4002: Successfully installed Oracle Autonomous Health Framework (AHF).
2021/08/20 08:40:14 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/20 08:40:14 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/20 08:40:19 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/20 08:40:24 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/20 08:41:52 CLSRSC-482: Running command: '/u01/app/21.3.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk e4f4713a678d4f04bf0cce6c0c14692d.
Successful addition of voting disk 7f85f487c7e34f42bff66f6c7cfc6032.
Successful addition of voting disk 408b301b47c84f97bfc109558258b289.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   e4f4713a678d4f04bf0cce6c0c14692d (/dev/asm-diskd) [OCR]
 2. ONLINE   7f85f487c7e34f42bff66f6c7cfc6032 (/dev/asm-diske) [OCR]
 3. ONLINE   408b301b47c84f97bfc109558258b289 (/dev/asm-diskf) [OCR]
Located 3 voting disk(s).
2021/08/20 08:43:30 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/08/20 08:44:43 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/08/20 08:44:43 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/08/20 08:47:14 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/08/20 08:47:36 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

最後輸出Configure Oracle Grid Infrastructure for a Cluster ... succeeded表示成功，跑完後，檢查一下：

[root@raclhr-21c-n1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.chad
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.net1.network
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.ons
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.OCR.dg(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.cdp1.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp2.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp3.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cvu
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.qosmserver
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.raclhr-21c-n1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
--------------------------------------------------------------------------------

節點2：

[root@raclhr-21c-n2 ~]# /u01/app/21.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/21.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/21.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/raclhr-21c-n2/crsconfig/rootcrs_raclhr-21c-n2_2021-08-20_08-51-42AM.log
2021/08/20 08:51:48 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/20 08:51:48 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/20 08:51:48 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/20 08:51:50 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/20 08:51:50 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2021/08/20 08:51:50 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/20 08:51:51 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/20 08:51:52 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/20 08:51:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/20 08:51:52 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/20 08:52:23 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/20 08:52:24 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/20 08:52:25 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/08/20 08:52:55 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/20 08:52:55 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/20 08:52:57 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/20 08:52:58 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/20 08:53:04 CLSRSC-4002: Successfully installed Oracle Autonomous Health Framework (AHF).
2021/08/20 08:53:08 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/08/20 08:54:09 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/08/20 08:54:09 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/08/20 08:54:28 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/08/20 08:54:38 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

最後輸出Configure Oracle Grid Infrastructure for a Cluster ... succeeded表示成功，查看：

[root@raclhr-21c-n1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.chad
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.net1.network
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.ons
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.OCR.dg(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.cdp1.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp2.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp3.cdp
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.cvu
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.qosmserver
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.raclhr-21c-n1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.raclhr-21c-n2.vip
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
--------------------------------------------------------------------------------

此時回到界面點擊OK，繼續運行：

最後集羣校驗報錯，看了一下是dns和ntp的問題，可以忽略，繼續安裝：

6.2、安裝db

[root@raclhr-21c-n1 ~]# su - oracle
Last login: Thu Aug 19 15:26:38 CST 2021 on pts/0
[oracle@raclhr-21c-n1 ~]$ cd $ORACLE_HOME
[oracle@raclhr-21c-n1 dbhome_1]$ export DISPLAY=192.168.59.1:0.0
[oracle@raclhr-21c-n1 dbhome_1]$ ./runInstaller 

節點1和節點2分別執行：

[root@raclhr-21c-n1 ~]# /u01/app/oracle/product/21.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/21.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

執行完後，點擊OK：

6.3、創建磁盤組

以 grid 用戶執行 asmca 命令，創建DATA和FRA兩個磁盤組。：

[root@raclhr-21c-n1 ~]# su - grid
Last login: Fri Aug 20 09:59:53 CST 2021
[grid@raclhr-21c-n1 ~]$ export DISPLAY=192.168.59.1:0.0
[grid@raclhr-21c-n1 ~]$ asmca

查看：

[grid@raclhr-21c-n1 ~]$ $ORACLE_HOME/bin/kfod disks=all st=true ds=true
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                    Disk Group    User     Group   
================================================================================
   1:       1024 MB MEMBER    /dev/asm-diskd                           OCR          grid     asmadmin
   2:       1024 MB MEMBER    /dev/asm-diske                           OCR          grid     asmadmin
   3:       1024 MB MEMBER    /dev/asm-diskf                           OCR          grid     asmadmin
   4:      10240 MB CANDIDATE /dev/asm-diskg                           #            grid     asmadmin
   5:      10240 MB CANDIDATE /dev/asm-diskh                           #            grid     asmadmin
   6:      10240 MB CANDIDATE /dev/asm-diski                           #            grid     asmadmin
   7:      15360 MB MEMBER    /dev/asm-diskj                           DATA         grid     asmadmin
   8:      15360 MB MEMBER    /dev/asm-diskk                           DATA         grid     asmadmin
   9:      15360 MB MEMBER    /dev/asm-diskl                           DATA         grid     asmadmin
  10:      10240 MB MEMBER    /dev/asm-diskm                           FRA          grid     asmadmin
  11:      10240 MB MEMBER    /dev/asm-diskn                           FRA          grid     asmadmin
  12:      10240 MB MEMBER    /dev/asm-disko                           FRA          grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                                          
================================================================================

6.4、創建數據庫

[oracle@raclhr-21c-n1 ~]$ export DISPLAY=192.168.59.1:0.0
[oracle@raclhr-21c-n1 ~]$ dbca

七、靜默安裝集羣和db

安裝之前重啓一次OS，並檢查網絡和共享盤是否正確。

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html/7

7.1、靜默安裝grid

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html/7

7.2、靜默安裝db軟件

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html/7

7.3、創建DATA和FRA磁盤組

參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html/7

7.4、靜默創建數據庫

-- rac數據庫
dbca -silent -ignorePreReqs  -ignorePrereqFailure  -createDatabase -templateName General_Purpose.dbc -responseFile NO_VALUE \
-gdbname rac21c  -sid rac21c \
-createAsContainerDatabase TRUE \
-sysPassword lhr -systemPassword lhr -dbsnmpPassword lhr \
-datafileDestination '+DATA' -recoveryAreaDestination '+FRA' \
-storageType ASM \
-characterset AL32UTF8 \
-totalMemory 1024 \
-databaseType OLTP \
-emConfiguration none \
-nodeinfo raclhr-21c-n1,raclhr-21c-n2

-- 單實例
dbca -silent -createDatabase -templateName General_Purpose.dbc -responseFile NO_VALUE \
-gdbname LHRCDB  -sid LHRCDB \
-createAsContainerDatabase TRUE \
-numberOfPDBs 1 \
-pdbName pdbxmm \
-pdbAdminPassword lhr \
-sysPassword lhr -systemPassword lhr \
-datafileDestination '/u01/app/oracle/oradata' \
-recoveryAreaDestination '/u01/app/oracle/flash_recovery_area' \
-redoLogFileSize 50 \
-storageType FS \
-characterset AL32UTF8 -nationalCharacterSet AL16UTF16 \
-totalMemory 1024 \
-databaseType OLTP  \
-emConfiguration NONE

創建過程：

[oracle@raclhr-21c-n1 ~]$ dbca -silent -ignorePreReqs  -ignorePrereqFailure  -createDatabase -templateName General_Purpose.dbc -responseFile NO_VALUE \
> -gdbname rac21c  -sid rac21c \
> -createAsContainerDatabase TRUE \
> -sysPassword lhr -systemPassword lhr -dbsnmpPassword lhr \
> -datafileDestination '+DATA' -recoveryAreaDestination '+FRA' \
> -storageType ASM \
> -characterset AL32UTF8 \
> -totalMemory 1024 \
> -databaseType OLTP \
> -emConfiguration none \
> -nodeinfo raclhr-21c-n1,raclhr-21c-n2
[WARNING] [DBT-06208] The 'SYS' password entered does not conform to the Oracle recommended standards.
   CAUSE: 
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
   ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'SYSTEM' password entered does not conform to the Oracle recommended standards.
   CAUSE: 
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
   ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'PDBADMIN' password entered does not conform to the Oracle recommended standards.
   CAUSE: 
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
   ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
Prepare for db operation
7% complete
Copying database files
27% complete
Creating and starting Oracle instance
28% complete
31% complete
35% complete
37% complete
40% complete
Creating cluster database views
41% complete
53% complete
Completing Database Creation
57% complete
59% complete
60% complete
80% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
 /u01/app/oracle/cfgtoollogs/dbca/rac21c.
Database Information:
Global Database Name:rac21c
System Identifier(SID) Prefix:rac21c
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/rac21c/rac21c.log" for further details.

[oracle@raclhr-21c-n1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.chad
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.net1.network
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.ons
               ONLINE  ONLINE       raclhr-21c-n1            STABLE
               ONLINE  ONLINE       raclhr-21c-n2            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.FRA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.OCR.dg(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            Started,STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.cdp1.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp2.cdp
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.cdp3.cdp
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.cvu
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.qosmserver
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.rac21c.db
      1        ONLINE  ONLINE       raclhr-21c-n1            Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
ora.raclhr-21c-n1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.raclhr-21c-n2.vip
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       raclhr-21c-n2            STABLE
--------------------------------------------------------------------------------

[oracle@raclhr-21c-n1 ~]$ sas

SQL*Plus: Release 21.0.0.0.0 - Production on Mon Aug 23 10:32:50 2021
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SYS@rac21c1> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO

7.5、創建PDB數據庫

SYS@rac21c1> show parameter db_create_file_dest

NAME                                 TYPE                   VALUE
------------------------------------ ---------------------- ------------------------------
db_create_file_dest                  string                 +DATA
SYS@rac21c1> create pluggable database PDBLHR1 admin user lhr identified by lhr;

Pluggable database created.

SYS@rac21c1> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBLHR1                        MOUNTED
SYS@rac21c1> alter pluggable database pdblhr1 open;

Pluggable database altered.

SYS@rac21c1> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBLHR1                        READ WRITE NO
SYS@rac21c1>  alter pluggable database all save state;

Pluggable database altered.

[oracle@raclhr-21c-n1 ~]$ export ORACLE_PDB_SID=PDBLHR1
[oracle@raclhr-21c-n1 ~]$ sas

SQL*Plus: Release 21.0.0.0.0 - Production on Mon Aug 23 10:45:11 2021
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SYS@rac21c1> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDBLHR1                        READ WRITE NO
SYS@rac21c1> show con_name

CON_NAME
------------------------------
PDBLHR1


[root@raclhr-21c-n1 ~]# crsctl status resource  -w "TYPE = ora.database.type" -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.rac21c.db
      1        ONLINE  ONLINE       raclhr-21c-n1            Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
--------------------------------------------------------------------------------
[root@raclhr-21c-n1 ~]# crsctl status resource  -w "TYPE = ora.pdb.type" -t        
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.rac21c.pdblhr1.pdb
      1        ONLINE  ONLINE       raclhr-21c-n1            STABLE
      2        ONLINE  ONLINE       raclhr-21c-n2            STABLE
--------------------------------------------------------------------------------

八、修改基本配置

8.1、禁用crs和db的自啓動

-- 禁用crs自啓動
crsctl disable has

-- 禁用db自啓動
crsctl modify resource ora.rac21c.db -attr AUTO_START=never -unsupported
crsctl stat res ora.rac21c.db -p | grep AUTO_START

8.2、修改SQL提示符

oracle和grid均修改：

cat >> $ORACLE_HOME/sqlplus/admin/glogin.sql <<"EOF"

set linesize 9999 pagesize 9999
set sqlprompt "_USER'@'_CONNECT_IDENTIFIER> "

EOF

參考

內容太多，放不下，部分內容有所刪減，全文可以參考：https://www.xmmup.com/zaicentos7-8shanganzhuangoracle-jiqunrac-21cxiangxiwanzhengban.html