oracle rac 12c环境脱离节点故障重建

爱被打了一巴掌 2022-06-17 08:16 617阅读 0赞

\#oracle用户在正常节点删除故障节点实例  
dbca -silent -deleteInstance -nodeList rac3 -gdbName rdato -instanceName rdato3 -sysDBAUserName sys -sysDBAPassword aa  
  
  
\#执行完grid用户查看资源池，此时rac3已经不在serverpool里，接下来继续删除cluster信息  
srvctl config serverpool  
  
  
\#root用户  
cd /此处grid安装目录/bin  
  
  
\#停止故障节点集群  
./crsctl stop cluster -n rac3  
\#查看资源状态  
./crsctl status res -t  
./crsctl check cluster -all  
\#删除应用层  
./crsctl delete node -n rac3  
  
  
\#grid用户重新配置正常节点，将rac3踢出集群  
cd /u01/grid/product/oui/bin  
./runInstaller -updateNodeList ORACLE\_HOME=/u01/grid/product "CLUSTER\_NODES=\{rac1,rac2\}" CRS=TRUE  
\#rac节点删除成功检查  
olsnodes -i -t -s  
cluvfy stage -post nodedel -n rac3 -verbose  
  
  
  
  
\#删除故障节点所有相关目录，如需要重新加入节点并重启，验证ssh连通，重新scan asmdisk  
rm -rf /u01/\*  
rm -rf /etc/ora\*  
rm -rf /etc/inittab.crs  
rm -rf /etc/init.d/init.ohasd  
rm -rf /var/tmp/.oracle  
mkdir -p /u01/oracle/db  
mkdir -p /u01/grid/product  
chown -R oracle:oinstall /u01  
chown -R grid:oinstall /u01/grid  
mkdir -p /u01/crs  
chown -R grid:oinstall /u01/crs  
chmod 775 /u01  
reboot  
  
  
\#移除dns  
mv /etc/resolv.conf /etc/resolv.conf.bak  
  
  
  
  
  
  
  
  
\#在已有节点上grid和oracle用户下验证  
cluvfy stage -pre nodeadd -n rac3 -verbose  
  
  
\#grid用户在正常节点重新加入故障节点  覆盖安装选项-invPtrLoc /u01/grid/product/oraInst.loc  
cd $ORACLE\_HOME/addnode  
./addnode.sh -silent "CLUSTER\_NEW\_NODES=\{rac3\}" "CLUSTER\_NEW\_PRIVATE\_NODE\_NAMES=\{rac3-priv\}" "CLUSTER\_NEW\_VIRTUAL\_HOSTNAMES=\{auto\}" "CLUSTER\_NEW\_NODE\_ROLES=\{hub\}"  
  
  
\#遇到一奇怪问题  
\[grid@rac1 addnode\]$ ./addnode.sh -silent "CLUSTER\_NEW\_NODES=\{rac3\}" "CLUSTER\_NEW\_PRIVATE\_NODE\_NAMES=\{rac3-priv\}" "CLUSTER\_NEW\_VIRTUAL\_HOSTNAMES=\{auto\}" "CLUSTER\_NEW\_NODE\_ROLES=\{auto\}"  
Starting Oracle Universal Installer...  
  
  
Checking Temp space: must be greater than 120 MB.   Actual 1383825 MB    Passed  
Checking swap space: must be greater than 150 MB.   Actual 24199 MB    Passed  
\[FATAL\] PRVG-11402 : New nodes cannot have AUTO node role unless the existing cluster consists entirely of AUTO nodes  
\[grid@rac1 addnode\]$ ./addnode.sh -silent "CLUSTER\_NEW\_NODES=\{rac3\}" "CLUSTER\_NEW\_PRIVATE\_NODE\_NAMES=\{rac3-priv\}" "CLUSTER\_NEW\_VIRTUAL\_HOSTNAMES=\{auto\}" "CLUSTER\_NEW\_NODE\_ROLES=\{hub\}"  
Starting Oracle Universal Installer...  
  
  
Checking Temp space: must be greater than 120 MB.   Actual 1383825 MB    Passed  
Checking swap space: must be greater than 150 MB.   Actual 24199 MB    Passed  
\[FATAL\] \[INS-43047\] Clusterware configured in Auto mode cannot have nodes of type HUB or LEAF.  
   ACTION: Ensure that the nodes provided for addnode configuration are of type Auto.  
  
  
\#查看集群模式是hub，感觉这错误提示有点自相矛盾了，反正没弄明白，试了各种参数，最终执行成功了，一脸懵B  
crsctl get node role config -all  
  
  
\#尝试各存活依次重启集群操作也没有成功  
./crsctl stop cluster -n rac1  
./crsctl start cluster -n rac1  
  
  
  
  
\#以oracle用户在已有节点上执行安装db操作  
cd $ORACLE\_HOME/addnode  
./addnode.sh -silent "CLUSTER\_NEW\_NODES=\{rac3\}"  
\#重装期间遇到如下错误  
Prepare Configuration successful.  
..................................................   8% Done.  
You can find the log of this install session at:  
 /u01/oraInventory/logs/addNodeActions2017-04-20\_10-29-33AM.log  
SEVERE:The new nodes 'rac3' are already part of the cluster.  
\[FATAL\] \[INS-10008\] Session initialization failed  
   CAUSE: An unexpected error occured while initializing the session.  
   ACTION: Contact Oracle Support Services or refer logs  
   SUMMARY:  
     
\#网上找到解决方案oracle用户生成如下超链接（所有存活节点都要执行）未解决问题  
mv /u01/oracle/db/bin/lsnodes     /u01/oracle/db/bin/lsnodes.bak  
mv /u01/oracle/db/oui/bin/lsnodes /u01/oracle/db/oui/bin/lsnodes.bak  
ln -s /u01/grid/product/bin/olsnodes     /u01/oracle/db/bin/lsnodes  
ln -s /u01/grid/product/bin/olsnodes.bin /u01/oracle/db/bin/lsnodes.bin  
ln -s /u01/grid/product/bin/olsnodes     /u01/oracle/db/oui/bin/lsnodes  
ln -s /u01/grid/product/bin/olsnodes.bin /u01/oracle/db/oui/bin/lsnodes.bin  
  
  
\#使用oracle用户重新配置正常节点，然后重新执行db安装  
cd /u01/oracle/db/oui/bin  
./runInstaller -updateNodeList ORACLE\_HOME=/u01/oracle/db "CLUSTER\_NODES=\{rac1,rac2\}" CRS=TRUE  
  
  
  
  
\#在已有节点上添加实例（在新增节点上执行失败），由于是admin managed模式，使用dbca安装实例，policy managed修改serverpool即可  
dbca -silent -addInstance -nodeList rac3 -gdbName rdato -instanceName rdato3 -sysDBAUserName sys -sysDBAPassword aa  
  
  
\[grid@rac3 ~\]$ srvctl start instance -d rdato -i rdato3  
PRCR-1013 : Failed to start resource ora.rdato.db  
PRCR-1064 : Failed to start resource ora.rdato.db on node rac3  
CRS-5017: The resource action "ora.rdato.db start" encountered the following error:   
ORA-00119: invalid specification for system parameter REMOTE\_LISTENER  
ORA-00132: syntax error or unresolved network name 'rclus-scan.rdato.jc.com:42401'  
. For details refer to "(:CLSN00107:)" in "/u01/crs/diag/crs/rac3/crs/trace/crsd\_oraagent\_oracle.trc".  
  
  
\#解决方案：将rac3的域名解析文件恢复mv /etc/resolv.conf.bak /etc/resolv.conf ，然后srvctl启动实例  
  
  
\#查看serverpool  
srvctl config serverpool  
  
  
  
  
  
  
\#执行完后检查集群层以及应用层是否正常  
\#集群层  
crsctl check crs  
crsctl check cluster -all  
crsctl status res -t  
\#应用层  
srvctl status nodeapps  
srvctl status asm  
srvctl status listener  
srvctl status instance -d rdato -i rdato1,rdato2,rdato3  
  
  
\#恢复dns  
mv /etc/resolv.conf.bak /etc/resolv.conf  
  
  
\#grid用户检查节点安装是否成功  
cluvfy stage -post nodeadd -n rac3