1.項目要求：

根據安全掃描漏洞整改要求，需對某系統（麒麟linux6.5+oracle12c rac）打220118版本的PSU補丁。

系統狀態、磁盤空間等等檢查都OK，根據保守經驗，申請停機時間3小時
由于在9i/10g/11g/12c/19c rac+centos/redhat/solaris/aix等環境都有過打補丁經驗。應該說這次打補丁差不多十拿九穩的事情，沒想到很快就被打臉了:)
曾經在19c rac打過補丁踩坑記錄，此時沒有參考意義:(http://m.sunline.cc/doc/46086

2.實施準備：

2.1介質主備：

下載PSU最新補丁集：Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文檔 ID 2118136.2)
下載最新opatch工具：How To Download And Install The Latest OPatch(6880880) Version (Doc ID 274526.1)

2.2兩節點備份權限：

根據：Script to capture and restore file permission in a directory (for eg. ORACLE_HOME) (Doc ID 1515018.1)

chmod 755 permission.pl 
./permission.pl /u01/app/12.2.0/grid
cp restore-perm-二-9月-20-14-22-39-2022.cmd r.cmd

2.3兩節點備份文件（注意保證空間足夠）：

cd /u01/app
tar -cvf grid.tar ./12.2.0 ./grid --exclude=./grid/admin --exclude=./grid/diag --exclude=./12.2.0/grid/network/log --exclude=./12.2.0/grid/log --exclude=./12.2.0/grid/rdbms/audit
tar -cvf oracle.tar ./oracle --exclude=./oracle/admin --exclude=./oracle/diag
tar -cvf oraInventory.tar ./oraInventory

2.4opatch工具解壓和替換：

先/etc/profile增加環境變量，方便root用戶執行和查看命令：

vi /etc/profile
export PATH=/u01/app/12.2.0/grid/OPatch:/u01/app/12.2.0/grid/bin:$PATH;
source /etc/profile

再替換oracle和grid下opatch工具：

cd /home/oracle/soft/2022
unzip p6880880_122010_Linux-x86-64-OPatch 12.2.0.1.32.zip
rm -rf /u01/app/oracle/product/12.2.0/db_1/OPatch
cp -rp OPatch /u01/app/oracle/product/12.2.0/db_1/
chown -R oracle:oinstall /u01/app/oracle/product/12.2.0/db_1/OPatch
rm -rf /u01/app/12.2.0/grid/OPatch
cp -rf OPatch /u01/app/12.2.0/grid/
chown -R grid:oinstall /u01/app/12.2.0/grid/OPatch
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatch version

2.5補丁沖突檢查

根據readme執行即可，如下示例,省略部分：

grid用戶下核查：
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33678030
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33116894
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/26839277
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33610989

oracle用戶下核查：
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128

root用戶下預執行分析：（此次預執行分析都成功，但實際執行卻報錯，繼續往下看）
opatchauto apply /home/oracle/soft/2022/33559966/33583921 -analyze

3.正式實施：

root用戶下在節點1先執行：

opatchauto apply /home/oracle/soft/2022/33559966/33583921

可以看到db補丁成功，但開始應用grid補丁的時候失敗了：


根據錯誤位置信息查看內容：

執行chmod -R 755 /u01/app/12.2.0/grid/jdk，然后opatchauto resume
繼續報錯jdk目錄權限：
查到相關問題參考：


賦權chown -R grid:oinstall /u01/app/12.2.0/grid/jdk，然后opatchauto resume,還是報錯bin、lib等目錄權限，約9千行的錯誤：

根據錯誤日志，提取grep Destionation err.txt > 1.txt，然后排序去重，過濾出300多行的文件。
編輯腳本執行賦權：

chown grid:oinstall /u01/app/12.2.0/grid/bin/HiveMetadata.jar
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsccm
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsdriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfshanfs
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsload
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsregistrymount
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_initializer
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_monitor
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_preapply
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_transport
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsreplcrs
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrm
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfssinglefsmount
chown grid:oinstall /u01/app/12.2.0/grid/bin/advmccb
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdboot
chown grid:oinstall /u01/app/12.2.0/grid/bin/afddriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdload
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/appagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg
chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/asmproxy
chown grid:oinstall /u01/app/12.2.0/grid/bin/bdschecksw
chown grid:oinstall /u01/app/12.2.0/grid/bin/cdpc.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutlo.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutls.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clscfg.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsdiaglsnr.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsecho.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsid.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clssproxy.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_getperm.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_profile.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_register.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_relocate.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_setperm.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_start.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stat.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stop.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_unregister.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crscdpd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsctl.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsdiag.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsrename.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/crstmpl.scr
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdagent
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdmonitor
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssvfupgd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/dbfs_client
chown grid:oinstall /u01/app/12.2.0/grid/bin/diagcollection.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/diagsnap_setup.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/diskmon
chown grid:oinstall /u01/app/12.2.0/grid/bin/dsml2ldif
chown grid:oinstall /u01/app/12.2.0/grid/bin/emcrsp.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evminfo.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmlogger.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmkbin.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmklib.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmpost.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmshow.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmsort.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmwatch.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evt.sh
chown grid:oinstall /u01/app/12.2.0/grid/bin/fmputl
chown grid:oinstall /u01/app/12.2.0/grid/bin/ghappctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/gipcd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd
chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnpd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnptool.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gsd
chown grid:oinstall /u01/app/12.2.0/grid/bin/jwcctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/kfod
chown grid:oinstall /u01/app/12.2.0/grid/bin/kgpmon
chown grid:oinstall /u01/app/12.2.0/grid/bin/lcsscan
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapadd
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapaddmt
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapbind
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapcompare
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapdelete
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmoddn
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodify
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodifymt
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsdb.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxchknlb
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxegen
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxinst
chown grid:oinstall /u01/app/12.2.0/grid/bin/mdnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/mkstore
chown grid:oinstall /u01/app/12.2.0/grid/bin/mtactl
chown grid:oinstall /u01/app/12.2.0/grid/bin/ochadctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/oclskd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oclumon.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrcheck.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrconfig.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrdump.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssdrim.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/octssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/odig
chown grid:oinstall /u01/app/12.2.0/grid/bin/odig.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/odnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ohasd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oifcfg.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd
chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/okadriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/okaload
chown grid:oinstall /u01/app/12.2.0/grid/bin/okaroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/olfscmd
chown grid:oinstall /u01/app/12.2.0/grid/bin/olfsroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/ologgerd
chown grid:oinstall /u01/app/12.2.0/grid/bin/olsnodes.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oprocd
chown grid:oinstall /u01/app/12.2.0/grid/bin/oraagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnfs
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oranetmonitor.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/orapki
chown grid:oinstall /u01/app/12.2.0/grid/bin/orarootagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/osysmond.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/owm
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgeut
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgevtf
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgmain
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgvip
chown grid:oinstall /u01/app/12.2.0/grid/bin/schema
chown grid:oinstall /u01/app/12.2.0/grid/bin/sclsspawn
chown grid:oinstall /u01/app/12.2.0/grid/bin/scriptagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/srvconfig
chown grid:oinstall /u01/app/12.2.0/grid/bin/usrvip
chown grid:oinstall /u01/app/12.2.0/grid/bin/xml
chown grid:oinstall /u01/app/12.2.0/grid/crs/install/s_crsconfig_defs
chown grid:oinstall /u01/app/12.2.0/grid/crs/install/tfa_setup
chown grid:oinstall /u01/app/12.2.0/grid/jlib/gns.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/netcfg.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvm.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmasm.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmhas.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster-common.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster.jar
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsdriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfshanfs.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsregistrymount.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsreplcrs.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfssinglefsmount.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfstoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/afddriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdtoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdafd.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdambr.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdamdu.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdbase.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdpasswd.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdqg.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdshare.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdsys.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdug.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcommand.xml
chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_common.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_lib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/libCurveFit.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libNeticaJ.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagent12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagfw12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagtsh.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclnt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclntsh12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmperl12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_asym.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcha12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclient12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so.12.1
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntshcore.so.12.1
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsce12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclscred12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsnsjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsr12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsra12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcommon12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcore12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcorejava.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf_mdb12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.sig
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libctx12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libctxc12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libdbcfg12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libeons.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libeonsserver.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgeneric12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgns12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgnsjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libhasgen12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libjagent12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libldapclnt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libn12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnbeq12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libncrypt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnhost12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnid.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnzst12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libntcps12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocr12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrb12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrutl12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/liboevm.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/liborabz2.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libordsdo12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libowm2.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpsa12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/librdjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libserver12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpg.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpr.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskjcx12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsql12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsqlplus.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmcred12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmhas12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmocr12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libuini12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libxml12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libzt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libztkg12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/naect.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/naeet.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/nautab.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/okadriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/okaload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okaroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okatoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/olfscmd.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/olfsroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsdriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfshanfs.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsregistrymount.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfssinglefsmount.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afddriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okadriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/s_jwcctl_lib.pm
chown grid:oinstall /u01/app/12.2.0/grid/srvm/jlib/srvctl.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable.pm
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable/Wrap.pm
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/collectfiles.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/discover_ora_stack.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/getppid.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/metric_iorm.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/patchtfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/profiling.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfasetup.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin/tnt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf/tnt.prop.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-cli-1.2.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-lang3-3.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-logging-1.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/tnt.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xmlparserv2.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xternal.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/input
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/init.tfa.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.aix
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.hpux
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.linux
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.sunos
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.conf
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/roottfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/.buildid
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/usableports.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/RATFA.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/Symlink.so
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/commons-io-2.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jdev-rt.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-4.1.27.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-5.0.84.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jewt4.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jsch-0.1.51.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/ojdbc5.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/share.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/public.jks
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/date_patterns.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignore_extensions.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignorefiles.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/mask_strings.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/problemset.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/searchStrings.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/tasks.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.jks
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.md5
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_directories.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_setup.txt

mkdir -p /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin
mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/jdk
mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/javafx
mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited
mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited
mkdir -p /u01/app/12.2.0/grid/jdk/legal/jdk
mkdir -p /u01/app/12.2.0/grid/jdk/legal/javafx

chown -R grid:oinstall /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/jdk
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/javafx
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/jdk
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/javafx
chown -R grid:oinstall /u01/app/12.2.0/grid/crs/install
chown grid:oinstall /u01/app/12.2.0/grid/bin
chown grid:oinstall /u01/app/12.2.0/grid/lib

繼續opatchauto resume，還是報錯

進程占用處理：

fuser -ck /u01/app/12.2.0/grid/bin/oracle            
fuser -ck /u01/app/12.2.0/grid/lib/libclntsh.so.12.1 
fuser -ck /u01/app/12.2.0/grid/lib/libasmclntsh12.so 

多次執行fuser刪進程，確認沒有進程了即可
繼續opatchauto resume，提示grid patch成功，但crsctl start crs最終還是報錯了：


第一個節點到此失敗后嘗試各種辦法，均無法啟動crsd集群服務，
比如根據CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)嘗試：

/u01/app/12.2.0/grid/crs/install/rootcrs.sh -prepatch
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -postpatch
opatchauto resume還是失敗

由于時間緊迫，沒有多余的時間再做研究分析，只能宣告此次打補丁失敗，決定回退。

ps -ef|grep d.bin后kill掉相關進程。
然后tar -xvf解壓原先備份的tar包。重啟集群服務恢復正常。通知應用檢查恢復業務。

收集好錯誤日志信息，第二天開始復盤分析問題。
由于手頭沒有麒麟linux6.5介質，所以嘗試先用centos6.10搭相同的12c rac環境，補丁版本保持一致。（高效的做法：虛機環境+克隆+nfs共享存儲+系統快照）
相關參考：
克隆的網卡問題https://sqlora.blog.csdn.net/article/details/126869607
nfs搭asmhttps://sqlora.blog.csdn.net/article/details/126951455
沒想到同樣的步驟很順利的打上了PSU補丁，沒有遇到任何錯誤。
此時懷疑麒麟系統是否存在兼容性bug問題？由于用戶要求做安全整改必須打上補丁，所以只能找用戶要來了麒麟linux6.5系統介質。
同樣搭好相同的12c rac環境，補丁版本保持一致。希望能重現故障。

功夫不負有心人，節點1操作果然重現了生產環境打補丁的故障現象。于是小心翼翼的分析每一步錯誤，并做好記錄。查閱大量資料及測試，最終順利打上了集群補丁。

測試驗證過程中，同樣遇到了上述問題crs無法啟動，通過查閱oracle資料發現：
CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)其中說到節點兩邊補丁級別不一致的情況下，肯定是無法啟動的，所以決定放棄嘗試，先不管。

繼續給第二個節點補丁

opatchauto apply /home/oracle/soft/2022/33559966/33583921

報錯如下：

然后檢查兩節點的補丁情況，級別和補丁完全一致，但狀態不對。
應該是【NORMAL】，但都顯示【ROLLING PATCH】
雖然狀態不對，但此時發現節點1和節點2的集群服務均是正常的，檢查命令：

crsctl query crs activeversion -f
kfod op=patches
kfod op=patchlvl
crsctl check crs
crsctl stat res -t -init
crsctl stat res -t 

[grid@rac1 ~]$ kfod op=patches
---------------
List of Patches
===============
26839277
33116894
33587128
33610989
33678030
[grid@rac1 ~]$ kfod op=patchlvl
-------------------
Current Patch level
===================
3975995681
[grid@rac1 ~]$ ssh rac2
Last login: Tue Sep 20 05:30:51 2022 from rac1
[grid@rac2 ~]$ kfod op=patches
---------------
List of Patches
===============
26839277
33116894
33587128
33610989
33678030
[grid@rac2 ~]$ kfod op=patchlvl
-------------------
Current Patch level
===================
3975995681

既然版本和補丁都一致，剩下就是狀態的處理了：
參考：Software Patch Mismatch After Applying Bundle Patch or PSU on Grid Home and the Cluster Upgrade State is [ROLLING PATCH] (Doc ID 2176790.1)
在節點1執行如下命令后，節點1、2狀態均恢復正常：

處理操作命令如下：

[root@rac1 ~]# clscfg -patch
clscfg: -patch mode specified
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
[root@rac1 ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [927320293].
[root@rac1 ~]# crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [3975995681].
[root@rac1 ~]# su - grid
[grid@rac2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681].
[grid@rac2 ~]$ exit
logout
Connection to rac2 closed.
[grid@rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681].

終于在麒麟虛機環境上第一次圓滿打上了補丁。
經過反復改進嘗試，最終整理了更順暢的打補丁步驟【麒麟Linux6.5+Oracle 12c rac psu 190716->220118驗證測試】