当前位置： 首页 > 升级, 故障处理 > 正文

10G RAC 10.2.0.5升级到10.2.0.5.4报错OPatch failed with error code 73

huangtingzhong 发布于 2013-04-28 16:07升级, 故障处理抢沙发

       最近霉，做什么事情都不顺利，下面是10RAC 从10.2.0.5升级到10.2.0.5.4，1节点正常升级，但是2节点遇到下面的报错 

[oracle@rac2 12419392]$ opatch apply -local
Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/10.2.0/db
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.5.1
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db/oui
Log file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2013-04-28_21-03-57PM.log

Patch history file: /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '12419392' to OH '/u01/app/oracle/product/10.2.0/db'
OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/10.2.0/db. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

ApplySession failed: ApplySession failed to prepare the system. Unable to create patchObject
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

环境：OS  RHEL 5.8 DB:10.2.0.5 RAC

1节点正常升级，在2节点遇到报错

故障现象：

1，opatch apply运行升级

[oracle@rac2 12419392]$ opatch apply -local
Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/10.2.0/db
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.5.1
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db/oui
Log file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2013-04-28_21-03-57PM.log

Patch history file: /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '12419392' to OH '/u01/app/oracle/product/10.2.0/db'
OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/10.2.0/db. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

ApplySession failed: ApplySession failed to prepare the system. Unable to create patchObject
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

故障处理结果：

      通过下载相应的补丁包，CP相关的文件后，升级正常

故障分析过程：

     1，使用opatch apply报下面的错误

[oracle@rac2 12419392]$ opatch apply -local
Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/10.2.0/db
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.5.1
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db/oui
Log file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2013-04-28_21-03-57PM.log

Patch history file: /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '12419392' to OH '/u01/app/oracle/product/10.2.0/db'
OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/10.2.0/db. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

ApplySession failed: ApplySession failed to prepare the system. Unable to create patchObject
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

使用opatch lsinventory也可样报错
[oracle@rac2 db]$ opatch lsinventory
Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/10.2.0/db
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.5.1
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db/oui
Log file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2013-04-28_21-08-30PM.log

Patch history file: /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt



Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject

OPatch failed with error code 73

升级报错，lsinventory同样也报错，通过opatch lsinventory可以看到报cannot load inventory。

2，查看inventory相关的目录、文件、权限

[oracle@rac2 db]$ cat /etc/oraInst.loc 
inventory_loc=/u01/app/oracle/oraInventory
inst_group=dba
[oracle@rac2 db]$ chown -R oracle:dba /u01/app/oracle/oraInventory


[oracle@rac2 12419392]$ ls -ld $ORACLE_HOME
drwxrwxrwx 65 oracle dba 4096 Apr 28 21:01 /u01/app/oracle/product/10.2.0/db
[oracle@rac2 12419392]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VG00-root
                       35G  7.3G   26G  23% /
/dev/sda1              99M   24M   71M  25% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
//192.168.111.1/install
                      932G  582G  350G  63% /soft
                      
[oracle@rac2 ContentsXML]$ cat inventory.xml 
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 2005 Oracle Corporation. All rights Reserved -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>10.2.0.1.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraCrs10g_home" LOC="/u01/app/oracle/product/10.2.0/crs" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rac1"/>
      <NODE NAME="rac2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDb10g_home1" LOC="/u01/app/oracle/product/10.2.0/db" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="rac1"/>
      <NODE NAME="rac2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

都是正常的，跟1节点的内容都是一样。

3，开启opatch 的debug功能

[oracle@rac2 12419392]$ export OPATCH_DEBUG=TRUE
[oracle@rac2 12419392]$  opatch lsinventory
_osArch is i386
/u01/app/oracle/product/10.2.0/db/jdk/bin/java -cp /u01/app/oracle/product/10.2.0/crs/OPatch/ocm/lib/emocmutl.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/ocm/lib/emocmclnt.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/OraPrereq.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/xmlparserv2.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/share.jar:/u01/app/oracle/product/10.2.0/db/oui/jlib/srvm.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatch.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatchutil.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatchprereq.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatchactions.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatchfmw.jar:/u01/app/oracle/product/10.2.0/crs/OPatch/jlib/opatchext.jar -DOPatch.ORACLE_HOME=/u01/app/oracle/product/10.2.0/db -DOPatch.DEBUG=true -DOPatch.RUNNING_DIR=/u01/app/oracle/product/10.2.0/crs/OPatch   oracle/opatch/OPatch lsinventory
Invoking OPatch 10.2.0.5.1
......................................
OUISessionManager::getInventorySession()
Caller Details: 
        Caller Name : OPatch    Caller Version : 10.2.0.5.1     Requested Read-only access : true       Oracle Home : /u01/app/oracle/product/10.2.0/db
OUISessionManager::register()
Registering the caller : OPatch
OracleHomeInventory::createInventoryObj() gets OUIInstallAreaControl object
OracleHomeInventory::createInventoryObj() gets OUIInstallInventory object
OracleHomeInventory::createInventoryObj() gets OUIOracleHomeInfo object
OracleHomeInventory::createInventoryObj() construction done
LsInventory::loadAndPrintInventory()
Retrieving inventory from Oracle Home...
OracleHomeInventory::load()
OracleHomeInventory::load() gets a Vector of all product entries
OracleHomeInventory::load() gets a Vector of all one-off entries
OracleHomeInventory::load() begins to process raw data from OUI to build up OPatch primitive classes
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/inventory.xml
PatchObject::PatchObject now looks for the actions file /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/actions
PatchObject::PatchObject now looks for the inventory file /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/inventory
Loading the information from inventory/filemap.


Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
LsInventorySession failed: Unable to create patchObject
Cleaning up the directory : "/u01/app/oracle/product/10.2.0/db/.patch_storage/patch_unzip"...

OPatch failed with error code 73

这里已经看到问题出在9952245这个补丁包的目录，确认相关文件不存在，于是手动创建相关目录，下载9952245的补丁，cp到相当的目录下面

[oracle@rac2 oneoffs]$ mkdir  -p 9952245/etc/config/
[oracle@rac2 oneoffs]$ mkdir -p 9952245/etc/config/
[oracle@rac2 oneoffs]$ cp /soft1/9952245/custom/server/9952245/etc/config/actions.xml /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/
[oracle@rac2 oneoffs]$ cp /soft1/9952245/custom/server/9952245/etc/config/inventory.xml  /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/
[oracle@rac2 oneoffs]$ cp /soft1/9952245/custom/server/9952245/etc/config/actions.xml /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/actions
[oracle@rac2 oneoffs]$ cp /soft1/9952245/custom/server/9952245/etc/config/inventory.xml  /u01/app/oracle/product/10.2.0/db/inventory/oneoffs/9952245/etc/config/inventory

再次运行opatch lsinventory正常

4，使用opatch apply –local升级

[oracle@rac2 12419392]$ unset OPATCH_DEBUG
[oracle@rac2 12419392]$ echo $OPATCH_DEBUG

[oracle@rac2 12419392]$ opatch apply -local
...................

Patching component oracle.xdk.rsf, 10.2.0.5.0...
Updating archive file "/u01/app/oracle/product/10.2.0/db/lib/libxml10.a"  with "lib/libxml10.a/lpxpar.o"
Updating archive file "/u01/app/oracle/product/10.2.0/db/lib32/libxml10.a"  with "lib32/libxml10.a/lpxpar.o"

Patching component oracle.precomp.common, 10.2.0.5.0...

Patching component oracle.rdbms.rman, 10.2.0.5.0...
Running make for target client_sharedlib
Running make for target ioracle
Running make for target iwrap
Running make for target client_sharedlib
Running make for target proc
Running make for target irman
ApplySession adding interim patch '12419392' to inventory

Verifying the update...
Inventory check OK: Patch ID 12419392 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 12419392 are present in Oracle Home.

The local system has been patched and can be restarted.

--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch 12419392 is a superset of the patch(es) [  12419392 ] in the Oracle Home
--------------------------------------------------------------------------------
OPatch Session completed with warnings.

OPatch completed with warnings

[oracle@rac2 12419392]$ opatch lsinventory;
Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/10.2.0/db
Central Inventory : /u01/app/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.5.1
OUI version       : 10.2.0.5.0
OUI location      : /u01/app/oracle/product/10.2.0/db/oui
Log file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2013-04-28_21-47-30PM.log

Patch history file: /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2013-04-28_21-47-30PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2): 

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 4                            10.2.0.5.0
There are 2 products installed in this Oracle Home.


Interim patches (2) :

Patch  12419392     : applied on Sun Apr 28 21:35:38 CST 2013
Unique Patch ID:  13794201
   Created on 29 May 2011, 22:29:59 hrs PST8PDT
   Bugs fixed:
     6402302, 10269717, 10327190, 8865718, 10017048, 9024850, 8394351, 8546356
     9360157, 9770451, 9020537, 9772888, 8664189, 10091698, 12551710, 7519406
     10132870, 8771916, 9109487, 10173237, 10068982, 8350262, 11792865
     11724962, 11725006, 9184754, 8544696, 9320130, 7026523, 8277300, 9726739
     8412426, 12419392, 6651220, 9150282, 9659614, 9949948, 10327179, 8882576
     7612454, 9711859, 9714832, 10248542, 9952230, 9469117, 9952270, 8660422
     10324526, 12419258, 9713537, 10010310, 9390484, 9963497, 12551700
     12551701, 10249537, 12551702, 12551703, 8211733, 12551704, 9548269
     12551705, 12551706, 9337325, 12551707, 7602341, 12551708, 9308296
     10157402, 11737047

Patch  9952245      : applied on Sun Apr 28 06:25:54 CST 2013
Unique Patch ID:  13197883
   Created on 17 Jan 2011, 03:04:34 hrs PST8PDT
   Bugs fixed:
     9952245, 9879181, 9610672, 9872299, 9577921, 8308495, 9103966, 8996060
     6899183, 9261837, 8487143, 9646836



Rac system comprising of multiple nodes
  Local node = rac2
  Remote node = rac1

--------------------------------------------------------------------------------

OPatch succeeded.

升级成功。

5，升级数据字典

[oracle@rac2 12419392]$ srvctl start listener -n rac2
[oracle@rac2 12419392]$ srvctl start asm -n rac2 -i +ASM2
[oracle@rac2 12419392]$ srvctl start database -d htz
[oracle@rac2 12419392]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.htz.db     application    ONLINE    ONLINE    rac1        
ora....z1.inst application    ONLINE    ONLINE    rac1        
ora....z2.inst application    ONLINE    ONLINE    rac2        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    ONLINE    ONLINE    rac1        
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   application    ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    ONLINE    ONLINE    rac2        
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   application    ONLINE    ONLINE    rac2 

[oracle@rac2 12419392]$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.5.0 - Production on Sun Apr 28 21:41:34 2013

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> @catbundle.sql psu apply
SP2-0310: unable to open file "catbundle.sql"
SQL> @?/rdbms/admin/catbundle.sql psu apply

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.

SQL> @?/rdbms/admin/utlrp

TIMESTAMP
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2013-04-28 21:44:34

DOC>   The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC>   objects in the database. Recompilation time is proportional to the
DOC>   number of invalid objects in the database, so this command may take
DOC>   a long time to execute on a database with a large number of invalid
DOC>   objects.
DOC>
DOC>   Use the following queries to track recompilation progress:
DOC>
DOC>   1. Query returning the number of invalid objects remaining. This
DOC>      number should decrease with time.
DOC>         SELECT COUNT(*) FROM obj$ WHERE status IN (4, 5, 6);
DOC>
DOC>   2. Query returning the number of objects compiled so far. This number
DOC>      should increase with time.
DOC>         SELECT COUNT(*) FROM UTL_RECOMP_COMPILED;
DOC>
DOC>   This script automatically chooses serial or parallel recompilation
DOC>   based on the number of CPUs available (parameter cpu_count) multiplied
DOC>   by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC>   On RAC, this number is added across all RAC nodes.
DOC>
DOC>   UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel
DOC>   recompilation. Jobs are created without instance affinity so that they
DOC>   can migrate across RAC nodes. Use the following queries to verify
DOC>   whether UTL_RECOMP jobs are being created and run correctly:
DOC>
DOC>   1. Query showing jobs created by UTL_RECOMP
DOC>         SELECT job_name FROM dba_scheduler_jobs
DOC>            WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>
DOC>   2. Query showing UTL_RECOMP jobs that are running
DOC>         SELECT job_name FROM dba_scheduler_running_jobs
DOC>            WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>#

PL/SQL procedure successfully completed.


TIMESTAMP
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END  2013-04-28 21:44:37


PL/SQL procedure successfully completed.

DOC> The following query reports the number of objects that have compiled
DOC> with errors (objects that compile with errors have status set to 3 in
DOC> obj$). If the number is higher than expected, please examine the error
DOC> messages reported with each object (using SHOW ERRORS) to see if they
DOC> point to system misconfiguration or resource constraints that must be
DOC> fixed before attempting to recompile these objects.
DOC>#

OBJECTS WITH ERRORS
-------------------
                  0

DOC> The following query reports the number of errors caught during
DOC> recompilation. If this number is non-zero, please query the error
DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors
DOC> are due to misconfiguration or resource constraints that must be
DOC> fixed before objects can compile successfully.
DOC>#

ERRORS DURING RECOMPILATION
---------------------------
                          0


PL/SQL procedure successfully completed.

SQL> select * from dba_registry_history;

ACTION_TIME                    ACTION          NAMESPACE  VERSION                                ID BUNDLE_SERIES                  COMMENTS
------------------------------ --------------- ---------- ------------------------------ ---------- ------------------------------ ------------------------------
28-APR-13 05.07.03.128164 AM   VIEW RECOMPILE                                               8289601                                view recompilation
28-APR-13 05.07.03.505281 AM   UPGRADE         SERVER     10.2.0.5.0                                                               Upgraded from 10.2.0.1.0
28-APR-13 09.42.14.182027 PM   APPLY           SERVER     10.2.0.5                                4 PSU                            PSU 10.2.0.5.4

整个数据库升级成功

赏

您可能还会对这些文章感兴趣！

10G RAC 10.2.0.5升级到10.2.0.5.4报错OPatch failed with error code 73：等您坐沙发呢！

发表评论

gravatar

? razz sad evil ! smile oops grin eek shock ??? cool lol mad twisted roll wink idea arrow neutral cry mrgreen

快捷键：Ctrl+Enter