Saturday, January 18, 2014

Oracle Rollup patch and Min Downtime Patch

From:

http://abcdba.com/abcdbarachowtoapplyrollingopatch

 

OVERVIEW

An Oracle OPatch for a RAC configuration can take take one of two forms, either as a Minimum Downtime OPatch or as a Rolling Opatch. The instructions for applying the former can be foundhere. Instructions for performing the latter can be found below.
In addition to being informed by Oracle Support, the type of OPatch can be determined by checking the <Patch No>/etc/config/inventory file. If the the variable online_rac_installable is set totrue then the patch is a rolling patch.

PATCH INSTALLATION INSTRUCTIONS

  • The worked example below will use the following settings:
    • CRS_HOME = /u01/crs/oracle/product/10/crs
    • RDBMS_HOME = /u01/app/oracle/product/db10.2.0
  • The patch being applied is p123456
  • The example output from the various commands has been abridged for readability.
1. Make sure all instances running under the ORACLE_HOME being patched are cleanly shutdown before installing this patch. Also ensure that the tool used to terminate the instance(s) has exited cleanly.
2. Ensure that the directory containing the opatch script appears in your $PATH. Execute which opatch to confirm.
oracle:> which opatch
no opatch in /u01/app/oracle/product/db10.2.0/bin /bin /usr/sbin /usr/bin /usr/local/bin /usr/ccs/bin 
/usr/bin /usr/local/bin /u01/crs/oracle/product/10/crs/bin /u01/crs/oracle/product/10/crs/bin

oracle:> export PATH=$PATH:$ORACLE_HOME/OPatch

oracle:> which opatch
/u01/crs/oracle/product/10/crs/OPatch/opatch
3. Each node of the cluster has its own CRS Home, the patch should be applied as a rolling upgrade. All of the following
steps should be followed for each node.
  • Do not patch two nodes at once.
4. As the Oracle Clusterware (CRS) software owner check CRS_HOME.
oracle:> opatch lsinventory -detail -oh /u01/crs/oracle/product/10/crs
Invoking OPatch 10.2.0.3.0
...
  Remote node = <node x>
--------------------------------------------------------------------------------

OPatch succeeded.
5. As the RDBMS server owner check ORACLE_HOME.
oracle:> opatch lsinventory -detail -oh /u01/app/oracle/product/db10.2.0/
Invoking OPatch 10.2.0.3.0
...
  Remote node = <node x>
--------------------------------------------------------------------------------

OPatch succeeded.
  • The above should list the components and the list of nodes. If the Oracle inventory is not setup correctly the OPatch utility will fail.
6. Unzip the patch set container file, this will create one or more sub-directories.
% unzip p123456.zip

Archive:  p123456.zip
   creating: 123456/
...
7. Shut down the RDBMS and ASM instances, listeners and nodeapps followed by CRS daemons on the local node.
  • To shutdown RDBMS instance on the local node run the following command:
% $ORACLE_HOME/bin/srvctl stop instance -d dbname -i instance_name
  • To shutdown ASM instances run the following command on each node:
% $ORACLE_HOME/bin/srvctl stop asm -n <node_name>
  • To shutdown nodeapps run the following comand on each node:
% $ORACLE_HOME/bin/srvctl stop nodeapps -n <node_name>
8. Now shutdown CRS daemons on each node by running as root:
root # $CRS_HOME/bin/crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
9. Prior to applying this part of the fix, invoke the unlock script as root to unlock protected files.
su -

root # cd <patch directory>/123456
root # custom/scripts/prerootpatch.sh -crshome /u01/crs/oracle/product/10/crs -crsuser oracle
root # exit
10. Now invoke an additional script as the crs software installer/owner. This script will save important configuration settings.
oracle:> cd <patch directory>/123456
custom/scripts/prepatch.sh -crshome /u01/crs/oracle/product/10/crs
custom/scripts/prepatch.sh completed successfully.
  • Note: Make sure the RDBMS portion is only applied to an RDBMS home that meets all the pre-requiste versions.
oracle:> cd <patch directory>/123456
oracle:> custom/server/123456/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/db10.2.0
custom/server/123456/custom/scripts/prepatch.sh completed successfully.
11. After unlocking any protected files and saving configuration settings run opatch as the Oracle Clusterware (CRS) software owner.
cd <patch directory>/123456
oracle:> opatch apply -local -oh /u01/crs/oracle/product/10/crs
Invoking OPatch 10.2.0.3.0

Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation.  All rights reserved..

Oracle Home       : /u01/crs/oracle/product/10/crs
Central Inventory : /u01/app/oracle/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 10.2.0.3.0
OUI version       : 10.2.0.3.0
OUI location      : /u01/crs/oracle/product/10/crs/oui
Log file location : /u01/crs/oracle/product/10/crs/cfgtoollogs/opatch/opatch2008-02-28_11-51-38AM.log

ApplySession applying interim patch '123456' to OH '/u01/crs/oracle/product/10/crs'
ApplySession: Optional component(s) [ oracle.rdbms, 10.2.0.3.0 ]  
 not present in the Oracle Home or a higher version is found.
Invoking fuser to check for active processes.

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/crs/oracle/product/10/crs')

Is the local system ready for patching?

Do you want to proceed? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '123456' for restore. This might take a while...
Backing up files affected by the patch '123456' for rollback. This might take a while...

Patching component oracle.rdbms.rsf, 10.2.0.3.0...
Updating archive file "/u01/crs/oracle/product/10/crs/lib/libgeneric10.a"  with "lib/libgeneric10.a/skgfr.o"
Updating archive file "/u01/crs/oracle/product/10/crs/lib32/libgeneric10.a"  with "lib32/libgeneric10.a/skgfr.o"
ApplySession adding interim patch '123456' to inventory

Verifying the update...
Inventory check OK: Patch ID 123456 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 123456 are present in Oracle Home.
Running make for target client_sharedlib

The local system has been patched and can be restarted.

OPatch succeeded.
12. Use opatch to patch the RDBMS software.
oracle:> opatch apply -local -oh /u01/app/oracle/product/db10.2.0
Invoking OPatch 10.2.0.3.0

Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation.  All rights reserved..

Oracle Home       : /u01/app/oracle/product/db10.2.0
Central Inventory : /u01/app/oracle/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 10.2.0.3.0
OUI version       : 10.2.0.3.0
OUI location      : /u01/app/oracle/product/db10.2.0/oui
Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2008-02-28_11-53-12AM.log

ApplySession applying interim patch '123456' to OH '/u01/app/oracle/product/db10.2.0'
Invoking fuser to check for active processes.
Invoking fuser on "/u01/app/oracle/product/db10.2.0/bin/oracle"

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/db10.2.0')

Is the local system ready for patching?

Do you want to proceed? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '123456' for restore. This might take a while...
Backing up files affected by the patch '123456' for rollback. This might take a while...

Patching component oracle.rdbms.rsf, 10.2.0.3.0...
Updating archive file "/u01/app/oracle/product/db10.2.0/lib/libgeneric10.a"  with "lib/libgeneric10.a/skgfr.o"
Updating archive file "/u01/app/oracle/product/db10.2.0/lib32/libgeneric10.a"  with "lib32/libgeneric10.a/skgfr.o"

Patching component oracle.rdbms, 10.2.0.3.0...
ApplySession adding interim patch '123456' to inventory

Verifying the update...
Inventory check OK: Patch ID 123456 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 123456 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target ioracle

The local system has been patched and can be restarted.

OPatch succeeded.
  • After opatch completes, some configuration settings need to be applied to the patched files.
13. As the RDBMS software owner execute the following:
% <patch directory>/123456/custom/server/123456/custom/scripts/postpatch.sh -dbhome <RDBMS_HOME>

oracle:> <patch directory>/123456/custom/server/123456/custom/scripts/postpatch.sh  \
> -dbhome /u01/app/oracle/product/db10.2.0
Reading ...
...
Reapplying file permissions on /u01/app/oracle/product/db10.2.0/lib/libsrvmhas10.so
14. Restore the security settings and restart CRS by running the following as root:
su -

Sourcing /root/.profile-EIS.....
root # cd <patch directory>/123456
root # custom/scripts/postrootpatch.sh -crshome /u01/crs/oracle/product/10/crs
Checking to see if Oracle CRS stack is already up...
Checking to see if Oracle CRS stack is already starting
WARNING: directory '/u01/crs/oracle/product/10' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
Startup will be queued to init within 30 seconds.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)

PATCH DEINSTALLATION INSTRUCTIONS

To roll back the patch, invoke the following opatch commands to roll back the patch in all homes.
% opatch rollback -id 123456 -local -oh <CRS_HOME>

% opatch rollback -id 123456 -local -oh <RDBMS_HOME>
  • Note: Any problems installing a patch should have a call logged with Oracle support.
========================================================================

OVERVIEW

An Oracle OPatch for a RAC configuration can take take one of two forms, either as a Rolling Opatch or as a Minimum Downtime OPatch. The instructions for applying the former can be foundhere. Instructions for performing the latter can be found below.
In addition to being informed by Oracle Support, the type of OPatch can be determined by checking the <Patch No>/etc/config/inventory file. If the the variable online_rac_installable is set totrue then the patch is a rolling patch.

A WORKED EXAMPLE

1. Ensure all of the instances that make up the RAC database can be shutdown.
2. On each node, shutdown the instance, using the following commands:
         . oraenv
         <SID name>
         sqlplus '/ as sysdba'
         shutdown transactional
         exit
3. On the primary node in the cluster follow the Opatch instructions as prescribed by theREADME.txt file.
  • Here is a real example:
         cd $HOME/DBA/patches
         unzip p6239052_10203_SOLARIS64.zip
         cd 6239052
         export OBJECT_MODE=32_64
         export PATH=$ORACLE_HOME/OPatch:$PATH
         opatch apply

         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-17-48AM.log

         ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
         Invoking fuser to check for active processes.
         Invoking fuser on "/u01/app/oracle/product/db10.2.0/bin/oracle"

         OPatch detected the node list and the local node from the inventory.  
         OPatch will patch the local system then propagate the patch to the remote nodes.

         This node is part of an Oracle Real Application Cluster.
         Remote nodes: 'node2' 'node3'
         Local node: 'node1'
         Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')

         Is the local system ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Backing up files and inventory (not for auto-rollback) for the Oracle Home
         Backing up files affected by the patch '6239052' for restore. This might take a while...
         Backing up files affected by the patch '6239052' for rollback. This might take a while...

         Patching component oracle.rdbms, 10.2.0.3.0...
         Updating archive file "/u01/app/oracle/product/db10.2.0/lib/libserver10.a"  
         with "lib/libserver10.a/kjbm.o"
         ApplySession adding interim patch '6239052' to inventory

         Verifying the update...
         Inventory check OK: Patch ID 6239052 is registered in Oracle Home inventory with proper meta-data.
         Files check OK: Files from Patch ID 6239052 are present in Oracle Home.
         Running make for target ioracle

         The local system has been patched.  You can restart Oracle instances on it.

         Patching in rolling mode.

         Remaining nodes to be patched:
         'node2' 'node3'
         What is the next node to be patched?
         node2
         You have selected 'node2' from 'node2' 'node3'

         The node 'node2' will be patched next.

         Please shutdown Oracle instances running out of this ORACLE_HOME on 'node2'.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')
         Is the node ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Updating nodes 'node2'
            Apply-related files are:
              FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
              DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
              MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

         Instantiating the  
         file "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" 
         by replacing $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" 
         with actual path.
         Propagating files to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt" with actual path.
         Propagating directories to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt" with actual path.
         Running command on remote node 'node2': cd /u01/app/oracle/product/db10.2.0/rdbms/lib; 
         /usr/ccs/bin/make -f ins_rdbms.mk
         ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

         The node 'node2' has been patched.  You can restart Oracle instances on it.

         The node 'node3' will be patched next.

         Please shutdown Oracle instances running out of this ORACLE_HOME on 'node3'.
         (Oracle Home = '/u01/app/oracle/product/db10.2.0')
         Is the node ready for patching?

         Do you want to proceed? [y|n]
         y
         User Responded with: Y
         Updating nodes 'node3'
            Apply-related files are:
              FP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt"
              DP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt"
              MP = "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/make_cmds.txt"

         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_files.txt" with actual path.
         Propagating files to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/.patch_storage/.../rac/copy_dirs.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
         Propagating directories to remote nodes...
         Instantiating the file 
         "/u01/app/oracle/product/db10.2.0/..../rac/make_cmds.txt.instantiated" by replacing
         $ORACLE_HOME in "/u01/app/oracle/product/db10.2.0/.patch_storage/....txt" with actual path.
         Running command on remote node 'node3': cd /u01/app/oracle/product/db10.2.0/rdbms/lib; 
         /usr/ccs/bin/make -f ins_rdbms.mk
         ioracle ORACLE_HOME=/u01/app/oracle/product/db10.2.0 || echo REMOTE_MAKE_FAILED::>&2

         The node 'node3' has been patched.  You can restart Oracle instances on it.

         There were relinks on remote nodes.  
         Remember to check the binary size and timestamp on the nodes 'node2' 'node3' .
         The following make commands were invoked on remote nodes:
         'cd /u01/app/oracle/product/db10.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk 
          ioracle $ORACLE_HOME=/u01/app/oracle/product/db10.2.0'

         OPatch succeeded.
4. Ensure the new patch appears in the inventory by running the following command:
         opatch lsinventory

         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/....log

         Lsinventory Output file location : /u01/app/oracle/product/db10.2.0/....txt

         --------------------------------------------------------------------------------
         Installed Top-level Products (2):

         Oracle Database 10g                                                  10.2.0.1.0
         Oracle Database 10g Release 2 Patch Set 2                            10.2.0.3.0
         There are 2 products installed in this Oracle Home.

         Interim patches (5) :

         Patch  6239052      : applied on Tue Nov 20 02:18:58 GMT 2007
            Created on 31 Oct 2007, 08:01:04 hrs PST8PDT
            Bugs fixed:
              6239052

         Patch  4655998      : applied on Fri May 04 10:23:04 BST 2007
            Created on 13 Apr 2007, 00:31:22 hrs PST8PDT
            Bugs fixed:
              4655998

         Patch  5749953      : applied on Fri Apr 20 16:13:34 BST 2007
            Created on 15 Feb 2007, 02:45:16 hrs US/Pacific
            Bugs fixed:
              5749953

         Patch  5556081      : applied on Mon Apr 16 10:18:22 BST 2007
            Created on 9 Nov 2006, 22:20:50 hrs PST8PDT
            Bugs fixed:
              5556081

         Patch  5557962      : applied on Mon Apr 16 10:17:33 BST 2007
            Created on 9 Nov 2006, 23:23:06 hrs PST8PDT
            Bugs fixed:
              4269423, 5557962, 5528974

         Rac system comprising of multiple nodes
           Local node = node1
           Remote node = node2
           Remote node = node3

         --------------------------------------------------------------------------------

         OPatch succeeded
5. On each node, start the instance, using the following commands:
         . oraenv
         <SID name>
         sqlplus '/ as sysdba'
         startup
         exit
6. On the primary node, ensure the RAC cluster has returned to normal with the following commands:
  • Note: It takes a few minutes to achieve a completely ONLINE state.
         cd $HOME
         . ./10gcrs.env
         crs_stat -t

         Name           Type           Target    State     Host
         ------------------------------------------------------------
         ora....L01.srv application    ONLINE    ONLINE    node1
         ora....L02.srv application    ONLINE    ONLINE    node2
         ora....L03.srv application    ONLINE    ONLINE    node3
         ...
         ora....b03.gsd application    ONLINE    ONLINE    node3
         ora....b03.ons application    ONLINE    ONLINE    node3
         ora....b03.vip application    ONLINE    ONLINE    node3

TROUBLESHOOTING

  • If the OPatch is NOT applied to the primary node in the cluster then the following error output will be produced:
    • ACTION: Restart the patch process on the primary node.
         Invoking OPatch 10.2.0.3.0

         Oracle interim Patch Installer version 10.2.0.3.0
         Copyright (c) 2005, Oracle Corporation.  All rights reserved..

         Oracle Home       : /u01/app/oracle/product/db10.2.0
         Central Inventory : /u01/app/oracle/oraInventory
            from           : /var/opt/oracle/oraInst.loc
         OPatch version    : 10.2.0.3.0
         OUI version       : 10.2.0.3.0
         OUI location      : /u01/app/oracle/product/db10.2.0/oui
         Log file location : /u01/app/oracle/product/db10.2.0/cfgtoollogs/opatch/opatch2007-11-20_02-16-36AM.log

         ApplySession applying interim patch '6239052' to OH '/u01/app/oracle/product/db10.2.0'
         OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/db10.2.0. Possible #
         causes are:
            No read or write permission to ORACLE_HOME/.patch_storage
            Central Inventory is locked by another OUI instance
            No read permission to Central Inventory
            The lock file exists in ORACLE_HOME/.patch_storage
            The Oracle Home does not exist in Central Inventory

         ApplySession failed: ApplySession failed to prepare the system.  No detail available.
         System intact, OPatch will not attempt to restore the system

         OPatch failed with error code 73
  • If the crs_stat -t output doesn't show all items in an ONLINE state after a suitable amount of time, e.g. 15 mins, the cluster may require restarting completely.
    • ACTION: On each node, ONE NODE AT A TIME, use the following commands:
         su -
         <root password>
         sync;sync;reboot

No comments:

Post a Comment