Patch fails --- because of missing Pre-requisite Patch.. APPS

Adpatch is running and fails on one of the workers.To fix this worker and continue with
the patch installation, a new patch (Prerequisite) needs to be applied.
Considering that only 1 adpatch session can run on an instance at any given time,how can
a patch be applied when adpatch is already running?

Solution:
=========

Step 1: Using the adctrl utility, shutdown the workers.
a. adctrl
b. Select option 3 ” Tell worker to shutdown/quit”

Step 2: Backup the FND_INSTALL_PROCESSES table which is owned by the APPS schema.
a. sqlplus apps/ b. create table fnd_install_processes_back as select * from fnd_install_processes;
c. The 2 tables should have the same number of records.

Select count(*) from fnd_install_processes_back;
Select count(*) from fnd_install_processes;

Step 3: Backup the AD_DEFERRED_JOBS table.
a. sqlplus apps/ b. create table AD_DEFERRED_JOBS_back as select * from AD_DEFERRED_JOBS;
c. The 2 tables should have the same number of records.

Select count(*) from AD_DEFERRED_JOBS_back;
Select count(*) from AD_DEFERRED_JOBS;

Step 4: Backup the .rf9 files located in $APPL_TOP/admin//restart directory.
At this point, the adpatch session should have ended and the cursor should be back at
the unix prompt.
a. cd $APPL_TOP/admin/
b. mv restart restart_back
c. mkdir restart

Step 5: Drop the FND_INSTALL_PROCESSES table and the AD_DEFERRED_JOBS table.
a. sqlplus apps/ b. drop table FND_INSTALL_PROCESSES;
c. drop table AD_DEFERRED_JOBS;

Step 6: Apply the new patch (The Prerequisite)

Step 7: Restore the .rf9 files located in $APPL_TOP/admin//restart_back directory.
a. cd $APPL_TOP/admin/
b. mv restart restart_ c. mv restart_back restart

Step 8: Retore the FND_INSTALL_PROCESSES table, which is owned by the APPS schema.
a. sqlplus apps/ b. create table fnd_install_processes as select * from fnd_install_processes_back;
c. The 2 tables should have the same number of records.

Select count(*) from fnd_install_processes;
Select count(*) from fnd_install_processes_back;

Step 9: Restore the AD_DEFERRED_JOBS table.
a. sqlplus apps/ b. create table AD_DEFERRED_JOBS as select * from AD_DEFERRED_JOBS_back;
c. The 2 tables should have the same number of records.

Select count(*) from AD_DEFERRED_JOBS;
Select count(*) from AD_DEFERRED_JOBS_back;

Step 10: Re-create synonyms
a. sqlplus apps/ b. create synonym AD_DEFERRED_JOBS for APPS.AD_DEFERRED_JOBS.
c. create synonym FND_INSTALL_PROCESSES for APPS.FND_INSTALL_PROCESSES;

Step 11: Start adpatch, it will resume where it stopped previously.

==================================

Common Adpatch issues:

While we apply patch get multiple issues like

1. Worker failed patching
2. Network failure
3. Pre-Requisite patch is missing

Whenever adpatch gives error and stops patching because some reason. First check the status of worker with ADCTRL  utility and then go through your adpatch log and get some hint why patch is failed, then try to fix accordingly. Each issue has specific approach for fixing it.

Worker failed patching:

While we check the adpacth log at location PPL_TOP/ADMIN/SID/LOG/adpatch.log and we notice that adpatch failed because of worker has failed patching. We must immediately check the worker log at location APPL_TOP/ADMIN/SID/LOG/adworkxxx.log and check for the issue and fix it.

Once issue is fixed then follow below steps:

1. Login with APPLMGR user
2. Source the environment
3. Start adpatch session again and will ask for continuation of previous session
4. Say yes
5. It will continue patching with previous session only where it was left.

.rf9 is the file which helps the adpatch to determine where patch was left patching. Patching information is held into tables AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES.

Network Issue:

While we are applying application patch and network is failed in between. In this case we have to apply the patch again with below steps:

1. Login with APPLMGR user
2. Source the environment
3. adpacth - It will ask for continue to previous session
4. Say yes

If still patching is not working, Please check the log at location $APPL_TOP/ADMIN/SID/LOG/adpatch.log and sometimes it shows some autopatch error. In this case, we must check the worker status with ADCTRL utility with option 1. It shows as worker is still under running.

This is the actual problem and will stop the worker with ADCTRL utility selecting option 3 as below:

AD Controller Menu

---------------------------------------------------

1. Show worker status

2. Tell worker to restart a failed job

3. Tell worker to quit

4. Tell manager that a worker failed its job

5. Tell manager that a worker acknowledges quit

6. Restart a worker on the current machine

7. Exit

But if you check the status of worker again with ADCTRL utility, It shows as worker is still running. Problem is still not resolved and for resolving the issues to continue with patching worker status should change to failed.

So in this case will get into ADCTRL utility menu and opt for option 4: Tell manager that a worker failed its job. Now if you check the worker status, it will change to failed.

You can start adpatch again and continue with session to patch where patch had stopped working.

Pre-Requisite patch is missing:

if patch is failed because of pre-requisite patch missing. In this case , we need to take the backup of AD_DEFERRED_JOBS, FND_INSTALLED_PROCESSES and restart directory(containing .rf9 file). Location of restart directory is $APPL_TOP/ADMIN/SID/restart.

Drop these tables and start with new session and apply the pre-requisite patch and check the log.

One pre-requisite patch is complete, recover the backed up tables and restart directory start adpatch session and continue with previous session.

Steps to apply pre-requisite patch:

1.      Stop the worker with ADCTRL utility.

2.      Backup FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables.

3.      Backup the .rf9 file located at $APPL_TOP/admin/<SID>/restart directory.

4.      Drop FND_INSTALL_PROCESSES and AD_DEFERRED_JOBS tables.

5.       Apply the pre-requisite patch.

6.      Restore the backed up .rf9 file or restart directory.

7.      Restore the FND_INSTALL_PROCESSES table.

8.      Restore the AD_DEFERRED_JOBS table.

9.      Start adpatch, it will resume where it stopped previously.

What are the tables created while we start adpatch session?

AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES tables are created.

What does AD_DEFERRED_JOBS and FND_INSTALLED_PROCESSES tables contain?

FND_INSTALLED_PROCESSES: adpacth creates FND_INSTALL_PROCESSES table, assigns each worker unique ID with new row for each worker. This table is used to store the information about the job for each worker. Once all jobs are completed, worker shuts down and drops FND_INSTALL_PROCESSES table.

AD_DEFERRED_JOBS: This table holds the information of failed (workers which has failed its job). This table is created on starting the session of adpatch utility and drops once it is complete.

From which table ADCTRL gathers information?

FND_INSTALL_PROCESSES