EBS 12.2 -- Recovering Cutover Problems, Adop phase=cutover, Flashback

We know that we can abort and cleanup the patching works till we run the cutover phase. (adop phase=cutover)
This document will be about recovering and aborting a patch cycle even during a cutover.
The methods described in here can be implemented in problematic scenarios such as event of a failed cutover, and as a last resort
If a cutover error occurs, you should first check the error message and try to determine if the problem can be fixed easily, or (as is true in many cases) cutover can be made to succeed simply by running the command again. Restoring to a point before cutover via Flashback recovery should only be done when the error cannot easily be fixed, and continues to fail on subsequent cutover attempts.

So, to rollback the system to a point before the patching cycle was started, we can use Oracle Database's Flashback feature. We can go back to our restore point that we created just before we run the cutover phase. Note that: While creating a restore point, it is recommended to stop application services..

We must create this restore point when; 

we are ready to perform cutover.
All concurrent managers have been shut down cleanly.
There are no current database transactions being performed by any third-party applications.

In order to be able to use flashback restore points we must enable the flashback in the database tier..
We must be in archivelog mode.
We must set our retention to a sufficient value. alter system set db_flashback_retention_target=120 (in minutes)

Activate flashback . alter database flashback on;

Okay, once we activate the flashback, we can create restore point

So just before cutover phase;

we create our restore point as follows;

SQL>alter system switch logfile;
System altered.
SQL>create restore point BEFORE_CUTOVER guarantee flashback database;
Restore point created.
SQL>alter system switch logfile;
System altered.

So we created our Restore point ..

Now suppose we have encountered a problem in the cutover phase, and want to restore our database just before the cutover;

To able to do this;

we first shutdown our database and startup in mount mode as follows;

SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.

Then we issue the following command to restore our database to the restore point;

SQL>flashback database to restore point BEFORE_CUTOVER;
Flashback complete.

Then we open our database in read only mode and check everyting is clear.

alter database open read only;

Lastly, we shutdown and startup our database with openresetlogs option..

SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
Total System Global Area 2142679040 bytes
Fixed Size 1346140 bytes
Variable Size 520095140 bytes
Database Buffers 1593835520 bytes
Redo Buffers 27402240 bytes
Database mounted.
SQL>alter database open resetlogs;
Database altered.

Once the database is altered, we can disable the flashback and drop the restore point as follows;

SQL>alter database flashback off;
Database altered.
SQL>drop restore point BEFORE_CUTOVER;
Restore point dropped.
SQL>alter system set db_recovery_file_dest='';
System altered.
SQL>select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------
NO

Okay we have seen how to revert our database back before the cutover phase , but we may also need to restore our applications file system before the cutover phase, as well..

We can understand if there is a need for doing by looking to the cutover logs...

$NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.
Case 1 - If the log messages indicate that cutover failed before the file systems were switched

If it is the case;

We just run adstpall.sh and clean the running services.. Then we restart them using adstrtal.sh.

Then we continue with aborting the patch cycle and doing a cleanup.

Case 2 - If the log messages indicate that cutover failed after the file systems were switched..

If this is the case; 

We shutdown the application services.

We switch the filesystems..

perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>

Lastly start up all services from the old run file system (using adstrtal.sh on UNIX).