RDBMS/GRID -- Upgrading RAC Database and GRID from 11.2.0.3 to 11.2.0.4

Here is the plan that I used in a critical RAC database upgrade.
The environment was a 2 Node RAC. The OS layer was Oracle Linux 6 64 bit. GRID and RDBMS versions were 11.2.0.3.
This plan worked like a charm and the steps(the methodology, flow and steps) in it, is indeed approved by Oracle Support. (through an Oracle SR)

So, tested, verified and approved :)
Note that, this plan includes after upgrade actions such as applying the latest GI PSU into the 11.2.0.4 environment , as well..

General overview :

-Check the readiness for GRID and RDBMS homes for the upgrade.
-Perform backup of OCR, voting disk and Database (IMPORTANT)
-Install/Upgrade the GRID home.
-Install/Upgrade the database/databases.
-Apply PSU (if required)

Important fact:

Oracle Clusterware and Oracle ASM upgrades are always out-of-place upgrades. With 11g Release 2 (11.2), we cannot perform an in-place upgrade of Oracle Clusterware and Oracle ASM to existing homes.
So, if we have an existing Oracle Clusterware installation, then we upgrade our existing cluster by performing an out-of-place upgrade. We cannot perform an in-place upgrade.
In prior releases, we could use Database Upgrade Assistant(DBUA) to upgrade either an Oracle Database, or Oracle ASM. That is no longer the case. We can only use DBUA to upgrade an Oracle Database instance. We can use Oracle ASM Configuration Assistant (ASMCA) to upgrade Oracle ASM.
Oracle recommends that we leave Oracle RAC instances running. When we start the root script on each node, that node's instances are shut down and then started up again by the rootupgrade.sh script.
Before, upgrading the ORacle databases, we need to upgrade Oracle Clusterware first.

                      --Grid+ASM upgrade section--
1)
Backup (RMAN + Guarenteed Restore point in 14t step actually)

2)
Run orachk and fix the errors reported by it: Document id:1457357.1

3)
Run Cluster Verification utility and fix the errors reported by it.

Example run:
./runcluvfy.sh stage -pre crsinst -upgrade -n node1,node2 -rolling -src_crshome
/u01/app/grid/11.2.0.1 -dest_crshome /u01/app/grid/11.2.0.2 -dest_version 11.2.0.3.0 -fixup -fixupdir /home/grid/fixup -verbose

4)
unset ORA_CRS_HOME (from shell and all the profile files. i.e .profile .cshrc .bash_profile etc..)
unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID

5)
Oracle recommends that you leave Oracle RAC instances running.
When you start the root script on each node, that node's instances are shut down and then started up again by the rootupgrade.sh script.

6)
Start the installer (runInstaller) with the OS owner ASM user. (i.e oracle, grid)
select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.
On the node selection page, select all nodes.
Select installation options as prompted.
When prompted, run the rootupgrade.sh script on each node in the cluster that you want to upgrade.
Run the script on the local node first. The script shuts down the earlier release installation, replaces it with the new Oracle Clusterware release, and starts the new Oracle Clusterware installation.
Because the Oracle Grid Infrastructure home is in a different location than the former
Oracle Clusterware and Oracle ASM homes, update any scripts or applications that use utilities, libraries, or other files that reside in the Oracle Clusterware and Oracle ASM homes.

Note that:
The recommended practice for upgrading Oracle ASM is to upgrade an Oracle ASM instance with the Oracle Universal Installer (OUI) executable file that is located in the Oracle Grid Infrastructure home directory. OUI automatically defaults to upgrade mode when it detects an Oracle ASM instance at an earlier release level.
Oracle ASM Configuration Assistant enables you to upgrade an existing Oracle ASM instance to the current software level and upgrade an older Oracle ASM instance to the latest Oracle Grid Infrastructure home.
You can upgrade an Oracle ASM instance to an Oracle Restart 11g release 2 (11.2) configuration. The recommended practice is to upgrade an Oracle ASM instance with Oracle Universal Installer (OUI).

7)
At the end of the upgrade, if you set the OCR backup location manually to the older release Oracle Clusterware home (CRS home), then you must change the OCR backup location to the Oracle Grid Infrastructure home (Grid home). If you did not set the OCR backup location manually, then this issue does not concern you.

                      --DB+RDBMS upgrade section--
8)
run pre-upgrade tool
sqlplus /as sysdba
SQL> SPOOL upgrade_info.log
SQL> @$11g_ORACLE_HOME/rdbms/admin/utlu112i.sql
SQL> SPOOL OFF
Check the output of the Pre-Upgrade Information Tool in upgrade_info.log.

Note: Any invalid SYS/SYSTEM objects found before upgrading the database are stored in the table named registry$sys_inv_objs. Any invalid non-SYS/SYSTEM objects found before upgrading the database are stored in registry$nonsys_inv_objs.
After the upgrade, run ORACLE_HOME/rdbms/admin/utluiobj.sql to  identify any new invalid objects due to the upgrade.

9)
EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

10)
Ensure the following--
No dbf in backup mode:  SELECT * FROM v$backup WHERE status != 'NOT ACTIVE';
No distributed transaction pending: SELECT * FROM dba_2pc_pending;
No gap between standby dbs: SELECT SUBSTR(value,INSTR(value,'=',INSTR(UPPER(value),'SERVICE'))+1) FROM v$parameter WHERE name LIKE 'log_archive_dest%' AND UPPER(value) LIKE 'SERVICE%';
All batch and cron jobs are disabled.
Ensure that the SHARED_POOL_SIZE , LARGE_POOL_SIZE as well as the JAVA_POOL_SIZE are greater than 150MB
Ensure parameter memory_target is not smaller than 1536m.

11)
PURGE dba_recyclebin

12)
Shut down the database cleanly.

13)
Check invalids and try to validate them.

 SQL>select substr(comp_name,1,40) comp_name, status, substr(version,1,10) version from
 dba_registry order by comp_name;
     
 SQL>select substr(object_name,1,40) object_name,substr(owner,1,15) owner,object_type from
 dba_objects where status='INVALID' order by owner,object_type;
     
 SQL>select owner,object_type,count(*) from dba_objects where status='INVALID' group by
 owner,object_type order by owner,object_type ;

 SQL>@utlrp.sql

14)
Create a guaranteed Restore point (just in case)
SQL>create restore point before_upgrade guarantee flashback database;
SQL>select * from v$restore_point;

15)
Lastly, we install and upgrade our databases using runInstaller
To do this, we simply unzip the 11.2.0.4 installation files, then we go to database subdirectory and execute RunInstaller. When the runInstaller starts, we select "Upgrade ...."
Once the upgrade is completed, we check init.ora parameters (especially, old home references), environment variables, invalid objects and v$version to ensure that our upgrade is complete.
Again, runInstaller (upgrade option) both installs 11.2.0.4 software and upgrades the chosen databases in one go.
At this point, if we want to upgrade more databases from the same home, we just execute dbua from the new home and that's it.

16)
*As a post uprade step, we update the Enterprise Manager configuration. The steps are as  follows;
Log in to dbconsole or gridconsole.
Navigate to the Cluster tab.
Click Monitoring Configuration
Update the value for Oracle Home with the new Grid home path.

17)
*As db post upgrade instructions, we should take the following actions.
We can review Upgrade Guide for these.(https://docs.oracle.com/cd/E11882_01/server.112/e23633/toc.htm)

Upgrading the Recovery Catalog After Upgrading Oracle Database
Upgrading the Time Zone File Version After Upgrading Oracle Database
Upgrading Statistics Tables Created by the DBMS_STATS Package After Upgrading Oracle Database
Upgrading Externally Authenticated SSL Users After Upgrading Oracle Database
Installing Oracle Text Supplied Knowledge Bases After Upgrading Oracle Database
Updating Your Oracle Application Express Configuration After Upgrading Oracle Database
Configuring Fine-Grained Access to External Network Services After Upgrading Oracle Database
Enabling Oracle Database Vault and Revoking the DV_PATCH_ADMIN Role After Upgrading Oracle Database

*Also, as db post upgrade instruction, we should take a look at the recommended tasks below...

Back Up the Database
Reset Passwords to Enforce Case-Sensitivity
Understand Changes with Oracle Grid Infrastructure
Understand Oracle ASM and Oracle Grid Infrastructure Installation and Upgrade
Add New Features as Appropriate
Develop New Administrative Procedures as Needed
Set Threshold Values for Tablespace Alerts
Migrate From Rollback Segments to Automatic Undo Mode
Configure Oracle Data Guard Broker
Migrate Tables from the LONG Data Type to the LOB Data Type
Test the Upgraded Production Database
Back Up the Database

                      --GI PSU apply section--
18)
Upgrade opatch using patch 6880880 (upgrade both GRID and RDBMS opatch)

19)
Download GI PSU:
Patch 24436338: GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.161018 (OCT2016)
11.2.0.4.161018 (Oct 2016) Grid Infrastructure Patch Set Update (GI PSU) (Doc ID 24436338.8)

20)
Reference: Readme of patch 24436338.

Create a ocm response file (ocm.rsp) -- My Oracle Support Document 966023.1 How To Create An OCM Response File For Opatch Silent Installation.

Apply PSU using opatch auto option:

root user > opatch auto <UNZIPPED_PATCH_LOCATION>/24436338 -ocmrf <ocm response file>

21)
Load Sql side of PSU:

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql