Exadata -- Image & GRID 12.2 upgrade

You may remember my article on upgrading Exadata software versions. ->

Exadata Patching-- Upgrading Exadata Software versions / Image upgrade

This time, I 'm extending this upgrade related topic.
So, in this post, I 'm writing about Exadata Image upgrade + 12.2 GRID infrastructure upgrade.

Well... Recently we needed to upgrade Exadata software and GRID infrastructure versions of an Exadata environment.

We divided this work into 2 parts. First we upgraded Exadata images and then we upgraded the GRID version.

Both these upgrades were rolling upgrades. So the databases remained working during these upgrade activities.

Let's take a look at how we do these upgrades.

Exadata Images Upgrades:

We upgraded the image version of a production Exadata envrionment from 12.1.2.1.2.150617.1 to 12.2.1.1.4.171128. We did this work by executing 3 the main processes, given below;

• Analysis and gathering info about the environment.
• Pre-check
• Upgrading the Images in order of ->

• Exadata Storage Servers(Cell nodes) 
• Infiniband Switches
• Compute Nodes (Database nodes)

So, we execute the 3 main phases above and while executing these phases, we actually take the following 8 actions;

1) Gathering info and controlling the current environment :

Image Info, DB Home & GRID Home patch levels opatch lsinventory outputs, SSH equivalency  check , ASM diskgroup repair times check, NFS shares, crontab outputs, .bash_profile contents, spfile/pfile backups, controlfile traces

Approx. duration : 3 hours (done before the operation day)
2) Running the Exack:

Downloading the up-to-date exachk and running it with the -a argument.
After running the exachk -> analyzing its output and taking the necessary actions if there are any.

Approx. duration : 2 hours (done before the operation day) 

 3) Downloading the new Exadata images and uploading it to the nodes.

Approx. duration : 2 hours (done before the operation day)

4) Creating the necessary group files for the Patchmgr . (cell_group, dbs_group, ibswitches.lst)

Approx. duration : 0.5 hours (done before the operation day)

5) Running Patchmgr precheck. After analyzing its output-> taking the necessary actions  (if there are any) For ex: if there are 3rd party rpms, we may decide to remove them manually before the upgrade.

Approx. duration : 0.5 hours (done before the operation day)

6) Running Patchmgr and upgrading the images. (we do the upgrade in rolling mode)

Before running the patchmgr, we kill all the ILOM sessions.. (active ILOM session may increase the duration of the upgrade)

Note: Upgrade is done in the following order;

Exadata Storage Servers(Cell nodes)  (1 hour per node)
Infiniband Switches (1 hour per switch )
Compute Nodes (Database nodes) ( 1.5 hours per node)
  
7) As the post upgrade actions; reconfiguring NFS & crontabs. Also reinstalling the 3rd party rpms (if removed before the upgrade)

Approx. duration : 0.5 hours

8) Post check: checking the databases, their connectivity and alert log files..
Note that : we also run exachk once again and analyze its output to ensure that everything is fine after the Image upgrade.

Approx. duration : 1 hour

GRID 12.2 Upgrade:

As for  the GRID 12.2 upgrade, we basically follow the MOS document below;

"12.2 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running 11.2.0.3 and later on Oracle Linux (Doc ID 2111010.1)"

First, we analyze our environment in conjuction with following the document above to determine the patches and prereq patches required for our environment.

Here is the list of patches that we used during our last GRID 12.2 upgrade work;

GI JAN 2018 RELEASE UPDATE 12.2.0.1.180116 Patch 27100009 
Oracle Database 12c Release 2 Grid Infrastructure (12.2.0.1.0) for Linux x86-64 V840012-01.zip 
OPatch 12.2.0.1.0 for Linux x86-64 Patch 6880880 
Opatch 11.2.0.0.0 for Linux x86-64 Patch 6880880 
CSSD : DUPLICATE RESPONSE IN GROUP DATA UPDATE Patch 21255373

Once all the required files/patches are in place, we do the upgrade GRID by following the steps below;

1. Creating the new GRID Home directories.
2. Unzipping the new GRID software into the relevant directories.
3. Unzipping up-to-date opatch and GRID patches.
4. If needed, configuring the ssh equivalencies.
5. Running runcluvfy.sh and doing the cluster verification. (In case of an error, we fix the error and rerun it)
6. Patching our current GRID home with the prereq patches (in our last upgrade work, we needed to apply the patch 21255373)
7. Increasing the sga_max_size and sga_target values of the ASM instances.
8. Configuring VNC (we do the actual upgrade using VNC)
9. Starting the GRID upgrade using the unzipped new GRID Software (on VNC)
10. Running the roolUpgrade.sh on all the nodes.
11. Controlling/Checking the cluster services.
12. Configuring the ASM compatibility levels.
13. Lastly, as a post upgrade step, we add the new GRID home in to the inventory.

As you may guess, the most critical steps in the list above, are step 9 and step 10..  (as the actual upgrade is done while executing those steps)

Approx Duration : 4 hours.. (for a 2 node Exadata GRID upgrade)

that's it :) I hope you will find this blog post useful :)