Exadata -- Image & GRID 12.2 upgrade

You may remember my article on upgrading Exadata software versions. ->

Exadata Patching-- Upgrading Exadata Software versions / Image upgrade

This time, I 'm extending this upgrade related topic.
So, in this post, I 'm writing about Exadata Image upgrade + 12.2 GRID infrastructure upgrade.

Well... Recently we needed to upgrade Exadata software and GRID infrastructure versions of an Exadata environment.

We divided this work into 2 parts. First we upgraded Exadata images and then we upgraded the GRID version.

Both these upgrades were rolling upgrades. So the databases remained working during these upgrade activities.

Let's take a look at how we do these upgrades.

Exadata Images Upgrades:

We upgraded the image version of a production Exadata envrionment from 12.1.2.1.2.150617.1 to 12.2.1.1.4.171128. We did this work by executing 3 the main processes, given below;

• Analysis and gathering info about the environment.
• Pre-check
• Upgrading the Images in order of ->

• Exadata Storage Servers(Cell nodes) 
• Infiniband Switches
• Compute Nodes (Database nodes)

So, we execute the 3 main phases above and while executing these phases, we actually take the following 8 actions;

1) Gathering info and controlling the current environment :

Image Info, DB Home & GRID Home patch levels opatch lsinventory outputs, SSH equivalency  check , ASM diskgroup repair times check, NFS shares, crontab outputs, .bash_profile contents, spfile/pfile backups, controlfile traces

Approx. duration : 3 hours (done before the operation day)
2) Running the Exack:

Downloading the up-to-date exachk and running it with the -a argument.
After running the exachk -> analyzing its output and taking the necessary actions if there are any.

Approx. duration : 2 hours (done before the operation day) 

 3) Downloading the new Exadata images and uploading it to the nodes.

Approx. duration : 2 hours (done before the operation day)

4) Creating the necessary group files for the Patchmgr . (cell_group, dbs_group, ibswitches.lst)

Approx. duration : 0.5 hours (done before the operation day)

5) Running Patchmgr precheck. After analyzing its output-> taking the necessary actions  (if there are any) For ex: if there are 3rd party rpms, we may decide to remove them manually before the upgrade.

Approx. duration : 0.5 hours (done before the operation day)

6) Running Patchmgr and upgrading the images. (we do the upgrade in rolling mode)

Before running the patchmgr, we kill all the ILOM sessions.. (active ILOM session may increase the duration of the upgrade)

Note: Upgrade is done in the following order;

Exadata Storage Servers(Cell nodes)  (1 hour per node)
Infiniband Switches (1 hour per switch )
Compute Nodes (Database nodes) ( 1.5 hours per node)
  
7) As the post upgrade actions; reconfiguring NFS & crontabs. Also reinstalling the 3rd party rpms (if removed before the upgrade)

Approx. duration : 0.5 hours

8) Post check: checking the databases, their connectivity and alert log files..
Note that : we also run exachk once again and analyze its output to ensure that everything is fine after the Image upgrade.

Approx. duration : 1 hour

GRID 12.2 Upgrade:

As for  the GRID 12.2 upgrade, we basically follow the MOS document below;

"12.2 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running 11.2.0.3 and later on Oracle Linux (Doc ID 2111010.1)"

First, we analyze our environment in conjuction with following the document above to determine the patches and prereq patches required for our environment.

Here is the list of patches that we used during our last GRID 12.2 upgrade work;

GI JAN 2018 RELEASE UPDATE 12.2.0.1.180116 Patch 27100009 
Oracle Database 12c Release 2 Grid Infrastructure (12.2.0.1.0) for Linux x86-64 V840012-01.zip 
OPatch 12.2.0.1.0 for Linux x86-64 Patch 6880880 
Opatch 11.2.0.0.0 for Linux x86-64 Patch 6880880 
CSSD : DUPLICATE RESPONSE IN GROUP DATA UPDATE Patch 21255373

Once all the required files/patches are in place, we do the upgrade GRID by following the steps below;

1. Creating the new GRID Home directories.
2. Unzipping the new GRID software into the relevant directories.
3. Unzipping up-to-date opatch and GRID patches.
4. If needed, configuring the ssh equivalencies.
5. Running runcluvfy.sh and doing the cluster verification. (In case of an error, we fix the error and rerun it)
6. Patching our current GRID home with the prereq patches (in our last upgrade work, we needed to apply the patch 21255373)
7. Increasing the sga_max_size and sga_target values of the ASM instances.
8. Configuring VNC (we do the actual upgrade using VNC)
9. Starting the GRID upgrade using the unzipped new GRID Software (on VNC)
10. Running the roolUpgrade.sh on all the nodes.
11. Controlling/Checking the cluster services.
12. Configuring the ASM compatibility levels.
13. Lastly, as a post upgrade step, we add the new GRID home in to the inventory.

As you may guess, the most critical steps in the list above, are step 9 and step 10..  (as the actual upgrade is done while executing those steps)

Approx Duration : 4 hours.. (for a 2 node Exadata GRID upgrade)

that's it :) I hope you will find this blog post useful :)

Exadata Cloud Machine -- first look, quick info and important facts

Recently started an ECM (Exadata Cloud Machine) migration project, or maybe I should say an ECC (Exadata Cloud at Customer) migration project.

This is a big migration project, including migration of the Core Banking databases.
It is a long run, but it is very enjoyable.
We have 2 ECCs to migrate to..

Finally, last week, initial deployment of the machines was completed by Oracle.
This week, we connected to the machines and started to play with them :)

I think, I will write several blog posts about this new toys in the coming months, but here is a quick info and some important facts about the ECC environments.

First of all, ECC is an Exadata :) Exadata hardware + Exadata software..

Technically, it is such a virtualized Exadata RAC environment, that we(consultants) and customers can not access its cells, Iloms, switches and hypervisor.

• It is a Cloud Machine, but it is behind the firewall of the customer.
• It has a Cloud Control Plane application , a GUI to manage the database services and this application is hosted in OCC (Oracle Cloud Machine), which can be thought as the satellite of ECC. 
• We do lots of stuff using this GUI. Database Service Creation (11.2.0.4, 12c, 18c) , Patching and etc..

• Database service creations and Grid operations are automatized. According to version of the database created using GUI, GRID is automatically created.. For ex: If we create a 12.2 database and if it is the first 12.2 database that we create in ECC, GRID 12.2 is also automatically created..(cloud operations) For ex: If we have GRID 12.1 and some DB 12.1 residing in ECC and if we want to create our first and new 12.2 Database, then GRID is automatically upgraded to 12.2 as well.
• The minimum supported DB version in ECC is 11.2.0.4. So we need to have our db compatible parameter set to 11.2.0.4 (mimum) in order to have a database on ECC -- this is related with the migration operations.
• We can install Enterprise Manager agents on ECC. So our customer can manage and monitor ECC nodes and databases using its current Enterprise Manager Cloud or Grid control.
• ECCs are virtualized. Only Oracle can access the hypervisor level. We and the customer can only access to the DOMu. In the DOMu RAC nodes , we and the customer do the OS administration.. Backups, patching, rpm installation and everything.. Customer is responsible for the DOMu machines, where GRID and Databases run on. Customer has root access for the DOMu nodes. (This means DB administration + OS administration is still continuing :))
• So customer can't access Cell servers, and even ILOM consoles..
• Administration for everyting that resides below the DomU layer, is done by Oracle
• Responsibility for everything that resides below DomU layer, is on Oracle.
• Currently, for every physical node, we have a VM node. For ex: If we have a 1/2 ECC. We have 4 physical nodes and 4 VMs.. (DOM u nodes) -- 1 to 1.
• We can create RAC multi-node or single node databases on ECC.
• We can also create databases manually on ECC. (without using the GUI).. Using scripts or runInstaller, everything can be done just like the old days. (as long as versions are compatible with ECC)
• If we create a 12C database using GUI, it comes as Pluggable.. So if we want to have a non-PDP 12C Database, we need to create it manually.
• Customer can connect to the RAC nodes (DOMu nodes) using SSH keys. (without password).. This is a must.
• Customer can install backup agents to ECC.. So without changing the current backup method and technology, customer can backup the databases running on ECC.
• There is no external infiniband connection to ECC.. External connection can be max 10Gbit.
• Enterprise Manager Express comes with ECC. We have direct links to Enterprise Manager Express in the Control plane.
• IORM is also available on GUI. Using GUI, we can do all the IORM configuration.. 
• In ECC, we can use In-memory and Active Dataguard .. Actually, we can use all the database options without paying any licenses.
• If we create 12.2 Databases, they are created with TDE.. So TDE is a must for 12.2 databases on ECC.
• However, we are not required to use TDE, if we are using 11G databases on ECC.
• The ASM diskgroups on ECC are High redundancy Diskgroups. This is the default an can not be changed!
• Exadata Image upgrade operations on the ECC environments are done by Oracle.

That'a all for now :) In my next blog post, I will show you how we can create database services on ECC. (using GUI)

RDBMS -- Be careful while activating a standby database (especially in cascaded configurations)

Recently, a customer reported an issue about a standby database, which was out-of-sync with the primary. This standby database was the endpoint of a cascaded configuration.

The cascaded dataguard configuration in that customer environment, was as follows;

Primary -> Standby1 -> Stanby2

So, the customer's requirement was to activate standby1 and continue applying redologs of primary directly to the standby2.

However; while activating, actually after activating the standby database named standby1, the customer accidentally made Standby2 to apply the redologs which were generated by standby1.

When standby2 received and applied the archivelogs from standby1, standby2 became a new standby database for standby1, and it became out of sync with the initial production database.

Interesting, right?

In order to bring the database Standby2 in sync with its original primary database, we did the following;

We used flashback database option to flashback the standby2 to the point before it applied the archivelogs from standby1

Then, we deleted the archivelogs received from standby1 and make sure that standby1 is not sending any archive logs to standby2 until it is converted back to physical standby. ( this way we could ensure that standby2 is applying the redologs only from the production database.)

Note that, if we didn't have the possibility to use the flashback option, we would have to recreate the standy database named standby2...

So, be careful while playing with the dataguard configuration.. Especially in cascaded environments... First check the configuration, then take the action.. In this real life case, the dataguard configuration was from primary to standby 1 and from standby 1 to standby 2.. So when standby1 became activated, that path "from standby1 to standby2" worked, and standby1 became the new primary for standby2.. Incarnation changed and standby2 became out-of-sync with the original primary. 

In order to prevent this to be happen, the dataguard flow(configuration) should have been changed before activating the standby1.

GTECH -- Summer School -- Oracle Database & Cloud & EBS for newly graduates

Once in a year, we as GTech provide training for newly graduated engineers.

In this training, we teach Sql, PL/SQL, Oracle Database & Cloud, EBS, OBIEE, BigData, ETL and more.

This year, I was the lecturer for "Database and Cloud".

Actually, I extended the lessons a little bit by explaining the EBS System Administration Fundamentals, as well. :)

The students of the classes were so curios about databases and actually Oracle in general..

It was a honour for me to present "the introduction to Oracle Database", to explain the "Cloud terms" ( including Cloud-at-customer model) and to explain the "EBS architecture".

I tried to shed a light on the important topics like Oracle Database Server Architecture, Oracle Database Process Architecture,  background processes, High availability configurations and so on..

The list of topics covered in the training was as follows;

• Introduction to RDBMS
• Introduction to Oracle
• Architecture (Oracle)
• Installation (Oracle)
• DBA role & DBA tools
• Introduction to Cloud
• APPS DBA role & EBS System Administration (EBS 12.2)

While explaing these topics, I tried to share real life stories all the time..  Tried to teach them the basics of Oracle, but I also dived deep when required.

The participants asked lots of good technical questions and these made our lessons more entertaining :)

The training for Database & Cloud lasted 3 days.

While, preparing the slides for the presentations that I have used in the training, I also wrote an exam for the students..

At the end of the training, we also gave this written examination to the participiants. (35 questions )

It was a pleasure for me to teach Oracle in GTech Academy ( GTech -- Oracle University Partner)

I hope, It was useful for these guys..
I also hope I will see them (at least some of them) as successful DBAs one day :)

Following is the picture of our class..  A good memory :)

RDBMS -- datapump -- 3 interesting problems and 3 things to consider

Recently dealed with some critical and interesting datapump problems.
These problems were basically realated with corrupted dump files, uncompleted dump files, missing Master tables and performance problems during a partitioned table import.

Let's start with the corrupted dump files..

This is very clear.. If you have corrupt dumpfile in your datapump backupset, then you won't be able to correct them.. At least Oracle doesn't have a method or tool to correct them..
There are 1 or 2 third party tools, but I didn't test them. Those tools doesn't support compressed dumpfile , anyways..
On the other hand, you can still import even if you have corrupted dumpfiles in your dumpfile backupset..
Of course, you will get fatal errors during your import.. That is unavoidable, but you can restart the import process and resume your import by skipping those errors.. At the end, you may have your data imported in your database partially..

The other interesting problem was related with the uncompleted dump files..

In our case, someone killed the expdp process during the export.. One day after that, we tried to import the dumpfiles , but we saw the errors .. Some of the dumpfiles were not there.. Master table was not there.. Master table was actually in the database, but it wasn't in any of the files in the export backup set. So we couldn't import the tables.. Datapump was complaining about the master table..

Actually, all the critical tables were already exported before someone killed the expdp.. So, what we needed to do was to find a way to import these dumpfiles.. However, impdp was not doing anything.. It was complaining about the Master table..

What we did was interesting.. We resumed the expdp job.. (it was killed 24 hours ago) So expdp continued, finished all the other tables in the list and exported the MASTER TABLE (expdp exports this table at the end) and finished its work..

Of course, the tables exported in this run were not from the same timestamp as the other ones, but at the end we got our MASTER TABLE included in the backup set and could import those critical tables using this backup set.

The last thing that I want to share with you is related with impdp performance.. Impdp can't import parallel when a partition table is already created in the database. At least, this is the case for 11gR2.

So, if you are going to import a partitioned table using parallel argument, I suggest you to let impdp create that partitioned table.. If you create the table beforehand, you will see impdp running serially although you pass the parallel argument to it. This is based on a real life case.. Hope you 'll find it useful.

EBS 12.2 -- Things that can be done for debugging WLS Managed Server performance and stability

Weblogic (a FMW component) is an important component in EBS 12.2.

FMW plays an important role in EBS 12.2 , as EBS 12.2 delivers Http services ,OAF and forms services through FMW.

That's why, time-to-time, a real diagnostics is required , especially for analyzing weird performance and hang issues on EBS OAF pages.

In this post, I will go through the things that can be done for debugging the Weblogic side, especially the managed server performance and stability.

Of course, when dealing with weblogic inside EBS, we directly check the managed server logs, admin server logs, Heap size configurations, managed server counts (whether it is aligned with the concurrent user count or not), connection pool limits and so on. On the other hand; these debugging activities that I will give you in this blog post, are a little more advanced. It is also needless to say that, these debugging activities require advanced skills on Weblogic and EBS administration.

Note that,I  won't give the full instructions for these diagnostics activities. In other words; I will explain them very briefly.
Also note that, these activities are not fully documented, that's why they are not fully supported --the risk is yours.

Garbage Collector Debug: for getting a more elaborated GC info and checking the time passed for each GC event.

We can get this debug info using -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps, jvm arguments.

Running technology stack inventory report: to collect the list of patches applied to all middle tier homes (besides Weblogic).. The output of this script may be used to identify unapplied performance patches.

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=<appspassword> -outfile=$APPLTMP/Report_App_Inventory.html

Diagnostic Connection Leaks: for getting leak connection-related diag info, we use "How To Detect a Connection Leak Using Diagnostic JDBC Dumps (Doc ID 1502054.1)"

Create heap dump & thread dumps: Especially for getting info about an outofmemory problems.

--review ->  How To Create a Java HeapDump in E-Business Suite (Doc ID 835909.1)

You add the following parameters to JVM startup parameters -> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>

Then you restart the relevant managed server and wait an outofmemory error to occur.
Note that a heap dump is written only on the first OutOfMemoryError.
You can also generate the heap dump by using jmap or jconsole..

Examples:

jmap -dump:format=b,file=<filename> <jvm_pid>

Launch jconsole --> attach to the java process --> and invoke operation dumpHeap() on HotSpotDiagnostic MBean

Note that, the generated trace file can be analyzed using 3rd party tools like Solaris tool jhat, IBM HeapAnalyzer, Eclipse Memory Analyze.

Creating Java Stack Trace : Especially for getting info about hanging, blocking, spinning processes. These diagnostics are done by using the necessary command line arguments in server start arguments section of the related managed server (using WLS console)

The related arguments are specified using the server start arguments section ->

Connect WLS console
Navigate to servers under EBS_domain_<SID> environment
Click on the managed server (ex:oacore_server1)
Click  on  Lock & Edit in Change Center
Click on Server start

Edit arguments (such as  -XX:HeapDumpOnCtrlBreak)

So once the necessary argument is given to a managed server, we restart the managed server and use  OS kill commands to generate these dumps.. (ex: kill -3 os_pid -- kill -3 - SIGQUIT - like ctrl-C but with a core dump)

--review -> How to create a Java stack trace on e-Business Suite ? (Doc ID 833913.1)

Once the error is reproduced we review the FMW logs -> 12.2 Ebusiness Suite - Collecting Fusion Middleware Log Files Note 1362900.1.

Consider increasing Stuck Thread timeouts : in case we have stuck threads.. We can increase the Stuch Thread Max Time using Weblogic console.

Connection Debugging: For JDBC connection debugging, we use Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics (Doc ID 1940996.1).

DB level trace: We enable trace at db level -> "alter system set events '10046 trace name context forever, level 12';"

We reproduce the issue and turn it off "alter system set events '10046 trace name context off';"

We check the traces (find the relevant trace using  "grep MODULE *.trc  and/or "grep ACTION   *.trc"

Tracing Managed Server sessions :  For diagnosing  managed server related db activity, and for diagnosing inactive (not closed) managed server sessions.

Reference: On E-Business Suite 12.2 V$SESSION.PROCESS incorrectly reports EBS Client Process ID as '1234' (Doc ID 1958352.1)

Connect to Weblogic Console and then do the following;
Services > Data Sources > EBSDataSource > Configuration > Connection Pool
Set "System Property" as below

v$session.program=weblogic.Name [Take note of the initial value one is changing as one will need to reset it once the fix is delivered and applied.]

Lastly we restart oacore managed servers and monitor the database using a query like;

SQL> select program, process, machine, sql_id, status, last_call_et from v$session where program like 'oacore_server%';

EXADATA -- Unique Articles Worth Reading ( imaging, upgrade, installation, configuration and so on)

Nowadays, my context is completely switched. That is, I have started to work more on Exadata and ECM/OCM migrations.. As a result of that, I produce more content on these areas.

Till the last month, I was more focused on Exadata.. But nowadays, I m not only focused on Exadata, but also Exadata Clould machines and cloud migration projects.

Of course, I documented the critical things that we have done on Exadata machines one by one and produced the following articles for sharing with you.

I find these articles unique, as they are mostly written for Exadata specific patching actions, installations and configurations.. So they are worth the time and effort to read.

There are 6 of them.

You can find the links for these articles below;

Links to blog posts:

• RAC / EXADATA-- switching Scan names -- removing the need for modifying the connection strings after a migration

• Exadata -- Cisco Switch Firmware upgrade

• Exadata X7-2 -- Installation & applyElasticConfig.sh

• Exadata Patching-- Upgrading Exadata Software versions / Image upgrade

• Exadata -- Elastic config, adding one compute and one cell node to Exadata X4-2

• Exadata -- How to Connect Oracle Exadata to 10G Networks Using SFP Modules

• Exadata -- Reimaging Oracle Exadata machines

Exadata -- Cisco Switch Firmware upgrade

In this post, I will explain upgrading the firmware of the Cisco Switch, which is delivered --built-in-- with the Exadata machines.
For explaning the process, I will go through a real life case, which was done in an Exadata X3-2 environment.

The Cisco switch version that I use for demonstrating this upgrade is Catalyst 4948e, which is the ethernet switch delivered with Exadata X3-2 machines. (In Exadata X7, we see Cisco Nexus switches..)

In Exadata environments, these cisco switches are used for systems management net interfaces access only. (ethernet based management network, ssh connection, ILOM and so on.)

So, during such an upgrade, no production traffic is affected, just consoles and node management...

The requirement for upgrading the firmware of these switches may arise after a security scan, which is usually performed regularly by the security teams in customer environments (enterprise customers..) 

Following is a list of vulnerabilities that were discovered in a customer environment.. These vulnerabilities were discovered on the cisco switch which was delivered with the Exadata X3-2. (cisco firmware version was : cat4500e-IPBASEK9-M Version 15.1(1)SG)

• Cisco IOS Cluster Management Protocol Telnet Option Handling 
• Cisco IOS IKEv2 Fragmentation DoS 
• Cisco IOS IKEv1 Fragmentation DoS 
• Cisco IOS Software DHCP Version 6 Server Denial of Service Vulnerability 
• Cisco IOS Software DHCP Denial of Service Vulnerability 
• Cisco IOS EnergyWise DoS 
• Cisco IOS Software Internet Key Exchange Version 2 (IKEv2) Denial of Service 
• Cisco IOS Software Smart Install Denial of Service Vulnerability 
• Cisco IOS Software RSVP DoS 
• Cisco IOS Multicast Routing Multiple DoS 
• Cisco IOS Multiple OpenSSL Vulnerabilities 
• Cisco IOS Software TFTP DoS 
• Cisco IOS Software DHCP Denial of Service Vulnerability 

These vulnerabilites are fixed in cisco firmware version "cat4500e-ipbasek9-mz.152-2.E8"  and here is the list of things that we did for upgrading this 15.2.2E8 target release;

• First, we connect to the cisco switch using telnet from db node 1 and check the current firware version;

[oracle@exanode1~]$ telnet <cisco_switch_ip_address>

exaswc0>show version

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 15.1(1)SG, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2012 by Cisco Systems, Inc.

Compiled Sun 15-Apr-12 02:55 by prod_rel_team

ROM: 12.2(44r)SG11

fbadmswc0 uptime is 4 years, 37 weeks, 2 days, 23 hours, 55 minutes

System returned to ROM by power-on

System restarted at 15:15:39 GDT Tue Jul 2 2013

System image file is "bootflash:cat4500e-ipbasek9-mz.151-1.SG.bin"

Hobgoblin Revision 21, Fortooine Revision 1.40

• Then, we download the new switch software from cisco -

https://software.cisco.com/download/release.html?mdfid=283027810&softwareid=280805680&release=15.2.2E8&flowid=3592

(Choose "IP Base Image" line from 15.2.2E8(MD) version.

File name : cat4500e-ipbasek9-tar.152-2.E8.tar)

• After downloading the new switch software , we create a tftp server and , and put the new cisco software bin (which comes out from the tar file) to a tftp directory like /tftpboot/switch_image.

[root@acs-vmmachine~]# mkdir /tftpboot/switch_image

[root@acs-vmmachine ~]# chmod 777 /tftpboot/switch_image/

[root@acs-vmmachine ~]# ls -l /tftpboot/switch_image/

total 0

-rwxrwxrwx 1 root root 0 Mar 19 09:16 new_image.bin

• Then again, in cisco switch; we list the files in the bootflash directory and check its size;

exaswc0>enable

Password: 

exaswc0#dir bootflash:

Directory of bootflash:/

    6  -rw-    25213107  Mar 19 2013 14:46:08 +04:00  cat4500e-ipbase-mz.150-2.SG2.bin

    7  -rw-    32288280   Jun 5 2013 20:04:54 +04:00  cat4500e-ipbasek9-mz.151-1.SG.bin

  
exaswc0>show file systems 
File Systems: 

Size(b) Free(b) Type Flags Prefixes 
* 60817408 45204152 flash rw bootflash:   --------> There are about 45 MB free space in bootflash. (Min 20 MB required.)

• We configure our cisco to boot from a specific firmware file. 

exaswc0#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
exaswc0(config)#no boot system
exaswc0(config)#boot system bootflash:cat4500e-ipbasek9-mz.151-1.SG.bin (current)

• Then, we save the running config and name it with the suffix "before-upgrade"

exaswc0#copy running-config startup-config all 

exaswc0#copy running-config bootflash:cat4500e-ipbasek9-mz.151-1.SG-before-upgrade

• Next, we copy this file to our tftp server. (for backup) -- we answer the prompts for the tftp-server name and the destination filename..

exaswc0#copy bootflash:cisco4948-ip-confg-before-upgrade tftp:

• After copying our running config to our tftp-server (installed earlier into our client machine), we copy the new image from tftp-server to our cisco switch by executing the following command on cisco.

copy tftp: bootflash:
Address or name of remote host []? acs-vmmachine
Source filename []? switch_image/new_image.bin
Destination filename [new_image.bin]?cat4500e-ipbasek9-mz.152-2.E8.bin

...

....

exaswc0# 

exaswc0# dir bootflash: 

Directory of bootflash:/

    6  -rw-    25213107  Mar 19 2013 14:46:08 +04:00  cat4500e-ipbase-mz.150-2.SG2.bin

    7  -rw-    32288280   Jun 5 2013 20:04:54 +04:00  cat4500e-ipbasek9-mz.151-1.SG.bin

25  -rw-    38791882  Mar 20 2018 15:24:24 +04:00  cat4500e-ipbasek9-mz.152-2.E8.bin -- this is the firmware that we are upgrading to.

• We verify the new image file;

exaswc0-ip#verify bootflash:cat4500e-ipbasek9-mz.152-2.E8.bin
File system hash verification successful.

• After our new image file is verified, we configure our cisco switch boot system to our new image bin and save the configuration into nvram.

exaswc0#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
exaswc0(config)#config-register 0x2102
exaswc0(config)#no boot system
exaswc0(config)#boot system bootflash:cat4500e-ipbasek9-mz.152-2.E8.bin
exaswc0(config)#
exaswc0(config)# (type <control-z> here to end)
exaswc0#show run | include boot
boot-start-marker
boot system bootflash:cat4500e-ipbasek9-mz.152-2.E8.bin
boot-end-marker

exaswc0# copy running-config startup-config all
exaswc0#write memory 

Note that: 0x2102 instructs the boot process to ignore any breaks, sets baudrate to 9600 and boots into ROM if the main boot process fails for some reason.

• Lastly, we boot our cisco switch with the new firmware and save running config.

exaswc0# reload 

exaswc0-#copy running-config startup-config all 

exaswc0#copy running-config bootflash:cat4500e-ipbasek9-mz.152-2.E8-after-upgrade

exaswc0#write memory 

• At this point, we can continue enabling SSH access and disabling telnet access. (although, this action is optional, it is highly recommended. Check the below references for the instructions.

References:

Upgrading firmware / Configuring SSH on Cisco Catalyst 4948 Ethernet Switch (Doc ID 1415044.1)
How To Update Exadata Management Network Switch Firmware (Doc ID 1593004.1)

RDBMS -- Interesting error on Duplicate From Active Database -> ORA-19845, ORA-17628, ORA-19571, ORA-19660

Recently encountered an interesting problem in a Rman Duplicate session.
We were trying to duplicate a database from active database using rman and although, we did everything fine, we ended up with the following error stack.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 05/03/2018 00:15:53
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script

ORA-19845: error in backupArchivedLog while communicating with remote database server
ORA-17628: Oracle error 19571 returned by remote Oracle server
ORA-19571: RECID STAMP not found in control file
ORA-19660: some files in the backup set could not be verified
ORA-19662: archived log thread 1 sequence 7643 could not be verified
ORA-19845: error in backupArchivedLog while communicating with remote database server
ORA-17628: Oracle error 19571 returned by remote Oracle server
ORA-19571: RECID STAMP not found in control file

As the error ORA-17628 suggests, RMAN couldn't comminucate with the remote server.

The remote server that is mentioned here was actually the auxiliary instance , which was the new database instance that we were creating from the active database.

This problem was closely related with the service_names parameter of this auxiliary.

As you may already know, when we duplicate from active database, rman restores the spfile from the source instance and update it according to the parameter settings that we used in our duplicate command..

In case of the service_names and other similar type of parameters, rman restores the spfile and updates it according to the value that we set for the "SPFILE PARAMETER VALUE_CONVERT", that we use in our rman duplicate command.

However; what we discovered in this case was, rman couldn't do that update properly.. (at least for the service_names parameter and at least for our case..)

So, although we set the correct value for the SPFILE PARAMETER_VALUE_CONVERT parameter, rman couldn't update the service_names parameter of the auxiliary instance properly.

As a result, we encountered "ORA-17628: Oracle error 19571 returned by remote Oracle server" error during our duplicate session.

I must admit that, this was weird and probably this was probably a bug.

Fortuneatly, we found the workaround.

As for the workaround, we did the following;

• we created an init.ora for the auxiliary and made the changes in init.ora (changes for the desired values)

db_unique_name='ERM'
set db_name='ERM'
set instance_name='ERM1'
set instance_number='1'
set db_create_file_dest='+DATA'
set db_recovery_file_dest_size='40G'
set db_recovery_file_dest='+RECO'
set control_files='+DATA','+RECO'
set db_create_online_log_dest_1='+DATA'
set db_create_online_log_dest_2='+RECO'
set diagnostic_dest='/u01/app/oracle'
set audit_file_dest='/u01/app/oracle/product/12.1.0.2/dbhome_3/rdbms/audit'
set log_archive_dest_1='location=USE_DB_RECOVERY_FILE_DEST'
set log_archive_dest=''
set local_listener=''
set cluster_database='FALSE';

• Then, we connected to the auxililary and created spfile from the pfile

SQL> CREATE SPFILE FROM PFILE='location of destination pfile'; ----this is the pfile created earlier.
SQL> STARTUP NOMOUNT;

• Lastly, we run our duplicate command without SPFILE clause. (without SPFILE PARAMETER_VALUE_CONVERT)

In brief, we set our desired parameters for the auxiliary database in a pfile (init.ora), then created spfile from that init.ora(pfile) and started up the auxiliary database in nomount mode using that spfile..

After that , we run our rman duplicate command  without specifying SPFILE.. parameter.

By doing this; we started up the auxilariy instance with the desired parameters and bypassed the automatic spfile update that is done from source instance to auxiliary by rman ( this automatic update is done when we use SPFILE PARAMETER_VALUE_CONVERT parameter and when the auxiliary instance is started up using pfile)..

This workaround saved the day, so I wanted to share it with you.

Note that: starting up the auxiliary database directly with the spfile (filled with the desired parameters) is actually a good thing to do. So we are considering using this approach in our next duplicate sessions as well.

EBS -- Upgrading EBS 12.1.3 to 12.2 -- the general steps

In this blog post, I want to give you a list which includes the general steps that can be included in the project plan of an EBS 12.1.3 to 12.2 upgrade project.

By taking the following phases and the related steps into account, you may calculate your effort and do your project plan accordingly.

I wanted to use EBS 12.1.3 as the source version because it is a very common version in EBS customer environments. While giving the steps, I also wanted to highlight the teams responsible for completing those steps. (apps DBA team, Functional team, core business users etc..).

Of course, some of these steps like the "upgrade database step" is optional .( if your db release is up-to-date enough.)

Phase 1      

• Upgrade the database on the existing EBS 12.1: apps DBA team      
• Execute a functional test: EBS functional team

Phase 2      

• Install all application pre-upgrade patches: apps DBA team      
• Verify the instance: EBS functional team

Phase 3      

• Execute all functional pre-upgrade tasks including customizations: functional team      
• Perform a full system backup: System and apps DBA team   

 Phase 4      

• Apply localization and 12.2 pre-upgrade patches: apps DBA team      
• Upgrade to 12.2.0: apps DBA team      
• Enable online patching: apps DBA team      
• Apply tech stack patches: apps DBA team      
• Upgrade to 12.2.6/12.2.7: apps DBA team      
• Perform all post-upgrade tasks: apps DBA and functional teams      
• Application function test cases: core business users