RMAN-- Commvault restore, controlfile autobackup and a little more especially about the controlfile

In this post, I will give a quick overview of Commvault-Rman based database cloning..
This post is based on a customer site story, where we wanted to implement Commvault based Rman restores and even if it was not our duty or responsibility at all, we still needed to get our hands dirty with this 3rd party tool . (3rd party from Oracle Site)
Basically, Commvault uses rman to do online backups and also the restores of this backups.
Restoring a backup a to different server, can be considered as cloning and here in this post, we will talk about that.
Ofcourse a Commvault admin is also required to administrate the Commvault side and to initiate backups and restore from there, but  what needed in the database side are as follows;

• init.ora must be in place.
• commvault seees the database from oratab , so the database entry must be there in oratab.
• Database should be opened in nomount mode for the commvault discovery.
• If controlfile is not supposed to restored by Commvault , then Database should be opened in mount mode before the commvault restore job.
• To be able restore rman wants an autobackup controlfile why?

With a control file autobackup, RMAN can recover the database even if the current control file, recovery catalog, and server parameter file are inaccessible. Because the path used to store the autobackup follows a well-known format, RMAN can search for and restore the server parameter file from that autobackup. Also -> If you have configured control file autobackups, then the server parameter file is backed up with the control file whenever an autobackup is taken. 

However, Rman does not want the autobackup of controlfile, but we make rman to want it. We tell rman to use the autobackup for controlfile restores with the command "restore controlfile from autobackup", and when we give that command, rman just tries to do the spfile and controlfile restore using the autobackups.

Commvault for instance; does use the "restore controlfile from autobackup" command , and because of that rman wants autobackups in place to be able to restore the controlfie.

Well, alternatively, we can use a a traditional controlfile backup(non autobackup) by specifying its backup piece (restore controlfile from '/BACKUP/09I594QI_1_1';) as well. Check: How To Restore Controlfile From A Backupset Without A Catalog Or Autobackup (Doc ID 403883.1)

When we are talking about the backups of controlfile, let's give some basic but important  information about the controlfiles.

Controlfiles contain information about the database, redolog files and other files that are required open an Oracle database.
All the structural database changes are stored in the controlfile (adding a datafile, renaming a datafile, dropping a database , or redolog files etc)
Without controlfile, we can't open the database, and that's why in the first place, we need to have a controlfile to provision our clone database environments.
The following information is stored in a Controlfile.

The database name 
The timestamp of database creation
The names and locations of associated datafiles and redo log files
Tablespace information
Datafile offline ranges
The log history
Archived log information
Backup set and backup piece information
Backup datafile and redo log information
Datafile copy information
The current log sequence number
Checkpoint information

Some of the information types in the controlfile may be overwritten when there is no space left in their region. 

These types of information are listed  below; 

Note that, the parameter named CONTROL_FILE_RECORD_KEEP_TIME plays a big role in this overwriting, as it specifies the minimum number of days before a reusable record in the control file can be reused. In the event a new record needs to be added to a reusable section and the oldest record has not aged enough, the record section expands. If this parameter is set to 0, then reusable sections never expand, and records are reused as needed.

Note that, the size of the database control file growth may depend on the number of: Backups taken and stored in the controlfile, the count of archived redo logs that our database generates, the number of days that the information is configured to be stored in the control file .

LOG HISTORY

OFFLINE RANGE

ARCHIVED LOG

BACKUP SET

BACKUP PIECE

BACKUP DATAFILE

BACKUP REDOLOG

DATAFILE COPY

BACKUP CORRUPTION

COPY CORRUPTION

DELETED OBJECT

PROXY COP

The other information types which are not listed above, such as the database name or the data file names can not be overwritten, so when there is a shortage on their region, the controlfile basically expands for storing the new records (such as when adding a new datafile and when there is no space left in the related area of the controlfile)

When there is no controlfile, or there is no autobackup or normal backup of controlfile , the controlfile can be created using CREATE CONTROLFILE command. However, this command will only a create the controlfile and will not load it with any information like archivelogs, deleted object, log history and so on. Similarly, using the "alter database backup controlfile to trace" command, we can create a controlfile creation statement for ourselves. This might be used in cloning. That is, we can restore our database without our controlfile and issue the command backup controlfile to trace in our source environment and then execute the created controlfile creation script in our target environment to have a brand new controlfile for it, but the same type of information will not be captured using this controlfile creation method. 

The I/O(read write) that is done from or to the controlfile, is done through the PGA.

Processes(foreground and background) write to the controlfile and when they write , they wait in Controlfile file parallel write wait.

The reads or done using the control file sequential read event, and the wait occurs here is the control file sequential read wait.

Control file is read in incidents such as when making a backup controlfiles, sharing info from the controlfile between instances (RAC), so in other words; the controlfile sequential read will be there for the admin activities.

These IOs are different than the DB block I/Os and they are done with the physical  block size of the underlying OS (For Linux it is 512 bytes)

Controlfiles are also written in single writes sometimes.. In those times, the wait event Control file single write wait occurs. This wait is signaled while the control file's shared information is written to disk.

These operations are under the control of Controlfile Enqueue (CF)

So when we see a high CF in our database, then there is a problem with the Controlfile transactions. The IO subsystem may be operating very slowly, or we may produce a high activity that will cause the controlfile is continously and aggressively updated.(such as producing lots lots lots of archivelogs)

Here are some recorded cause for that:

Very slow I/O subsystem where the Control files are stored. 

Frequent log switching - redo logs too small or insufficient logs 

Async I/O issue or multiple db_writers, you can't use both of them, back out one.

OS / Hardware issues

Why did I get into these details? I don't know:) maybe I wanted to give some solutions for the cloning problems that may arise because of lacking the controlfile backups..

Anyways, let's get back to our main topic:

• db_create_file_dest should be set to the new path, that becausee: "ORA-01119 error in creating file +data" (415579.1 )

How commvault does the restore, an example:

-->
It connects with rman target / to the target environment, and then restore the controlfile(from autobackup) , database and lastly recovers the database and opens it in resetlogs mode.

run {

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

restore controlfile ;

sql 'alter database mount';

}

exit;

RESTORE & RECOVER the DATABASE: 

run {

set newname for database to '/u02/ERMAN/Data/%U';

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

allocate channel ch2 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)"

TRACE 0;

allocate channel ch3 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch3)"

TRACE 0;

allocate channel ch4 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch4)"

TRACE 0;

allocate channel ch5 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch5)"

TRACE 0;

## send " -jm 14 -a 2:0 -cl 2320 -ins 234 -at 0 -j 309830 -bal 0 -rcp 0 -ms 5 -p 2 -PREVIEW";

restore database  until time = "TO_DATE('06/22/2016 01:25:42','MM/DD/YYYY HH24:MI:SS')" ;

switch datafile all;

}

exit;

run {

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

allocate channel ch2 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)"

...

.....

.....

recover database  until time = "TO_DATE('06/22/2016 01:25:42','MM/DD/YYYY HH24:MI:SS')" ;

alter database open resetlogs;

}

exit;