RDBMS -- Interesting error on Duplicate From Active Database -> ORA-19845, ORA-17628, ORA-19571, ORA-19660

Recently encountered an interesting problem in a Rman Duplicate session.
We were trying to duplicate a database from active database using rman and although, we did everything fine, we ended up with the following error stack.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 05/03/2018 00:15:53
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script

ORA-19845: error in backupArchivedLog while communicating with remote database server
ORA-17628: Oracle error 19571 returned by remote Oracle server
ORA-19571: RECID STAMP not found in control file
ORA-19660: some files in the backup set could not be verified
ORA-19662: archived log thread 1 sequence 7643 could not be verified
ORA-19845: error in backupArchivedLog while communicating with remote database server
ORA-17628: Oracle error 19571 returned by remote Oracle server
ORA-19571: RECID STAMP not found in control file

As the error ORA-17628 suggests, RMAN couldn't comminucate with the remote server.

The remote server that is mentioned here was actually the auxiliary instance , which was the new database instance that we were creating from the active database.

This problem was closely related with the service_names parameter of this auxiliary.

As you may already know, when we duplicate from active database, rman restores the spfile from the source instance and update it according to the parameter settings that we used in our duplicate command..

In case of the service_names and other similar type of parameters, rman restores the spfile and updates it according to the value that we set for the "SPFILE PARAMETER VALUE_CONVERT", that we use in our rman duplicate command.

However; what we discovered in this case was, rman couldn't do that update properly.. (at least for the service_names parameter and at least for our case..)

So, although we set the correct value for the SPFILE PARAMETER_VALUE_CONVERT parameter, rman couldn't update the service_names parameter of the auxiliary instance properly.

As a result, we encountered "ORA-17628: Oracle error 19571 returned by remote Oracle server" error during our duplicate session.

I must admit that, this was weird and probably this was probably a bug.

Fortuneatly, we found the workaround.

As for the workaround, we did the following;

• we created an init.ora for the auxiliary and made the changes in init.ora (changes for the desired values)

db_unique_name='ERM'
set db_name='ERM'
set instance_name='ERM1'
set instance_number='1'
set db_create_file_dest='+DATA'
set db_recovery_file_dest_size='40G'
set db_recovery_file_dest='+RECO'
set control_files='+DATA','+RECO'
set db_create_online_log_dest_1='+DATA'
set db_create_online_log_dest_2='+RECO'
set diagnostic_dest='/u01/app/oracle'
set audit_file_dest='/u01/app/oracle/product/12.1.0.2/dbhome_3/rdbms/audit'
set log_archive_dest_1='location=USE_DB_RECOVERY_FILE_DEST'
set log_archive_dest=''
set local_listener=''
set cluster_database='FALSE';

• Then, we connected to the auxililary and created spfile from the pfile

SQL> CREATE SPFILE FROM PFILE='location of destination pfile'; ----this is the pfile created earlier.
SQL> STARTUP NOMOUNT;

• Lastly, we run our duplicate command without SPFILE clause. (without SPFILE PARAMETER_VALUE_CONVERT)

In brief, we set our desired parameters for the auxiliary database in a pfile (init.ora), then created spfile from that init.ora(pfile) and started up the auxiliary database in nomount mode using that spfile..

After that , we run our rman duplicate command  without specifying SPFILE.. parameter.

By doing this; we started up the auxilariy instance with the desired parameters and bypassed the automatic spfile update that is done from source instance to auxiliary by rman ( this automatic update is done when we use SPFILE PARAMETER_VALUE_CONVERT parameter and when the auxiliary instance is started up using pfile)..

This workaround saved the day, so I wanted to share it with you.

Note that: starting up the auxiliary database directly with the spfile (filled with the desired parameters) is actually a good thing to do. So we are considering using this approach in our next duplicate sessions as well.