Friday, October 29, 2021

Exadata X8M-2 & PCA X8-2 -- Part 3 Migration / add-on - XTTS may fail ORA-17503: ksfdopn:2 Failed to open file

We are continuing our journey on Exadata and PCA migration, and here is the tip of the day;

Although XXTS know reached the 4th version, 4th generation, there are still some cases that it can't handle, and there are still some bugs that we need to pay attention to. Especially when your source database consists of a high number of datafiles, multiple incremental backups to restore and when you use a high degree of parallelism.

Well.. you may end up with the following; while trying to roll forward your target database.. I mean, when applying incremental backups to the datafile copies on the destination system ... (Remember-> we use RMAN incremental backups for accelerating our XXTS migation operations) 

-- Main Document: V4 Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backup (Doc ID 2471245.1)

So, you may end up with;

ORA-17503: ksfdopn:2 Failed to open file +DATAC1/ERM/DATAFILE/erm_732.dbf

ORA-19625: error identifying file +DATAC1/ERM/DATAFILE/erm_732.dbf

ORA-15173: entry 'erm_732.dbf' does not exist in directory 'DATAFILE'

Don't push yourself hard to find your workaround, because there are issues associated with the following bugs ->

bug 31620676: V4 XTTs RECOVERY FAILS WITH MULTIPLE INCREMENTAL BACKUPS

Bug 31054229 : XTTS V4 CREATING INCORRECT RECOVER COMMAND -- note that, this bug currently being investigated by Oracle.. Status : 11 - Code/Hardware Bug (Response/Resolution). Escalated

Luckily, we have workarounds documened in MOS and the ones given in the following MOS document clearly help ->

V4 XTTs: Restore Returns Errors (ORA-19625 or ORA-19641) With Large Number of Datafiles (Doc ID 2689397.1)

We just do some tricks in res.txt and apply one incremental at a time to these both bugs...

However; this workaround alone may not be sufficient in certain cases, where you have a high degree of parallelism. I mean;

It is also recommended to use a lower degree of parallelism in xtt.properties to prevent those RMAN errors . In our case, we concluded that XTTS can not manage 128 parallel thread/workers so, we decreased it to a lower value (32) .

This was a real life case and I wanted to share it with you.. I hope it helps..

Wednesday, October 27, 2021

EBS -- Workflow Mailer -- Validation Error Occured & Wrong Header on outbound e-mails / some "Apps DBA gymnastics" :)

We are approaching the end of the week and even the month. I 'm having busy days with the critical migration and upgrade projects, and that's why having hard time getting here.. but! the month is not over without an EBS post :)

Okay! It is time to do some Apps DBA gymnastics :)  Wf mailer is today's subject!

-- If you want to read some advanced stuff about the wf mailer, you can also read the following blog posts;

https://ermanarslan.blogspot.com/2014/09/ebs-r12-notification-mailer.html

https://ermanarslan.blogspot.com/2016/09/ebs-r12-after-disabling-ssl-on-apps.html

https://ermanarslan.blogspot.com/2020/12/ebs-r12-workflow-mailer-ora-00054-due.html

https://ermanarslan.blogspot.com/2014/08/ebs-workflow-notification-mailer.html

Okay. Today, 2 different issues on the same subject, the workflow notification mailer.

The first one is, the unexpected "Validation Error Occured". Always expect the unexpected! :) 

The SMTP server IP adress was correct. Apps node could connect to the SMTP server and we tested reaching the port with telnet. We even used SMTP command to send an email through linux command line ..There was no auth required, so we had relay. But! no matter what we did , the same error was encountered.. There were no reasons. This was a new EBS 12.1.3 clone and it was obvious that this workflow mailer configuration screen could not manage it! :)

The solution was updating the workflow mailer configuration using @afsvcpup.sql from backend..

$ cd $FND_TOP/sql

$ sqlplus apps/***

SQL> @afsvcpup.sql

Actually, we fixed all the inbound related incorrect configuration that was still there.. The configuration of the source env was still there in the config tables. Unfortuneatly, OAM - WF mailer configuration page (System Administrator > Oracle Application Manager > Workfow > Notification Mailers > Edit ) was still trying to validate them.. 

We updated a bunch of configuration parameters and then retried the same action in the same configuration page.. This time (as expected), we could configure the outbound..

Note that, we didn't have inbound for this clone env but the config was still there! We didn't have SSL  config for the mailer either, but we had SSL there.. So the config was copied from the source as is and that's was the cause.. 

Just updated the following problematic ( incorrect for the clone env) configuration parameters;

Inbound SSL Enabled

Inbound Server Name

Inbound Thread Count

Outbound Server Name

SSL Trust store

Inbound Server Name

The second issue was a peace of cake :) but still want to share it with you.

After configuring the mailer, we tested it and saw that, the value in the From field of the test mail was wrong, as you may expect; we still had the source config even there!

We updated the value of the From field through the advanced config page of the workflow mailer and the issue dissapeared.

 System Administrator > Oracle Application Manager > Workfow > Notification Mailers > Edit > Advanced > step 4 of 8

Well. That 's it.. Don't tell me "these are not interesting", remember I already wrote in the beginning -> 

I m writing this post for doing some Apps DBA  gymnastics  :) and they are real life cases :)

Note that, I'm thinking about writing a patch for these 2 issues. If I have find some time and space :), I may write a custom patch for it.. We 'll see.. A successful clone shouldn't copy the workflow mailer configuration from source to target as is (without changing anything).

Thursday, October 21, 2021

Exadata -- I/O Resource Manager / IORM / Inter-database configuration

As you may already know (actually, I think you definetely know since we are in 2021 and we are now talking about 9th generation of Exadata:), we have an IO Resource Manager in Exadata.. Actually, this is pretty old subject, but! I just could find a space to write it down. :)

Also check my previous post ( written in 2014) named "Exadata -- For Exadata Database Machine Admins -- Filtered Information" ->  https://ermanarslan.blogspot.com/2014/03/exadata-for-exadata-database-machine.html . 

In that post , I give a filtered information (including IORM , migration and other stuff)  for Exadata Admins.

The I/O resource manager is called IORM (acronym) and it is used for managing the Storage /Cell I/O resources in Exadata. In addition to the database resource manager and instance caging for the CPU-like resource management; we can also manage our I/O resource with IORM in Exadata)

Here is a diagram for the description of the architecture of IORM. (reference: Centroid)

So, we can manage our IO resources based on the Categories, Databases and consumer groups. There is a hierarchy as you see in the picture.. The hierarchy used to distribute I/O.

IORM should be used on Exadata especially if you have a lot of databases running on Exadata Machine.. IORM is a friend of consolidation projects, in my opinion..

In this post, I will give some info about the implementation and some example commands along with their purposes, used in real life.

This will be about implementing inter-database IO resource management and what we do in these types of implementations is basically managing the I/O resources of the Oracle Databases that are running on Exadata.. So, we do our work in the Cell/Storage level .. The configuration is per cell basis and we use cellcli and/or dcli to configure the Cells accordingly.

We use the unique names of the databases ( db unique name) while configuring the IORM inter database plans and basically we tell IORM to manage our I/O by following a set of rules.

Here is an example;

"Let %80 of the IO Resources to be used by EBS (while the Store layer is not under a heavy load).. Let %70 of the IO Resources to be used by DWH (while the Storage layer is not under a heavy load).. Don't let EBS to occupy more than %65 of the I/O Resources, while DWH is doing a heavy I/O. Don't Let DWH to occupy more than %35 of the I/O Resources while EBS is doing a heavy I/O.  Let other databases (not EBS, not DWH) to use the remaining I/O resources.. Don't let other databases to prevent EBS or DWH from using I/O resources when they needed."

Here is an example for configuring an IORM inter database plan;

We first check the current plan.. 

-----------------------

dcli -g ~/cell_group -l root cellcli -e list iormplan detail       (I hope you have a proper cell_group.. If you don't, you can create one or you can use cellcli for issuing the commands in each cell)

We set iorm plan objective to auto -> 

-----------------------

alter iormplan objective=auto (in each cell)  -- auto is a must .. This objective lets the IORM to decide the appropriate objective depending on the active workload on the cell.

or 

dcli -g ~/cell_group -l root cellcli -e alter iormplan objective = auto   (in one go using dcli)

We alter the cell to set our IORM plan.. Note that, we use "-" in the end of the lines (except the last line) and we can't use limit for "other" databases..  Using LIMIT attribute for other databases is not allowed -> CELL-00048: The limit attribute is not permitted when specifying dbplan "other" directives.  

Also note that, using LIMIT attribute, we can limit max I/O for a database. So, we ensure that database can not utilize more than that % of I/O resources.

Well, we connect to each cell and issue the following alter iormplan commands to tell IORM what to do while managing the IO resources of our databases;

-----------------------

alter iormplan -

dbplan=((name=EBSPRD, level=1, allocation=65, limit=80, flashcache=on),  -

 (name=DWPRD,  level=1, allocation=35, limit=70, flashcache=on),  -

 (name=other, level=2, allocation=10, flashcache=on))

With this action, we actually configured the IORM and we are done.. Still, we check the IORM plan is active -> 

-----------------------

dcli -g ~/cell_group -l root cellcli -e list iormplan detail

As for monitoring; we use metric_iorm.pl script. We get that script from MOS Note "Tool for Gathering I/O Resource Manager Metrics: metric_iorm.pl (Doc ID 1337265.1)" and follow the instructions documented there.

Okay.. This is pretty much it! I hope you find this useful.

Sunday, October 10, 2021

Erman Arslan's Oracle Forum / September 2021 - "Questions and Answers Series"

Question: How much time do you spend/lose?

Answer: Well, how much time I gain? :) 


Remember, you can ask questions and get remote support using my forum. 36 issues this month!
Just click on the link named "Erman Arslan's Oracle Forum is available now.
Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog
-- or just use the direct link:  


Come on, let's see what we've been up to in September + in the beginning of October 2021. #amaerman
Do not forget to read the blog posts too :)

DR setup for ebs by satish

Reduce downtime for patching by satish

workflow notifications not received by raiq1

ORA-29024: Certificate validation failure by big

High load average in test server by satish

2572809.1 confusion by satish

Workflow oam configuration by satish

Applying missing patches ETCC by satish

dualfs by big

Authentication in apex by satish

JAR expiration date by big

Jar signing: issue with adsign.txt and admin not creating proper jarlist.txt by iqbaljawaid138

incremental update scripts by Roshan

Patch "27491934" overlays " 14825718 ". Conflict check between them is skipped. by satish

RMAN RETENTION POLICY by big

Smtp server - backup scripts by satish

Additional Core addition R12.2 RAC by satish

FA custom value set - error disabling values by satish

Security question by satish

forefully change a password after login by raiq1

slow performance by big

ACL by big

Apex Ebs queries by satish

Workflow issue by satish

sysadmin cannot connect by big

adkeystore and adsign by big

golden gate index issue by Roshan

Oracle data block corruption error in ebs database by raiq1

Apex ORDS EBS by satish

House Keeping in EBSR12 by raiq1

find command impact. by raiq1

Upgradation of grid infrastructure on Exadata X7-2 by Zaheer

Restore RMAN Backup of 11g on 19C by Zaheer

Disk satuaration by satish

DB upgrade (11.2.0.3 to 11.2.0.4) by Barzu

Smtp server - sending email by satish


Saturday, October 9, 2021

Exadata / RDBMS -- DBCA fails KFOD-00101: LRM error [110] - [DBT-05801] There are no ASM disk groups detected.

This time I will keep it short..

This is for the ones who use RAC or Exadata or ASM or GRID .. Should I keep going ? :)

Well, in a migration project we were using XTTS. The target was Exadata ( actually it is needless to say:) It was a new one though.. An Exadata X8-2M.. (Actually, not so new.. Already started talking about Exadata X9-2) 

Anyways, we were creating the target database using DBCA to do the things by the MAA documents and to make it clean.. DBCA failed with KFOD-00101: LRM error [110] .. 

KFOD-00101: LRM error [110] while parsing command line arguments
SEVERE: Oct 08, 2021 2:02:42 PM oracle.install.commons.util.exception.AbstractErrorAdvisor getDetailedMessage
SEVERE: [FATAL] [DBT-05801] There are no ASM disk groups detected.
CAUSE: ASM may not be configured, or ASM disk groups are not created yet.
ACTION: Create ASM disk groups, or change the storage location to File System.

KFOD 101 error was a well known one but! the LRM "110" error was not so famous, so we did a strace check. strace showed that DBUA was trying to read an unrelated init.ora file which was located in the $GRID_HOME/dbs directory.. 

That init.ora was placed there accidentally by the customer. So we removed it. This action solved the problem.  Okay. I think this post is over :)

However, let me also point out one more thing;

I found this worth sharing with you, because it looked pretty interesting before we found the cause. 

Maybe it still is..!  It shows how fragile the environments, the binaries and the tools are. So we need to keep it clean not only to increase our efficieny but also to avoid unnecessary errors.

So , we should keep the environment clean (including our GRID & Database environment:)

EBS / RDBMS -- 19C Upgrade -- Post Upgrade fails "Unable to determine whether the database is pluggable (PDB) or not."

Here I 'am here writing after a long investigation. A long investigation on an error encountered txkPostPDBCreationTasks.pl. I hear you saying -> txkPostPDBCreationTasks again! :) 

Yes. It is a little buggy I think :)

Actually, this time (again!:) the problem was super clear.  I checked the java code and here is what was happening;

The txkPostsPDBCreationTasks.pl was trying to run adbldxml and adbldxml was trying to run the java "GenerateContext" in turn and, GenerateContext was calling getPDBCDB method and that getPDBCDB method was just trying to execute a simple query to check the container configuration -> "select name from v$pdbs" ! just a query to v$pdbs.. No sophisticated check was there! That was it and that v$pdbs was already populated correctly in the database.. No problems with the connection of sqlplus, listeners (both scan and local) were  registered, v$pdbs was correct, no related invalids in the db, the documents followed line by line! .. So the java code was failing somehow.. 

Note that, adbldxml.pl was also failing..

I must admit that, we tried lots of things even though we believed that it was a pure java problem..

As for the solution, we created an Oracle SR and Oracle Support sent us a new adbldxml.pl. We replaced the original adbldxml.pl with the original one and the problem dissapeared!

A quick check showed the difference in getJava_Cmd method.

It seemed the new adbldxml had hard coded $ORACLE_HOME/appsutil/jre , which is set to jre_top.

Earlier, (probably) it was set to $ORACLE_HOME/jdk/jre.

The problem was associated with the following bug : 

Bug 31222688 - ADBLDXML.PL SCRIPT FAILS IN TXKPOSTPDBCREATIONTASKS.PL

I will revisit this blog post and give you the patch number (Ofcourse if we will have one...)

EBS / RDBMS -- 19C Upgrade -- Post Upgrade fails due to missing jar (orai18n.jar) in classpath

Upgrading an EBS RAC Database to 19C (RU 12 & OJVM 12) requires the MOS note "Using Oracle 19c RAC Multitenant (Single PDB) with Oracle E-Business Suite Release 12.1 (Doc ID 2530680.1)" to be followed, and there you will do lots of stuff including the post upgrade tasks... 
Today's issue is encountered while executing the txkPostPDBCreationTasks.pl. ( a post upgrade task actually)

I won't go into the details about the actions that we take for this complex upgrade work (it is complex, because it is EBS! :), but! I will give you the solution that we designed and implemented for getting rid of the following errors in txkPostPDBCreationTasks.pl run;

StackTrace:
java.sql.SQLException: Non supported character set (add orai18n.jar in your classpath): WE8ISO8859P9
at oracle.sql.CharacterSetUnknown.failCharsetUnknown(CharacterSetFactoryThin.java:233)
at oracle.sql.CharacterSetUnknown.convert(CharacterSetFactoryThin.java:194)
at oracle.jdbc.driver.PhysicalConnection.throughDbCharset(PhysicalConnection.java:10425)
at oracle.jdbc.driver.PhysicalConnection.enquoteIdentifier(PhysicalConnection.java:10502)
at oracle.jdbc.driver.OracleStatement.enquoteIdentifier(OracleStatement.java:6452)
at oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3853)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.findColumn(InsensitiveScrollableResultSet.java:270)
at oracle.apps.ad.autoconfig.RestoreProfile.createRestoreFile(RestoreProfile.java:294)
at oracle.apps.ad.autoconfig.RestoreProfile.generateRestoreScript(RestoreProfile.java:128)
at oracle.apps.ad.autoconfig.AppltopDrivers.main(AppltopDrivers.java:1426)

It is needless to say that, we've already put orai18n.jar in the correct directories, as it is stated in the upgrade documents but here we are, still failing unexpectedly in the post upgrade tasks..

The exact phase that we failed was the autoconfig and the failing script (autoconfig executes a bunch of script to do its work, as you know..) was adgentns.pl (+adconfig.pl).

It  was obvious that the issue was related with the classpath.. classpath that is passed to java.. 
adgentns.pl was failing but adclone.pl was also requiring a change.

So I checked the code ( the perl scripts adgentns.pl and adconfig.pl) and added the ora18in.jar in to the classpaths defined there in those perl scripts.. Those perls script execute java and I modified their classpaths (added ora18in.jar) used for executing those java .. (this action requires some knowledge in perl, bytheway) With this action, the issue was fixed! So we could continue the remaining tasks and finished the upgrade! 


No other solution is available at the moment.. 

This should also be directed to development, right? Ofcourse we did it at by opening a SR. However; we fixed the issue on ourselves and could continue upgrading.

I will revisit this blog post and update it with a patch number.. (probably.. because I think the development will write a fix for it.)

Tuesday, October 5, 2021

EBS / RDBMS -- 19C Upgrade -- DBUA fails at %50 -- due to failing JAVA check -- JServer has not been correctly loaded into the database

Recently encountered this issue while upgrading an EBS 12.1 database to 19C ( 11.2.0.4 to 19C RU 12--currently the latest one).

Oracle 19C home opatch information:

opatch lspatches

28318139;ORA-31003 ERROR WHEN IMPORTING FULL DATABASE IN PARALLEL

31424070;APPSST19C XTTS PDB - TABLE IMPORT/CREATION FAILED WITH ORA-39083 ORA-14334

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

These kinds of unexpected failures are annoying, especially in EBS 19C upgrades where we upgrade the OS, GRID and Databases.. Note that we even containerize the database (in a Single CDB and Single PDB form) during these upgrades and failing even during the DBUA run brings us an extra and unexpected cost..

Besides, we apply lots of apps patches to be in the tatest Tech. code level and to make apps tier be interoperable with 19C.. Well, we do and need to do lots of thing but we fail during DBUA run! 

Luckily we have fixes /or workarounds :)

Let's see today's issue and the solution we implemented;

We were trying to upgrade our 11.2.0.4 EBS database to 19C using DBUA.

DBUA failed at %50.. It basically could not validate the Java packages and Java Virtual Machine.. Those components were VALID before the upgrade, but during the upgrade they became INVALID.



Note that, we didn't cancel , we didn't abort our DBUA sesssion and checked the logs.. (if we could fix the issue,we could just push the Retry button and we could make DBUA continue from where it left off)

The log clearly showed the following and it was the reason that made DBUA complain ->

SQL> BEGIN

IF sys.dbms_registry.is_loaded('JAVAVM',sys.dbms_registry.release_version) != 1 THEN

RAISE_APPLICATION_ERROR(-20000,

'JServer has not been correctly loaded into the database.');

END IF;

END;

 /

ERROR at line 1:

ORA-20000: JServer has not been correctly loaded into the database.

ORA-06512: at line 3

SQL> comp_name, status from dba_registry

Oracle Database Catalog Views UPGRADED

Oracle Database Packages and Types UPGRADED

JServer JAVA Virtual Machine VALID

Oracle Database Java Packages INVALID

Oracle XDK UPGRADED

Oracle Real Application Clusters UPGRADED

OLAP Analytic Workspace UPGRADED

Oracle Multimedia UPGRADED

Spatial UPGRADED

Oracle Text UPGRADED

Oracle XML Database UPGRADED

Oracle OLAP API UPGRADED

OLAP Catalog OPTION OFF


Oracle Database Java Packages was INVALID.  As a result of that; sys.dbms_registry.is_loaded couldn't return 1.. The problem was invalid objects and actually those were supposed compiled in the later stage of the upgrade.. However; it seems we needed to have some of those objects valid even during the upgrade.. So this seemed like an issue which couldn't be predicted by Oracle.

Note that we also had ORA-06598 and it was ignorable.. Unrelated with this issue but just wanted to share: ORA-06598: insufficient INHERIT PRIVILEGES privilege error during upgrade from 11.2.0.2 to 12.1 using DBUA ( Doc ID 1911119.1 )

Anyways; we had a guaranteed restore point but we didn't want to restore our database and retry the failing operation because, probably; we would get exactly the same error in our next try. 

Okay let's jump to the conclusion;

We fixed the issue by following the action plan below;

We executed urltp.sql (compiled the invalids) .. We executed it from the 19C env.. We run utlrp.sql 3-4 times to make sure that the dependent objects are validated as well (if they could be valid ofcourse)) .. Executing ultrp 3-4 times was logical.. We were in the middle of an upgrade, and normally we had thousands of invalid at that moment.. So multiple executions of utlrp was logical to be sure that all the objects that can be validated, were handled correctly by this compilation process.

Once the compilation was finished, we pushed the retry button in the failed DBUA session.. It failed in the first attempt (due to its failed/closed) session, but the second attempt it could continue and the issue was resolved.. Upgrade wass fnished successfully , no invalid components in the dba registry and we just got it resolved!

Seems like this error we faced during the upgrade may be a common issue for 11.2.0.4 to 12.1(or higher) upgrades. But it was interesting. So I just noted it down and wanted to share!