Monday, November 8, 2021

Erman Arslan's Oracle Forum / October 2021 - "Questions and Answers Series"

Question: How much time do you spend/lose?

Answer: Well, how much time I gain? :) 

Remember, you can ask questions and get remote support using my forum. 29 issues this month!
Just click on the link named "Erman Arslan's Oracle Forum is available now.
Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog
-- or just use the direct link:  


Come on, let's see what we've been up to in October+ in the beginning of November 2021. #amaerman
Do not forget to read the blog posts too :)



PCA-X8- Engineered system - OEL 6.9 by Sheikh Rehan

Ebs apex integration by satish

Planing for Migration from Existing Hardware to ODA by raiq1

permission to file by Roshan

Integration from oracle to PostgreSQL database by satish

Purge/Archival in ebs by satish

Authentication in login page takes long time by satish

Purging ebs data by satish

ORA-01111: Name For Data File Is Unknown by big

Erp apex integration by satish

No data in Shareable memory section of AWR by big

service statistics in AWR by big

Load avrage in AWR by big

Log buffer by big

Bursting error R12.2 by satish

Bursting ssl R12.2.5 by satish

EBS DR by satish

CPU busy by oracle by big

Apex in rac nodes by satish

Ad_zd and wf_event valid but still shows invalid ot dropped by Mohsin Patel

2572809.1 confusion by satish

DR setup for ebs by satish

Reduce downtime for patching by satish

workflow notifications not received by raiq1

ORA-29024: Certificate validation failure by big

High load average in test server by satish

Workflow oam configuration by satish

Applying missing patches ETCC by satish

dualfs by big


Friday, November 5, 2021

EBS 12.2 -- ORA-06508 + ORA-04065 for APPS.AD_ZD / 6508 trace name errorstack level 3 and all that

Yesterday, I worked on a weird case... The case was on a production EBS 12.2 instance. The problem appeared after recreating the APPS.AD_ZD package body along with its spec.. 

Actually, this kind of a compilation should not be done manually (and suddenly:), but it seems some "consultant" (yes in quotation marks! :) recommended such an action to fix some ADOP problems and then the environment ended up with lots of ORA-06508 errors, thus lots of OAF and Forms pages could not function. 

Lots of ORA-06508 blocking lots of EBS operations (and as you may guess)-> preventing the Business.. 

So it is needless to say it was a Severity 1... Well.. At the end of the story, I have been contacted to fix the problem ASAP.

You know, I like to work on the details/on these weird cases, because those tiny little gaps/problems/errors/unexpected things that we see in those details/in such cases, have the potential to reveal lots information..

Actually, I was already familiar with these kinds of errors and I even wrote some blog post about them .. (https://ermanarslan.blogspot.com/2015/03/ebs-r12-app-fnd-01926-ora-06508-ora.html

But! this one was different. 

Following is an example of an ORA-06508 error encountered on a standard EBS form;


Normally, firstly, I suggested the following action plan;

1)Shutdown apps tier services on all apps nodes. Ensure there are no apps-related OS processes left running there.
2)Run utlrp.sql - sqlplus “/as sysdba” @$ORACLE_HOME/rdbms/admin/urltp.sql
3)Run apps schema compile using adadmin.
4)Run -> Alter package AD_ZD compile
5)Alter package AD_ZD compile body
6)Repeat step 2&3
7)Restart the EBS database
8)Start Apps Tier and retest

Note that, there were no invalid objects in the database and AD_ZD was already valid and seemed healthy..

Well, this action plan didn't help and then I saw a grant on the AD_ZD table.. An exectute grant was given manually.. Remember we have EBR (Edition Based Redefinition) in EBS 12.2, so we need to pay attention to that..  So I told the admin to revoke that grant and use AD_ZD.grant_privs to grant it back. (of course, I told -> take those necessary compilation actions in case you see an increase in the count of invalid objects)

Example: exec AD_ZD.GRANT_PRIVS ('EXECUTE','AD_ZD','OAM_MONITOR_ROLE');

This was a good move but it didn't solve the issue either. -- Still, alesson learned there..(for the admin at least)

Then I decided the check the call stack.. So I made the admin set the following event in the database and reproduce the error; 

ALTER SYSTEM SET EVENTS '6508 trace name errorstack level 3’;

After setting the event and reproducing the error, we checked the event trace and the call stack was there;

----- Error Stack Dump -----

ORA-04065: package body "APPS.AD_ZD" 

ORA-06508: PL/SQL: APPS.AD_ZD"

----- SQL Statement (None) -----

Current SQL information unavailable - no cursor.

----- PL/SQL Stack -----

----- PL/SQL Call Stack -----

object line object

handle number name

327fccceb8 1074 package body APPS.WF_CORE

253df749c0 489 package body APPS.WF_EVENT

253df749c0 2141 package body APPS.WF_EVENT

253df749c0 3021 package body APPS.WF_EVENT

319d17c378 190 package body APPS.JTF_TASK_WF_EVENTS_PVT

30ccddb468 2137 package body APPS.JTF_TASK_ASSIGNMENTS_PVT

30ccddb468 1213 package body APPS.JTF_TASK_ASSIGNMENTS_PVT

31e85eb340 4080 package body APPS.JTF_TASKS_PVT

2efd1fc9d0 4477 package body APPS.JTF_TASKS_PUB

It was cearly seen that the code in 1074th line of WF_CORE package was failing..
We had a call to ad_zd.get_edition() function there, in that line.. 
However; AD_ZD package was already valid and when we tried to execute the same function via sqlplus or/ Toad (using an APPS session), we could succeed without any problem..

This was one of my AHA moments actually.. Just after seeing the contents of the trace and ensuring the APPZ_ZD.GET_EDITION can be executed by an Apps session, I started to think that there was a kind of a timestamp discrepancy exist for the objects that we see in the call stack of the event trace.

At the end of the story, I suggested compiling WF_CORE,WF_EVENT and all the other packages listed above in the call stack.. Of course these kinds of compilations would make some dependent objects invalid and that's why I also suggested to run utlrp.sql and apps schema compile as the post actions..The compilations saved the day, resolved the timestamp discrepancies and the issue was solved! 

It was a risky move (compiling a bunch of standard packages), but it was necessary in this case..


a reminder -> Don't forget to disable the event '6508 trace name errorstack level 3' after reproducing the error and catching the call stack :) Those events are just for diagnosing purposes, so they should not be  left enabled.

Friday, October 29, 2021

Exadata X8M-2 & PCA X8-2 -- Part 3 Migration / add-on - XTTS may fail ORA-17503: ksfdopn:2 Failed to open file

We are continuing our journey on Exadata and PCA migration, and here is the tip of the day;

Although XXTS know reached the 4th version, 4th generation, there are still some cases that it can't handle, and there are still some bugs that we need to pay attention to. Especially when your source database consists of a high number of datafiles, multiple incremental backups to restore and when you use a high degree of parallelism.

Well.. you may end up with the following; while trying to roll forward your target database.. I mean, when applying incremental backups to the datafile copies on the destination system ... (Remember-> we use RMAN incremental backups for accelerating our XXTS migation operations) 

-- Main Document: V4 Reduce Transportable Tablespace Downtime using Cross Platform Incremental Backup (Doc ID 2471245.1)

So, you may end up with;

ORA-17503: ksfdopn:2 Failed to open file +DATAC1/ERM/DATAFILE/erm_732.dbf

ORA-19625: error identifying file +DATAC1/ERM/DATAFILE/erm_732.dbf

ORA-15173: entry 'erm_732.dbf' does not exist in directory 'DATAFILE'

Don't push yourself hard to find your workaround, because there are issues associated with the following bugs ->

bug 31620676: V4 XTTs RECOVERY FAILS WITH MULTIPLE INCREMENTAL BACKUPS

Bug 31054229 : XTTS V4 CREATING INCORRECT RECOVER COMMAND -- note that, this bug currently being investigated by Oracle.. Status : 11 - Code/Hardware Bug (Response/Resolution). Escalated

Luckily, we have workarounds documened in MOS and the ones given in the following MOS document clearly help ->

V4 XTTs: Restore Returns Errors (ORA-19625 or ORA-19641) With Large Number of Datafiles (Doc ID 2689397.1)

We just do some tricks in res.txt and apply one incremental at a time to these both bugs...

However; this workaround alone may not be sufficient in certain cases, where you have a high degree of parallelism. I mean;

It is also recommended to use a lower degree of parallelism in xtt.properties to prevent those RMAN errors . In our case, we concluded that XTTS can not manage 128 parallel thread/workers so, we decreased it to a lower value (32) .

This was a real life case and I wanted to share it with you.. I hope it helps..

Wednesday, October 27, 2021

EBS -- Workflow Mailer -- Validation Error Occured & Wrong Header on outbound e-mails / some "Apps DBA gymnastics" :)

We are approaching the end of the week and even the month. I 'm having busy days with the critical migration and upgrade projects, and that's why having hard time getting here.. but! the month is not over without an EBS post :)

Okay! It is time to do some Apps DBA gymnastics :)  Wf mailer is today's subject!

-- If you want to read some advanced stuff about the wf mailer, you can also read the following blog posts;

https://ermanarslan.blogspot.com/2014/09/ebs-r12-notification-mailer.html

https://ermanarslan.blogspot.com/2016/09/ebs-r12-after-disabling-ssl-on-apps.html

https://ermanarslan.blogspot.com/2020/12/ebs-r12-workflow-mailer-ora-00054-due.html

https://ermanarslan.blogspot.com/2014/08/ebs-workflow-notification-mailer.html

Okay. Today, 2 different issues on the same subject, the workflow notification mailer.

The first one is, the unexpected "Validation Error Occured". Always expect the unexpected! :) 

The SMTP server IP adress was correct. Apps node could connect to the SMTP server and we tested reaching the port with telnet. We even used SMTP command to send an email through linux command line ..There was no auth required, so we had relay. But! no matter what we did , the same error was encountered.. There were no reasons. This was a new EBS 12.1.3 clone and it was obvious that this workflow mailer configuration screen could not manage it! :)

The solution was updating the workflow mailer configuration using @afsvcpup.sql from backend..

$ cd $FND_TOP/sql

$ sqlplus apps/***

SQL> @afsvcpup.sql

Actually, we fixed all the inbound related incorrect configuration that was still there.. The configuration of the source env was still there in the config tables. Unfortuneatly, OAM - WF mailer configuration page (System Administrator > Oracle Application Manager > Workfow > Notification Mailers > Edit ) was still trying to validate them.. 

We updated a bunch of configuration parameters and then retried the same action in the same configuration page.. This time (as expected), we could configure the outbound..

Note that, we didn't have inbound for this clone env but the config was still there! We didn't have SSL  config for the mailer either, but we had SSL there.. So the config was copied from the source as is and that's was the cause.. 

Just updated the following problematic ( incorrect for the clone env) configuration parameters;

Inbound SSL Enabled

Inbound Server Name

Inbound Thread Count

Outbound Server Name

SSL Trust store

Inbound Server Name

The second issue was a peace of cake :) but still want to share it with you.

After configuring the mailer, we tested it and saw that, the value in the From field of the test mail was wrong, as you may expect; we still had the source config even there!

We updated the value of the From field through the advanced config page of the workflow mailer and the issue dissapeared.

 System Administrator > Oracle Application Manager > Workfow > Notification Mailers > Edit > Advanced > step 4 of 8

Well. That 's it.. Don't tell me "these are not interesting", remember I already wrote in the beginning -> 

I m writing this post for doing some Apps DBA  gymnastics  :) and they are real life cases :)

Note that, I'm thinking about writing a patch for these 2 issues. If I have find some time and space :), I may write a custom patch for it.. We 'll see.. A successful clone shouldn't copy the workflow mailer configuration from source to target as is (without changing anything).

Thursday, October 21, 2021

Exadata -- I/O Resource Manager / IORM / Inter-database configuration

As you may already know (actually, I think you definetely know since we are in 2021 and we are now talking about 9th generation of Exadata:), we have an IO Resource Manager in Exadata.. Actually, this is pretty old subject, but! I just could find a space to write it down. :)

Also check my previous post ( written in 2014) named "Exadata -- For Exadata Database Machine Admins -- Filtered Information" ->  https://ermanarslan.blogspot.com/2014/03/exadata-for-exadata-database-machine.html . 

In that post , I give a filtered information (including IORM , migration and other stuff)  for Exadata Admins.

The I/O resource manager is called IORM (acronym) and it is used for managing the Storage /Cell I/O resources in Exadata. In addition to the database resource manager and instance caging for the CPU-like resource management; we can also manage our I/O resource with IORM in Exadata)

Here is a diagram for the description of the architecture of IORM. (reference: Centroid)

So, we can manage our IO resources based on the Categories, Databases and consumer groups. There is a hierarchy as you see in the picture.. The hierarchy used to distribute I/O.

IORM should be used on Exadata especially if you have a lot of databases running on Exadata Machine.. IORM is a friend of consolidation projects, in my opinion..

In this post, I will give some info about the implementation and some example commands along with their purposes, used in real life.

This will be about implementing inter-database IO resource management and what we do in these types of implementations is basically managing the I/O resources of the Oracle Databases that are running on Exadata.. So, we do our work in the Cell/Storage level .. The configuration is per cell basis and we use cellcli and/or dcli to configure the Cells accordingly.

We use the unique names of the databases ( db unique name) while configuring the IORM inter database plans and basically we tell IORM to manage our I/O by following a set of rules.

Here is an example;

"Let %80 of the IO Resources to be used by EBS (while the Store layer is not under a heavy load).. Let %70 of the IO Resources to be used by DWH (while the Storage layer is not under a heavy load).. Don't let EBS to occupy more than %65 of the I/O Resources, while DWH is doing a heavy I/O. Don't Let DWH to occupy more than %35 of the I/O Resources while EBS is doing a heavy I/O.  Let other databases (not EBS, not DWH) to use the remaining I/O resources.. Don't let other databases to prevent EBS or DWH from using I/O resources when they needed."

Here is an example for configuring an IORM inter database plan;

We first check the current plan.. 

-----------------------

dcli -g ~/cell_group -l root cellcli -e list iormplan detail       (I hope you have a proper cell_group.. If you don't, you can create one or you can use cellcli for issuing the commands in each cell)

We set iorm plan objective to auto -> 

-----------------------

alter iormplan objective=auto (in each cell)  -- auto is a must .. This objective lets the IORM to decide the appropriate objective depending on the active workload on the cell.

or 

dcli -g ~/cell_group -l root cellcli -e alter iormplan objective = auto   (in one go using dcli)

We alter the cell to set our IORM plan.. Note that, we use "-" in the end of the lines (except the last line) and we can't use limit for "other" databases..  Using LIMIT attribute for other databases is not allowed -> CELL-00048: The limit attribute is not permitted when specifying dbplan "other" directives.  

Also note that, using LIMIT attribute, we can limit max I/O for a database. So, we ensure that database can not utilize more than that % of I/O resources.

Well, we connect to each cell and issue the following alter iormplan commands to tell IORM what to do while managing the IO resources of our databases;

-----------------------

alter iormplan -

dbplan=((name=EBSPRD, level=1, allocation=65, limit=80, flashcache=on),  -

 (name=DWPRD,  level=1, allocation=35, limit=70, flashcache=on),  -

 (name=other, level=2, allocation=10, flashcache=on))

With this action, we actually configured the IORM and we are done.. Still, we check the IORM plan is active -> 

-----------------------

dcli -g ~/cell_group -l root cellcli -e list iormplan detail

As for monitoring; we use metric_iorm.pl script. We get that script from MOS Note "Tool for Gathering I/O Resource Manager Metrics: metric_iorm.pl (Doc ID 1337265.1)" and follow the instructions documented there.

Okay.. This is pretty much it! I hope you find this useful.

Sunday, October 10, 2021

Erman Arslan's Oracle Forum / September 2021 - "Questions and Answers Series"

Question: How much time do you spend/lose?

Answer: Well, how much time I gain? :) 


Remember, you can ask questions and get remote support using my forum. 36 issues this month!
Just click on the link named "Erman Arslan's Oracle Forum is available now.
Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog
-- or just use the direct link:  


Come on, let's see what we've been up to in September + in the beginning of October 2021. #amaerman
Do not forget to read the blog posts too :)

DR setup for ebs by satish

Reduce downtime for patching by satish

workflow notifications not received by raiq1

ORA-29024: Certificate validation failure by big

High load average in test server by satish

2572809.1 confusion by satish

Workflow oam configuration by satish

Applying missing patches ETCC by satish

dualfs by big

Authentication in apex by satish

JAR expiration date by big

Jar signing: issue with adsign.txt and admin not creating proper jarlist.txt by iqbaljawaid138

incremental update scripts by Roshan

Patch "27491934" overlays " 14825718 ". Conflict check between them is skipped. by satish

RMAN RETENTION POLICY by big

Smtp server - backup scripts by satish

Additional Core addition R12.2 RAC by satish

FA custom value set - error disabling values by satish

Security question by satish

forefully change a password after login by raiq1

slow performance by big

ACL by big

Apex Ebs queries by satish

Workflow issue by satish

sysadmin cannot connect by big

adkeystore and adsign by big

golden gate index issue by Roshan

Oracle data block corruption error in ebs database by raiq1

Apex ORDS EBS by satish

House Keeping in EBSR12 by raiq1

find command impact. by raiq1

Upgradation of grid infrastructure on Exadata X7-2 by Zaheer

Restore RMAN Backup of 11g on 19C by Zaheer

Disk satuaration by satish

DB upgrade (11.2.0.3 to 11.2.0.4) by Barzu

Smtp server - sending email by satish


Saturday, October 9, 2021

Exadata / RDBMS -- DBCA fails KFOD-00101: LRM error [110] - [DBT-05801] There are no ASM disk groups detected.

This time I will keep it short..

This is for the ones who use RAC or Exadata or ASM or GRID .. Should I keep going ? :)

Well, in a migration project we were using XTTS. The target was Exadata ( actually it is needless to say:) It was a new one though.. An Exadata X8-2M.. (Actually, not so new.. Already started talking about Exadata X9-2) 

Anyways, we were creating the target database using DBCA to do the things by the MAA documents and to make it clean.. DBCA failed with KFOD-00101: LRM error [110] .. 

KFOD-00101: LRM error [110] while parsing command line arguments
SEVERE: Oct 08, 2021 2:02:42 PM oracle.install.commons.util.exception.AbstractErrorAdvisor getDetailedMessage
SEVERE: [FATAL] [DBT-05801] There are no ASM disk groups detected.
CAUSE: ASM may not be configured, or ASM disk groups are not created yet.
ACTION: Create ASM disk groups, or change the storage location to File System.

KFOD 101 error was a well known one but! the LRM "110" error was not so famous, so we did a strace check. strace showed that DBUA was trying to read an unrelated init.ora file which was located in the $GRID_HOME/dbs directory.. 

That init.ora was placed there accidentally by the customer. So we removed it. This action solved the problem.  Okay. I think this post is over :)

However, let me also point out one more thing;

I found this worth sharing with you, because it looked pretty interesting before we found the cause. 

Maybe it still is..!  It shows how fragile the environments, the binaries and the tools are. So we need to keep it clean not only to increase our efficieny but also to avoid unnecessary errors.

So , we should keep the environment clean (including our GRID & Database environment:)

EBS / RDBMS -- 19C Upgrade -- Post Upgrade fails "Unable to determine whether the database is pluggable (PDB) or not."

Here I 'am here writing after a long investigation. A long investigation on an error encountered txkPostPDBCreationTasks.pl. I hear you saying -> txkPostPDBCreationTasks again! :) 

Yes. It is a little buggy I think :)

Actually, this time (again!:) the problem was super clear.  I checked the java code and here is what was happening;

The txkPostsPDBCreationTasks.pl was trying to run adbldxml and adbldxml was trying to run the java "GenerateContext" in turn and, GenerateContext was calling getPDBCDB method and that getPDBCDB method was just trying to execute a simple query to check the container configuration -> "select name from v$pdbs" ! just a query to v$pdbs.. No sophisticated check was there! That was it and that v$pdbs was already populated correctly in the database.. No problems with the connection of sqlplus, listeners (both scan and local) were  registered, v$pdbs was correct, no related invalids in the db, the documents followed line by line! .. So the java code was failing somehow.. 

Note that, adbldxml.pl was also failing..

I must admit that, we tried lots of things even though we believed that it was a pure java problem..

As for the solution, we created an Oracle SR and Oracle Support sent us a new adbldxml.pl. We replaced the original adbldxml.pl with the original one and the problem dissapeared!

A quick check showed the difference in getJava_Cmd method.

It seemed the new adbldxml had hard coded $ORACLE_HOME/appsutil/jre , which is set to jre_top.

Earlier, (probably) it was set to $ORACLE_HOME/jdk/jre.

The problem was associated with the following bug : 

Bug 31222688 - ADBLDXML.PL SCRIPT FAILS IN TXKPOSTPDBCREATIONTASKS.PL

I will revisit this blog post and give you the patch number (Ofcourse if we will have one...)

EBS / RDBMS -- 19C Upgrade -- Post Upgrade fails due to missing jar (orai18n.jar) in classpath

Upgrading an EBS RAC Database to 19C (RU 12 & OJVM 12) requires the MOS note "Using Oracle 19c RAC Multitenant (Single PDB) with Oracle E-Business Suite Release 12.1 (Doc ID 2530680.1)" to be followed, and there you will do lots of stuff including the post upgrade tasks... 
Today's issue is encountered while executing the txkPostPDBCreationTasks.pl. ( a post upgrade task actually)

I won't go into the details about the actions that we take for this complex upgrade work (it is complex, because it is EBS! :), but! I will give you the solution that we designed and implemented for getting rid of the following errors in txkPostPDBCreationTasks.pl run;

StackTrace:
java.sql.SQLException: Non supported character set (add orai18n.jar in your classpath): WE8ISO8859P9
at oracle.sql.CharacterSetUnknown.failCharsetUnknown(CharacterSetFactoryThin.java:233)
at oracle.sql.CharacterSetUnknown.convert(CharacterSetFactoryThin.java:194)
at oracle.jdbc.driver.PhysicalConnection.throughDbCharset(PhysicalConnection.java:10425)
at oracle.jdbc.driver.PhysicalConnection.enquoteIdentifier(PhysicalConnection.java:10502)
at oracle.jdbc.driver.OracleStatement.enquoteIdentifier(OracleStatement.java:6452)
at oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3853)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.findColumn(InsensitiveScrollableResultSet.java:270)
at oracle.apps.ad.autoconfig.RestoreProfile.createRestoreFile(RestoreProfile.java:294)
at oracle.apps.ad.autoconfig.RestoreProfile.generateRestoreScript(RestoreProfile.java:128)
at oracle.apps.ad.autoconfig.AppltopDrivers.main(AppltopDrivers.java:1426)

It is needless to say that, we've already put orai18n.jar in the correct directories, as it is stated in the upgrade documents but here we are, still failing unexpectedly in the post upgrade tasks..

The exact phase that we failed was the autoconfig and the failing script (autoconfig executes a bunch of script to do its work, as you know..) was adgentns.pl (+adconfig.pl).

It  was obvious that the issue was related with the classpath.. classpath that is passed to java.. 
adgentns.pl was failing but adclone.pl was also requiring a change.

So I checked the code ( the perl scripts adgentns.pl and adconfig.pl) and added the ora18in.jar in to the classpaths defined there in those perl scripts.. Those perls script execute java and I modified their classpaths (added ora18in.jar) used for executing those java .. (this action requires some knowledge in perl, bytheway) With this action, the issue was fixed! So we could continue the remaining tasks and finished the upgrade! 


No other solution is available at the moment.. 

This should also be directed to development, right? Ofcourse we did it at by opening a SR. However; we fixed the issue on ourselves and could continue upgrading.

I will revisit this blog post and update it with a patch number.. (probably.. because I think the development will write a fix for it.)

Tuesday, October 5, 2021

EBS / RDBMS -- 19C Upgrade -- DBUA fails at %50 -- due to failing JAVA check -- JServer has not been correctly loaded into the database

Recently encountered this issue while upgrading an EBS 12.1 database to 19C ( 11.2.0.4 to 19C RU 12--currently the latest one).

Oracle 19C home opatch information:

opatch lspatches

28318139;ORA-31003 ERROR WHEN IMPORTING FULL DATABASE IN PARALLEL

31424070;APPSST19C XTTS PDB - TABLE IMPORT/CREATION FAILED WITH ORA-39083 ORA-14334

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

These kinds of unexpected failures are annoying, especially in EBS 19C upgrades where we upgrade the OS, GRID and Databases.. Note that we even containerize the database (in a Single CDB and Single PDB form) during these upgrades and failing even during the DBUA run brings us an extra and unexpected cost..

Besides, we apply lots of apps patches to be in the tatest Tech. code level and to make apps tier be interoperable with 19C.. Well, we do and need to do lots of thing but we fail during DBUA run! 

Luckily we have fixes /or workarounds :)

Let's see today's issue and the solution we implemented;

We were trying to upgrade our 11.2.0.4 EBS database to 19C using DBUA.

DBUA failed at %50.. It basically could not validate the Java packages and Java Virtual Machine.. Those components were VALID before the upgrade, but during the upgrade they became INVALID.



Note that, we didn't cancel , we didn't abort our DBUA sesssion and checked the logs.. (if we could fix the issue,we could just push the Retry button and we could make DBUA continue from where it left off)

The log clearly showed the following and it was the reason that made DBUA complain ->

SQL> BEGIN

IF sys.dbms_registry.is_loaded('JAVAVM',sys.dbms_registry.release_version) != 1 THEN

RAISE_APPLICATION_ERROR(-20000,

'JServer has not been correctly loaded into the database.');

END IF;

END;

 /

ERROR at line 1:

ORA-20000: JServer has not been correctly loaded into the database.

ORA-06512: at line 3

SQL> comp_name, status from dba_registry

Oracle Database Catalog Views UPGRADED

Oracle Database Packages and Types UPGRADED

JServer JAVA Virtual Machine VALID

Oracle Database Java Packages INVALID

Oracle XDK UPGRADED

Oracle Real Application Clusters UPGRADED

OLAP Analytic Workspace UPGRADED

Oracle Multimedia UPGRADED

Spatial UPGRADED

Oracle Text UPGRADED

Oracle XML Database UPGRADED

Oracle OLAP API UPGRADED

OLAP Catalog OPTION OFF


Oracle Database Java Packages was INVALID.  As a result of that; sys.dbms_registry.is_loaded couldn't return 1.. The problem was invalid objects and actually those were supposed compiled in the later stage of the upgrade.. However; it seems we needed to have some of those objects valid even during the upgrade.. So this seemed like an issue which couldn't be predicted by Oracle.

Note that we also had ORA-06598 and it was ignorable.. Unrelated with this issue but just wanted to share: ORA-06598: insufficient INHERIT PRIVILEGES privilege error during upgrade from 11.2.0.2 to 12.1 using DBUA ( Doc ID 1911119.1 )

Anyways; we had a guaranteed restore point but we didn't want to restore our database and retry the failing operation because, probably; we would get exactly the same error in our next try. 

Okay let's jump to the conclusion;

We fixed the issue by following the action plan below;

We executed urltp.sql (compiled the invalids) .. We executed it from the 19C env.. We run utlrp.sql 3-4 times to make sure that the dependent objects are validated as well (if they could be valid ofcourse)) .. Executing ultrp 3-4 times was logical.. We were in the middle of an upgrade, and normally we had thousands of invalid at that moment.. So multiple executions of utlrp was logical to be sure that all the objects that can be validated, were handled correctly by this compilation process.

Once the compilation was finished, we pushed the retry button in the failed DBUA session.. It failed in the first attempt (due to its failed/closed) session, but the second attempt it could continue and the issue was resolved.. Upgrade wass fnished successfully , no invalid components in the dba registry and we just got it resolved!

Seems like this error we faced during the upgrade may be a common issue for 11.2.0.4 to 12.1(or higher) upgrades. But it was interesting. So I just noted it down and wanted to share!