EBS / RDBMS -- 19C Upgrade -- DBUA fails at %50 -- due to failing JAVA check -- JServer has not been correctly loaded into the database

Recently encountered this issue while upgrading an EBS 12.1 database to 19C ( 11.2.0.4 to 19C RU 12--currently the latest one).

Oracle 19C home opatch information:

opatch lspatches

28318139;ORA-31003 ERROR WHEN IMPORTING FULL DATABASE IN PARALLEL

31424070;APPSST19C XTTS PDB - TABLE IMPORT/CREATION FAILED WITH ORA-39083 ORA-14334

32904851;Database Release Update : 19.12.0.0.210720 (32904851)

32876380;OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)

29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

These kinds of unexpected failures are annoying, especially in EBS 19C upgrades where we upgrade the OS, GRID and Databases.. Note that we even containerize the database (in a Single CDB and Single PDB form) during these upgrades and failing even during the DBUA run brings us an extra and unexpected cost..

Besides, we apply lots of apps patches to be in the tatest Tech. code level and to make apps tier be interoperable with 19C.. Well, we do and need to do lots of thing but we fail during DBUA run! 

Luckily we have fixes /or workarounds :)

Let's see today's issue and the solution we implemented;

We were trying to upgrade our 11.2.0.4 EBS database to 19C using DBUA.

DBUA failed at %50.. It basically could not validate the Java packages and Java Virtual Machine.. Those components were VALID before the upgrade, but during the upgrade they became INVALID.

Note that, we didn't cancel , we didn't abort our DBUA sesssion and checked the logs.. (if we could fix the issue,we could just push the Retry button and we could make DBUA continue from where it left off)

The log clearly showed the following and it was the reason that made DBUA complain ->

SQL> BEGIN

IF sys.dbms_registry.is_loaded('JAVAVM',sys.dbms_registry.release_version) != 1 THEN

RAISE_APPLICATION_ERROR(-20000,

'JServer has not been correctly loaded into the database.');

END IF;

END;

 /

ERROR at line 1:

ORA-20000: JServer has not been correctly loaded into the database.

ORA-06512: at line 3

SQL> comp_name, status from dba_registry

Oracle Database Catalog Views UPGRADED

Oracle Database Packages and Types UPGRADED

JServer JAVA Virtual Machine VALID

Oracle Database Java Packages INVALID

Oracle XDK UPGRADED

Oracle Real Application Clusters UPGRADED

OLAP Analytic Workspace UPGRADED

Oracle Multimedia UPGRADED

Spatial UPGRADED

Oracle Text UPGRADED

Oracle XML Database UPGRADED

Oracle OLAP API UPGRADED

OLAP Catalog OPTION OFF

Oracle Database Java Packages was INVALID.  As a result of that; sys.dbms_registry.is_loaded couldn't return 1.. The problem was invalid objects and actually those were supposed compiled in the later stage of the upgrade.. However; it seems we needed to have some of those objects valid even during the upgrade.. So this seemed like an issue which couldn't be predicted by Oracle.

Note that we also had ORA-06598 and it was ignorable.. Unrelated with this issue but just wanted to share: ORA-06598: insufficient INHERIT PRIVILEGES privilege error during upgrade from 11.2.0.2 to 12.1 using DBUA ( Doc ID 1911119.1 )

Anyways; we had a guaranteed restore point but we didn't want to restore our database and retry the failing operation because, probably; we would get exactly the same error in our next try. 

Okay let's jump to the conclusion;

We fixed the issue by following the action plan below;

We executed urltp.sql (compiled the invalids) .. We executed it from the 19C env.. We run utlrp.sql 3-4 times to make sure that the dependent objects are validated as well (if they could be valid ofcourse)) .. Executing ultrp 3-4 times was logical.. We were in the middle of an upgrade, and normally we had thousands of invalid at that moment.. So multiple executions of utlrp was logical to be sure that all the objects that can be validated, were handled correctly by this compilation process.

Once the compilation was finished, we pushed the retry button in the failed DBUA session.. It failed in the first attempt (due to its failed/closed) session, but the second attempt it could continue and the issue was resolved.. Upgrade wass fnished successfully , no invalid components in the dba registry and we just got it resolved!

Seems like this error we faced during the upgrade may be a common issue for 11.2.0.4 to 12.1(or higher) upgrades. But it was interesting. So I just noted it down and wanted to share!