EBS 12.2 -- Failed to load webapp: /OA_HTML , java.lang.NullPointerException , WebAppModule.java:397 & ClassFinderMetaDataEnumaration.java:32

Today, I want to share an interesting issue that I came across in an EBS 12.2 environment. The issue started after applying an EBS CPU... When the issue was escalated to me, the limits of tolerance had already been crossed, so we had to save the day.

Issue: 

After applying an EBS CPU (2022), oacore server couldn't be started.. It was just putting itself to the ADMIN state and normally didn't service anything.. This issue resulted service interruption, as the EBS application lost a large part of its functionality.(login page was throwing HTTP - Error 404 Not Found)

Running autoconfig and taking some general actions didn't solve it. (Regeneration tasks... In case we might have a corrupted/or irregular ebs3rdpartyManifest_xml.tmp).  

Applying a more recent CPU came to mind, but it was an intense work, because all the tests were done for CPU 2022, and those tests should be repeated for 2023 as well.. Besides, we needed to save the day, cause EBS was not functioning.

Here is  part of the actual call/error stack:

weblogic.application.ModuleException: Failed to load webapp: /OA_HTML because of DeploymentException: java.lang.NullPointerException

        at weblogic.servlet.internal.WebAppModule.prepare(WebAppModule.java:397)

        at weblogic.application.internal.flow.ScopedModuleDriver.prepare(ScopedModuleDriver.java:176)

        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)

        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)

        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)

        .....

        ..................

Caused By: java.lang.NullPointerException

        at weblogic.application.utils.ClassFinderMetaDataEnumaration.hasMoreElements(ClassFinderMetaDataEnumaration.java:32)

        at weblogic.servlet.internal.WebAnnotationProcessorImpl.getAnnotatedClasses(WebAnnotationProcessorImpl.java:139)

        at weblogic.servlet.internal.War.getAnnotatedClasses(War.java:1038)

Cause:

Some nonEBS-related files (for example: backup files created manually before applying some apps patches) were present in the APPS filesystem. Weblogic (oacore managed server in this case) was trying to use them, and that was creating the misbehaviour. 

Note that, in some rare cases;  file permissions can also be a cause of this issue..

The solution:

*Remove any nonEBS-related files which reside in $OA_HTML/WEB_INF .. (there were some backups in our case)

*Remove any nonEBS-related files which reside in $OA_HTML (some backup JSP files were there)

*Recompile JSP files.

cd $FND_TOP/patch/115/bin
$ perl -x ojspCompile.pl --compile –flush

*Restart the Oacore Services (managed servers) and that's it.

Some References:

java.lang.NullPointerException at weblogic.application.utils.ClassFinderMetaDataEnumaration.hasMoreElements During The Deployment Of Applications At The Server Startup (Doc ID 1937917.1)

12.2 E-Business Technology Stack Weblogic Service Oacore Services Are Down And Will Not Start Due To Error 'ModuleException: Failed to load webapp' And 'DeploymentException: java.lang.ClassNotFoundException: csrfguard.servlet.JavaScriptServlet' (Doc ID 2533082.1)