EBS 12.2 --OACORE -- java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext

Recently, I have faced with a weird situation where I find myself again an inoperative EBS 12.2..
Login page was encountring the following error..

We are already familiar with the error "Failure of server Apache Bridge".. We know, that It appears when the oacore managed server is in shutdown status, or when the oacore managed server is in hang status..
But in this problem, oacore was running. Restarting oacore or even restarting the whole application tier was not a solution.

Further diagnosis showed that, the problem in oacore was the following;

 java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext
        at oracle.apps.fnd.security.AppsServletFilter.init(AppsServletFilter.java:148)
        at oracle.apps.fnd.security.LeakDetectionFilter.init(LeakDetectionFilter.java:69)
        at weblogic.servlet.internal.FilterManager$FilterInitAction.run(FilterManager.java:343)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.FilterManager.loadFilter(FilterManager.java:96)
        at weblogic.servlet.internal.FilterManager.preloadFilters(FilterManager.java:57)
        at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1874)
        at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3163)
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1520)
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:486)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
        at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247)

So ,basically there was problem in loading classes, as java could not find the the class named WebAppsContext..
This made me think that there was a classpath problem.. Someone could have changed something in Managed server configurations.. Maybe the classpath was changed....
According to this info, I decided to run an autoconfig in appsTier.

Autoconfig would change the configurations back to their factory defaults, or lets say to their proper values :)
But, autoconfig was also encountering errors. 2 scripts i n profile phase of autoconfig was not run successfully.
One of these scripts was ; afwebprf.sql.
The error at the top of the error stack  was : PL/SQL: numeric or value error: character to number conversion error..
When I looked at the sql script , I saw that ; the script was failing while checking the NODE_TRUST_LEVEL profile..

When I checked the value of that profile; I saw that it was set to "Y"

select * from (
select p.profile_option_name SHORT_NAME, n.user_profile_option_name NAME,
decode(v.level_id, 10001, 'Site', 10002, 'Application',
10003, 'Responsibility', 10004, 'User', 10005, 'Server',
10007, 'SERVRESP', 'UnDef') LEVEL_SET,
decode(to_char(v.level_id), '10001', '',
'10002', app.application_short_name,
'10003', rsp.responsibility_key,
'10005', svr.node_name,
'10006', org.name,
'10004', usr.user_name,
'10007', 'Serv/resp',
'UnDef') "CONTEXT",
v.profile_option_value VALUE
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
and upper(n.user_profile_option_name) like '%'
and usr.user_id (+) = v.level_value
and rsp.application_id (+) = v.level_value_application_id
and rsp.responsibility_id (+) = v.level_value
and app.application_id (+) = v.level_value
and svr.node_id (+) = v.level_value
and org.organization_id (+) = v.level_value
order by short_name, level_set
) where LEVEL_SET='Site'
order by name

So, this profile should have been set to a number not a char! ("Y")..
This was a clear problem, so I have updated it using the following;

DECLARE
stat boolean;
BEGIN
dbms_output.disable;
dbms_output.enable(100000);
stat := FND_PROFILE.SAVE('NODE_TRUST_LEVEL', '2', 'SITE');
IF stat THEN
dbms_output.put_line( 'Stat = TRUE - profile updated' );
ELSE
dbms_output.put_line( 'Stat = FALSE - profile NOT updated' );
END IF;
commit;
END;

This action have fixed the problem , but only for afwebprf.sql...
Other scripts in autoconfig's profile phase was also encountering the same error; "PL/SQL: numeric or value error: character to number "

To analyze further, I checked another script, it was trying to read the orgid profile..
When I checked the orgid profile setting in site and user level, I have seen that they all were set to "Y"..

This showed me that the problem is clearly in fnd_profile_option_values.. Someone could have updated the fnd_profile_option_values wrongly, using a direct update statement.. I say "direct update statement" because using psql apis or EBS forms screens , something like this could not be done.. I mean, you just cant set a profile  to a abnormal value using Standart screens..
Also, finding orgid profile values changed to "Y" made me think that this may be a widespread problem.. Thus, there could be some other profiles which may also be affected.. Especially some profiles which might have affect loading of classes in some way..(remembering the actual error : java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.WebAppsContext)

For the solution;
I used flashback query, get the rows from the unchanged fnd_profile_option_values using "as of timestamp option"  and created a table named fnd_profile_option_values_erm.. Then I renamed the fnd_profile_option_values to fnd_profile_option_values_err and renamed fnd_profile_options_erm to fnd_profile_otpions..
Lastly , compiled the invalid objects and started the application services..
That was all.. The problem have dissapeared..

 In conclusion;

We have seen that a problem/unexpected value in profile options of EBS  may lead to service level problems..
In such problems, the cause may not be the things which come our minds in the first thoughts. As you see above, a situation which may be seen like a classpath problem, may actually be a profile problem..
EBS is complex system, and almost everyting is connected with everyting.. Keep in mind that..
Lastly, It is needles to say, updating data using generic update statements is dangerous in EBS :)