Weblogic -- AppSecurityContext.setApplicationID.null error when using Nodemanager //Diagnostics

You may encounter AppSecurityContext.setApplicationID.null error when starting your Weblogic application in Weblogic console.
It happened to me and I have made a 6 hour diagnostic to find the cause of this error.
On the other hand; once the cause is found, it is easy to take the corrective action.

Actually, in my opinion we dont need to deal with this kind of errors as Apps Dba's or even as Weblogic Admins..
On the other hand; dealing with this kind of errors and correcting them make me happy in a way :).
But I must admit that I started to think that day by day things in Oracle World are getting difficult rather than getting easier...

Okay, lets take a look at the error.

The error stack is something like the following;
java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission AppSecurityContext.setApplicationID.null)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
at java.security.AccessController.checkPermission(AccessController.java:546)
The error appears when starting an Entprise Application deployed in the Weblogic.

The error only appears if the Managed Server that configured to run the Entprise Application is started using the Node Manager.. In other words; if we start the Managed Server using command line(sh startManagedServer ERMANServer t3://ermanhost:7003) ; the error is not seen, Enterprise Application can be started without any problems..

So, as you may guess, the problem is related with the Node Manager, but it is hard to find the root cause as this is a specific java error..

Fortunately, it is possible to understand that it is related with a java security pattern, as it comes from the java.security package.

Anyways , I will keep it short..
One way or another I have found that the problem is caused by a missing environment variable or a missing command line arguments related with the java process that represents the Managed Server.
When the process (Managed Server) is started using Node  Manager, there must be a missing/errornous environment variable or java command line argument which causes this error.

To find this missing argument or lets say environment variable; I have made a test..
I started Managed Server two times and recorded its command line using ps -ef..

Firstly started the Managed Server using NodeManager and recorded its command line.
Then started the Managed Server using startManagedServer script and recorded its command line.
Lastly I compared these two with each other and saw the difference..

The command line of the Manager Server process which was started using startManagedServer.sh was  longer..
Thus by using this testing technique, I concluded the problem..
"The problem was caused by the start command that was used by the Nodemanager to start the Managed Server."

The start command was missing some command line arguments, as it was shorter than the command line of the startManagedServer script.

Then I realized that the error is related with security.. Also realized that the security is maintained using jps-config.xml. This config file is actually the configuration of the Java Policy Store (JPS) it is used to configure Policy, Credential, Key Store, Login Module and User Identity Store location.
It is located in $DOMAIN_HOME/config/fmwconfig, and it is supplied to the java processes using the parameter -Doracle.security.jps.config..

So , when started from the Nodemanager , it was clear that the ManagedServer was missing this config and the Application located in this ManagedServer can not be started because of this.

The solution have been stated when I saw the fine setDomainEnv.sh..
This file was sourced before starting the Managed Server using startManagedServer.sh..

In setDomainEnv.sh  I saw an export of the environment variable EXTRA_JAVA_OPTIONS.
This environment variable was used to set some additional command line arguments like jps-config!

There were several environment variables in EXTRA_JAVA_OPTIONS though..
It was like EXTRA_JAVA_OPTIONS=/erman/java/blabla:$ERMAN:$ERMAN2... and so on.

To have a plain text of it; I used echo $EXTRA_JAVA_OPTIONS just after sourcing the setDomainEnv.sh..

Once I got the plain text, I opened the Weblogic console and write this plain text into the server start tab of the problematic Managed Server..

After I have done this, the error dissapeared. Our ADF Enterprise Application could start even if its Managed Server was started using Node Manager..



Another info that I have gathered is that Node Manager reads the needed command line arguments from the server-start options of related Managed Server. If it cant find any server-start option for a particular Managed Server, it starts the Managed Server with default java arguments and leads this error.

Following is the server-start configuration that I have made to overcome this error.

I hope you will find this useful.