EBS 12.2 -- libjava.so problem in Solaris environment / libjava.so: open failed: No such file or directory

I wrote about the problems that I encountered on EBS 12.2 - Solaris Sparc environments.
I also have given you the solutions, which are found after a great deal of diagnostic works.
I have dealed with forms makefiles, reports makefiles and so on to make EBS 12.2 be stable on Solaris 11.3.
Here is a blog post that I wrote about an interesting problem that I encountered during the initial installation of EBS 12.2 on to a Solaris 11.3 Operating System.
http://ermanarslan.blogspot.com.tr/2017/06/ebs-1220-installation-on-solaris-511.html

However, this one is more interesting :)
This time, I encountered a problem after the installation.
Note that, this problem appear after running adop's cutover phase (even if you are on the latest AD and TXK levels).
Also it may appear in a freshly cloned environment.

The problem was related with Oracle Reports that comes built in with EBS.
When this problem was encountered, the reports can not be run. Any reports related Concurrent requests can not be run successfully(complete with error) and any report related tool, such as rwconventer can not be executed.

They all failed with  libjava.so: open failed: No such file or directory.

Example of the error stack:

Error occurred during initialization of VM

Unable to load native library: ld.so.1: rwrun: fatal: /u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib/libjava.so: open failed: No such file or directory

As for the diagnostics, I reviewed the ins_reports.mk and related env file.. Everything was okay and seemed fine. (sparcv9 related modifications were there already)

I also used ldd command to check the related binaries and libraries. No trails of "/u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib/libjava.so"...

Actually, in Solaris, we have this libjava.so file in "<10.1.2_Oracle_HOME>/jdk/jre/lib/sparc/" and this location is correct, but somehow reports executables like rwconverter wanted to use it from "/u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib."

I checked it from various places and concluded that this is not configurable..

But, at the end of the day, it was not normal and needed to be fixed...

Then I made a research on the libjava.so file and gathered the following info about it;

It is a shared library and used when you need to  invoke the Java Virtual Machine from your own code. For ex: a C program that invokes the Java Virtual Machine and calls the Erman.main method defined in Erm.java..

So, in order to be able to do this, you need to compile your C program with java libraries, that comes with JDK.

libjava.so is closely related with libjvm.so. You can think like, one of them is for creating the virtual machines and other one is for loading classses. Probably libjava.so is loaded in the jvm startup..

So, once I gathered this info, I started to think that there may a JDK related problem, a wrong library link or something like that in this environment.

After trying lots of things (rebuild reports, relinking binaries and so on), I decided to recreate the JDK that comes with the EBS 12.2 installation.

I aimed at the JDK, which was located in 10.1.2 Oracle Home, because the problem was there.

As a solution, I did a fake JDK upgrade..

That is, I installed the same JDK version once again to the EBS 12.2's 10.1.2 Oracle Home using the document: "Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2 (Doc ID 1530033.1)" -> "Section 4: Upgrading to Latest Java 7.0 in OracleAS 10.1.2 Oracle_Home"

Remember: startCD 12.2.0.47 or higher delivers JDK 7

For our case, it was 1.7.0.85

-bash-4.4$ ./java -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Server VM (build 24.85-b06, mixed mode)

-bash-4.4$ pwd

/u01/app/fs2/EBSapps/10.1.2/jdk/jre/bin

So, I downloaded JDK 1.7.0.85 for Solaris and took the following actions to install it.

soure run env.

cd $ORACLE_HOME

$ mv jdk jdk_old 

$ mv jdk1.7.0_85 jdk 

$ rm -rf jdk_old

That was it.. This move solved the libjava.so problems.. (no other modifications needed, no autoconfig or nothing) 

So it was caused by a misconfiguration in JDK, a wrong library link maybe..

What a hard issue it was... It resolved easily but the diagnostic work and the effort that I given for that, was huge..

I hope you will find this undocumented solution useful. See you in my next blog post :)

One last thing, the issue is documented in "Unable To Load native library: ld.so.1: rwrun: fatal: (Doc ID 1529558.1)", but the provided there was odd and irrelevant , at least for this case..