EBS 12.2 -- libjava.so problem in Solaris environment / libjava.so: open failed: No such file or directory

I wrote about the problems that I encountered on EBS 12.2 - Solaris Sparc environments.
I also have given you the solutions, which are found after a great deal of diagnostic works.
I have dealed with forms makefiles, reports makefiles and so on to make EBS 12.2 be stable on Solaris 11.3.
Here is a blog post that I wrote about an interesting problem that I encountered during the initial installation of EBS 12.2 on to a Solaris 11.3 Operating System.
http://ermanarslan.blogspot.com.tr/2017/06/ebs-1220-installation-on-solaris-511.html

However, this one is more interesting :)
This time, I encountered a problem after the installation.
Note that, this problem appear after running adop's cutover phase (even if you are on the latest AD and TXK levels).
Also it may appear in a freshly cloned environment.

The problem was related with Oracle Reports that comes built in with EBS.
When this problem was encountered, the reports can not be run. Any reports related Concurrent requests can not be run successfully(complete with error) and any report related tool, such as rwconventer can not be executed.

They all failed with  libjava.so: open failed: No such file or directory.

Example of the error stack:

Error occurred during initialization of VM

Unable to load native library: ld.so.1: rwrun: fatal: /u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib/libjava.so: open failed: No such file or directory

As for the diagnostics, I reviewed the ins_reports.mk and related env file.. Everything was okay and seemed fine. (sparcv9 related modifications were there already)

I also used ldd command to check the related binaries and libraries. No trails of "/u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib/libjava.so"...

Actually, in Solaris, we have this libjava.so file in "<10.1.2_Oracle_HOME>/jdk/jre/lib/sparc/" and this location is correct, but somehow reports executables like rwconverter wanted to use it from "/u01/app/fs1/EBSapps/10.1.2/jdk/jre/lib."

I checked it from various places and concluded that this is not configurable..

But, at the end of the day, it was not normal and needed to be fixed...

Then I made a research on the libjava.so file and gathered the following info about it;

It is a shared library and used when you need to  invoke the Java Virtual Machine from your own code. For ex: a C program that invokes the Java Virtual Machine and calls the Erman.main method defined in Erm.java..

So, in order to be able to do this, you need to compile your C program with java libraries, that comes with JDK.

libjava.so is closely related with libjvm.so. You can think like, one of them is for creating the virtual machines and other one is for loading classses. Probably libjava.so is loaded in the jvm startup..

So, once I gathered this info, I started to think that there may a JDK related problem, a wrong library link or something like that in this environment.

After trying lots of things (rebuild reports, relinking binaries and so on), I decided to recreate the JDK that comes with the EBS 12.2 installation.

I aimed at the JDK, which was located in 10.1.2 Oracle Home, because the problem was there.

As a solution, I did a fake JDK upgrade..

That is, I installed the same JDK version once again to the EBS 12.2's 10.1.2 Oracle Home using the document: "Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2 (Doc ID 1530033.1)" -> "Section 4: Upgrading to Latest Java 7.0 in OracleAS 10.1.2 Oracle_Home"

Remember: startCD 12.2.0.47 or higher delivers JDK 7

For our case, it was 1.7.0.85

-bash-4.4$ ./java -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Server VM (build 24.85-b06, mixed mode)

-bash-4.4$ pwd

/u01/app/fs2/EBSapps/10.1.2/jdk/jre/bin

So, I downloaded JDK 1.7.0.85 for Solaris and took the following actions to install it.

soure run env.

cd $ORACLE_HOME

$ mv jdk jdk_old 

$ mv jdk1.7.0_85 jdk 

$ rm -rf jdk_old

That was it.. This move solved the libjava.so problems.. (no other modifications needed, no autoconfig or nothing) 

So it was caused by a misconfiguration in JDK, a wrong library link maybe..

What a hard issue it was... It resolved easily but the diagnostic work and the effort that I given for that, was huge..

I hope you will find this undocumented solution useful. See you in my next blog post :)

One last thing, the issue is documented in "Unable To Load native library: ld.so.1: rwrun: fatal: (Doc ID 1529558.1)", but the provided there was odd and irrelevant , at least for this case..

FMW -- 12C Forms and Reports Cluster installation (2 node, High Available)

Recently, installed a Forms & Reports 12C Cluster on Oracle Linux servers.
This installation was actually a pure FMW installation, rather than the FMW deployment that comes built in with EBS.
It was a little tricky, but at the end of the day; it completed successfully.

Here is the list of the components that were used along with their versions;

FMW 12.2.1.3 (latest version) for Solaris (FMW infra 12.2.1.3.0 )
Certified jdk: 1.8.0_131 or higher (64 bit)
Following Solaris packages (installed on Solaris servers):

SRU 11.3.3.6.0+
SUNWlibC
developer/assembler
libxp
motif

An Oracle database to be used for placing the RCU schemas. (11.2.0.4+)

Client Side - Browser: Microsoft Edge 40.*, Microsoft Internet Explorer 11.*, Google Chrome 60+,Mozilla Firefox 52+,Apple Safari 9.* vor Apple Safari 10.*

RCU -> Required for Forms and Reports RCU schemas Its version should be the same as, FMW infra version. In 12C, RCU comes built it with FMW.. (no need to download it seperately)

Forms_report_binary/installer : "Oracle Forms and Reports 12c (12.2.1.3.0)", for Solaris Sparc 64 bit

I did the installation in 6 steps..

Installation steps:

Install FMW INFRA on both of the nodes.
Install Forms and Reports on both of the nodes.
Create Forms and Reports related Database schemas using RCU. (only on 1 node, first node)
Configure Weblogic Domain using config.sh (only on one node, first node)
PACK and UNPACK the domain (Pack on node1, unpack on node2)
Start the services
Do the tests and fix the problems (if they exist)

Okay.. Let's take a closer look at the installation process ->

FMW infra installation:

We first install FMW Infra (on both of the nodes)

-bash-4.4$ export JAVA_HOME=/u01/java/jdk1.8.0
-bash-4.4$ export PATH=$JAVA_HOME/bin:$PATH

Next, we install Form and Reports 12c (on both of the nodes)

Forms & Reports 12C installation:

We unzip the package in both of the nodes and run ./fmw_12.2.1.3.0_fr_solaris_sparc64.bin.. (this was a Solaris env)

Note: We first install motif and libXp packages to both of our Solaris nodes to not to get forms makefile errors during the installation.

Next, we use RCU to place our Forms and Reports related schemas in to our database.. (RCU is executed on one of the nodes)

Using RCU to create the database schemas:

Note: We don't need to download RCU for FMW 12C.. RCU comes built in with FMW 12C.

Note: There is no need to create tablespaces for RCU schemas, beforehand.. RCU creates them during its run automatically.

cd /u01/FMWHOME/oracle_home/oracle_common/bin
./rcu

Next, we config our Weblogic Domain..
This step is done only on node 1.

config.sh run:

Note that, we have a bug to bypass..

FMW 12.2.1.3.0 ships with JDBC driver 12.2.0.1.0 where OMS error can occur after the driver FAN is auto-enabled.

The issue is reported in unpublished Bug 26045997 : ENABLING DRIVER FAN WITHOUT RUNNING ONS DAEMONS CAUSES CONNECT REQUEST ERROR.

In order to not to get these ONS errors we modify the config_internal.sh (set the fanEnabled to false)..

If we don't do this, we get ONS related errors as seen in the below screenshot
inside /u01/FMWHOME/oracle_home/oracle_common/common/bin/config.internal.sh ->

JVM_ARGS="-Dpython.cachedir=/tmp/cachedir ${JVM_D64} ${UTILS_MEM_ARGS} ${SECURITY_JVM_ARGS} ${CONFIG_JVM_ARGS}"
if [ -d "${JAVA_HOME}" ]; then
eval '"${JAVA_HOME}/bin/java"' -Doracle.jdbc.fanEnabled=false ${JVM_ARGS} com.oracle.cie.wizard.WizardController '"$@"' ${CAM_ARGUMENTS}
fi

Next we pack the domain from Node 1 and unpack it in Node 2 ..

PACK & UNPACK:

In NODE1 ->

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./pack.sh -managed=true -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain -template=/u01/FMWHOME/frsdomain.jar -template_name=frsdomainTemplate

<< read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
>> succeed: read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
<< set config option Managed to "true"
>> succeed: set config option Managed to "true"
<< write template to "/u01/FMWHOME/frsdomain.jar"
..............................
>> succeed: write template to "/u01/FMWHOME/frsdomain.jar"
<< close template
>> succeed: close template


In NODE2 ->

scp oracle@192.168.1.69:/u01/FMWHOME/frsdomain.jar /u01/FMWHOME
The authenticity of host '192.168.1.69 (192.168.1.69)' can't be established.
RSA key fingerprint is 9f:3d:b4:10:60:a7:f0:1f:ba:bb:da:42:6f:6e:2e:c2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.69' (RSA) to the list of known hosts.
Password:
frsdomain.jar 100% |******************************************************************************************************************************| 1232 KB 00:00

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./unpack.sh -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain template=/u01/FMWHOME/frsdomain.jar -log_priority=DEBUG -log=/tmp/unpack.log -app_dir=/u01/FMWHOME/oracle_home/user_projects/applications/base_domain

<< read template from "/u01/FMWHOME/frsdomain.jar"
>> succeed: read template from "/u01/FMWHOME/frsdomain.jar"
<< set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
>> succeed: set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
<< set config option DomainName to "base_domain"
>> succeed: set config option DomainName to "base_domain"
>> validateConfig "KeyStorePasswords"
>> succeed: validateConfig "KeyStorePasswords"
<< write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
...........................................................................
>> succeed: write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"

<< close template
>> succeed: close template

* At this point, we start our services.. (optionally, we may configure a load balancer in front of our Http Servers..)

After starting the services, we may get some errors actually. So, we may need to do some extra work to have a stable environment.

Here are my notes about these known issues:

Known Issues & Solutions:

Note 1:

Again: Bug 26045997 -- We disable the FAN/ONS in all the managed server nodes.

Note 2:

After pack and unpack operations, the config.xml is not built on node2. This can be a bug, but it is not documented.
In order to solve this, we start the managed servers using startManagedWeblogic.sh by specifying an ADMIN_URL.
After this move, the config.xml in node 2 gets created and subsequent start/stop operations can be done using the weblogic console.
In this step, we also do the FAN disabling thing in startManagedWeblogic.sh to disable ons..
Again: Bug 26045997

We write the following in startManagedWeblogic.sh->

export JAVA_OPTIONS=” -Doracle.jdbc.fanEnabled=false”:$JAVA_OPTIONS
Example of start commands: (“script <managed_server_adı> <admin_url>”)

Examples:

sh startManagedWebLogic.sh WLS_FORMS1 http://forms01:7001
sh startManagedWebLogic.sh WLS_REPORTS1 http://forms01:7001

Note 3:

In some cases, the health state of Managed servers in node 2 can be "not reachable".
However, if they are started using startmanagedweblogic.sh, their status seem okay.
In other words, their Health status become Not Reachable, if they are started from the weblogic console.

In order to fix this, I first tried setting Invocation timeout to 10..

Weblogic Console > base domain > Configuration > General > Advanced > Invocation Timeout Seconds = 10

However, this move didn't solve it.

As for the solution, we set the listen address for Admin server and restart everything.

Note 4:

Again: Bug 26045997
We disable FAN/ONS in startWeblogic.sh.. If we don't do this, admin server can not be started because of ONS errors.

JAVA_OPTIONS -Doracle.jdbc.fanEnabled=false

Note 5:

In order to be able to start the OHS components without supplying password everytime, the following script is executed once..

/u01/FMWHOME/oracle_home/user_projects/domains/base_domain/bin/startComponent.sh ohs1 storeUserConfig
--we give password when prompted and it is saved for the subsequent executions.

Note 6:

We just cant start OHS on node2 (OHS2) using startComponent.sh script. This is a restriction of Oracle. That's why, we need to use WLST or FMW console.

Here is an example of doing it via WLST ->

cd $FMW_HOME/oracle_common/common/bin
./wlst.sh

WLST> nmConnect('nodemanager','xxxxx','forms02.oracle.com’,'5556','base_domain','/u01/FMWHOME/oracle_home/user_projects/domains/base_domain','ssl'); ## at this point, we connect to thenodemanager ##of node 2.
nmStart(serverName='ohs2', serverType='OHS');

Note 7:

We just enable the ssh equivalency between our nodes to ease the scripting works that were done for starting and stopping the whole stack from one node in one go.

Well.. After the installation, we can check the forms and reports services to ensure that they are running successfuly.

In order to do these tests, we use the following urls:

Forms: http://ip_adress:port/forms/frmservlet
Reports: http://ip_adress:port/reports/rwservlet

We do these test for both of the nodes and expect to see the following outputs in our browsers:

That 's it :)

Ohh, almost forgot.
Here is a handy script that I write for controlling this cluster and all the services accross nodes easily ->
FMW -- Starting/ Stopping a 2 Node Forms&Reports 12C Cluster with a single command. SCRIPTS.. Automated start/stop for High Available FMW environments