FMW -- 12C Forms and Reports Cluster installation (2 node, High Available)

Recently, installed a Forms & Reports 12C Cluster on Oracle Linux servers.
This installation was actually a pure FMW installation, rather than the FMW deployment that comes built in with EBS.
It was a little tricky, but at the end of the day; it completed successfully.

Here is the list of the components that were used along with their versions;

FMW 12.2.1.3 (latest version) for Solaris (FMW infra 12.2.1.3.0 )
Certified jdk: 1.8.0_131 or higher (64 bit)
Following Solaris packages (installed on Solaris servers):

SRU 11.3.3.6.0+
SUNWlibC
developer/assembler
libxp
motif

An Oracle database to be used for placing the RCU schemas. (11.2.0.4+)

Client Side - Browser: Microsoft Edge 40.*, Microsoft Internet Explorer 11.*, Google Chrome 60+,Mozilla Firefox 52+,Apple Safari 9.* vor Apple Safari 10.*

RCU -> Required for Forms and Reports RCU schemas Its version should be the same as, FMW infra version. In 12C, RCU comes built it with FMW.. (no need to download it seperately)

Forms_report_binary/installer : "Oracle Forms and Reports 12c (12.2.1.3.0)", for Solaris Sparc 64 bit

I did the installation in 6 steps..

Installation steps:

Install FMW INFRA on both of the nodes.
Install Forms and Reports on both of the nodes.
Create Forms and Reports related Database schemas using RCU. (only on 1 node, first node)
Configure Weblogic Domain using config.sh (only on one node, first node)
PACK and UNPACK the domain (Pack on node1, unpack on node2)
Start the services
Do the tests and fix the problems (if they exist)

Okay.. Let's take a closer look at the installation process ->

FMW infra installation:

We first install FMW Infra (on both of the nodes)

-bash-4.4$ export JAVA_HOME=/u01/java/jdk1.8.0
-bash-4.4$ export PATH=$JAVA_HOME/bin:$PATH

Next, we install Form and Reports 12c (on both of the nodes)

Forms & Reports 12C installation:

We unzip the package in both of the nodes and run ./fmw_12.2.1.3.0_fr_solaris_sparc64.bin.. (this was a Solaris env)

Note: We first install motif and libXp packages to both of our Solaris nodes to not to get forms makefile errors during the installation.

Next, we use RCU to place our Forms and Reports related schemas in to our database.. (RCU is executed on one of the nodes)

Using RCU to create the database schemas:

Note: We don't need to download RCU for FMW 12C.. RCU comes built in with FMW 12C.

Note: There is no need to create tablespaces for RCU schemas, beforehand.. RCU creates them during its run automatically.

cd /u01/FMWHOME/oracle_home/oracle_common/bin
./rcu

Next, we config our Weblogic Domain..
This step is done only on node 1.

config.sh run:

Note that, we have a bug to bypass..

FMW 12.2.1.3.0 ships with JDBC driver 12.2.0.1.0 where OMS error can occur after the driver FAN is auto-enabled.

The issue is reported in unpublished Bug 26045997 : ENABLING DRIVER FAN WITHOUT RUNNING ONS DAEMONS CAUSES CONNECT REQUEST ERROR.

In order to not to get these ONS errors we modify the config_internal.sh (set the fanEnabled to false)..

If we don't do this, we get ONS related errors as seen in the below screenshot
inside /u01/FMWHOME/oracle_home/oracle_common/common/bin/config.internal.sh ->

JVM_ARGS="-Dpython.cachedir=/tmp/cachedir ${JVM_D64} ${UTILS_MEM_ARGS} ${SECURITY_JVM_ARGS} ${CONFIG_JVM_ARGS}"
if [ -d "${JAVA_HOME}" ]; then
eval '"${JAVA_HOME}/bin/java"' -Doracle.jdbc.fanEnabled=false ${JVM_ARGS} com.oracle.cie.wizard.WizardController '"$@"' ${CAM_ARGUMENTS}
fi

Next we pack the domain from Node 1 and unpack it in Node 2 ..

PACK & UNPACK:

In NODE1 ->

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./pack.sh -managed=true -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain -template=/u01/FMWHOME/frsdomain.jar -template_name=frsdomainTemplate

<< read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
>> succeed: read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
<< set config option Managed to "true"
>> succeed: set config option Managed to "true"
<< write template to "/u01/FMWHOME/frsdomain.jar"
..............................
>> succeed: write template to "/u01/FMWHOME/frsdomain.jar"
<< close template
>> succeed: close template


In NODE2 ->

scp oracle@192.168.1.69:/u01/FMWHOME/frsdomain.jar /u01/FMWHOME
The authenticity of host '192.168.1.69 (192.168.1.69)' can't be established.
RSA key fingerprint is 9f:3d:b4:10:60:a7:f0:1f:ba:bb:da:42:6f:6e:2e:c2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.69' (RSA) to the list of known hosts.
Password:
frsdomain.jar 100% |******************************************************************************************************************************| 1232 KB 00:00

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./unpack.sh -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain template=/u01/FMWHOME/frsdomain.jar -log_priority=DEBUG -log=/tmp/unpack.log -app_dir=/u01/FMWHOME/oracle_home/user_projects/applications/base_domain

<< read template from "/u01/FMWHOME/frsdomain.jar"
>> succeed: read template from "/u01/FMWHOME/frsdomain.jar"
<< set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
>> succeed: set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
<< set config option DomainName to "base_domain"
>> succeed: set config option DomainName to "base_domain"
>> validateConfig "KeyStorePasswords"
>> succeed: validateConfig "KeyStorePasswords"
<< write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
...........................................................................
>> succeed: write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"

<< close template
>> succeed: close template

* At this point, we start our services.. (optionally, we may configure a load balancer in front of our Http Servers..)

After starting the services, we may get some errors actually. So, we may need to do some extra work to have a stable environment.

Here are my notes about these known issues:

Known Issues & Solutions:

Note 1:

Again: Bug 26045997 -- We disable the FAN/ONS in all the managed server nodes.

Note 2:

After pack and unpack operations, the config.xml is not built on node2. This can be a bug, but it is not documented.
In order to solve this, we start the managed servers using startManagedWeblogic.sh by specifying an ADMIN_URL.
After this move, the config.xml in node 2 gets created and subsequent start/stop operations can be done using the weblogic console.
In this step, we also do the FAN disabling thing in startManagedWeblogic.sh to disable ons..
Again: Bug 26045997

We write the following in startManagedWeblogic.sh->

export JAVA_OPTIONS=” -Doracle.jdbc.fanEnabled=false”:$JAVA_OPTIONS
Example of start commands: (“script <managed_server_adı> <admin_url>”)

Examples:

sh startManagedWebLogic.sh WLS_FORMS1 http://forms01:7001
sh startManagedWebLogic.sh WLS_REPORTS1 http://forms01:7001

Note 3:

In some cases, the health state of Managed servers in node 2 can be "not reachable".
However, if they are started using startmanagedweblogic.sh, their status seem okay.
In other words, their Health status become Not Reachable, if they are started from the weblogic console.

In order to fix this, I first tried setting Invocation timeout to 10..

Weblogic Console > base domain > Configuration > General > Advanced > Invocation Timeout Seconds = 10

However, this move didn't solve it.

As for the solution, we set the listen address for Admin server and restart everything.

Note 4:

Again: Bug 26045997
We disable FAN/ONS in startWeblogic.sh.. If we don't do this, admin server can not be started because of ONS errors.

JAVA_OPTIONS -Doracle.jdbc.fanEnabled=false

Note 5:

In order to be able to start the OHS components without supplying password everytime, the following script is executed once..

/u01/FMWHOME/oracle_home/user_projects/domains/base_domain/bin/startComponent.sh ohs1 storeUserConfig
--we give password when prompted and it is saved for the subsequent executions.

Note 6:

We just cant start OHS on node2 (OHS2) using startComponent.sh script. This is a restriction of Oracle. That's why, we need to use WLST or FMW console.

Here is an example of doing it via WLST ->

cd $FMW_HOME/oracle_common/common/bin
./wlst.sh

WLST> nmConnect('nodemanager','xxxxx','forms02.oracle.com’,'5556','base_domain','/u01/FMWHOME/oracle_home/user_projects/domains/base_domain','ssl'); ## at this point, we connect to thenodemanager ##of node 2.
nmStart(serverName='ohs2', serverType='OHS');

Note 7:

We just enable the ssh equivalency between our nodes to ease the scripting works that were done for starting and stopping the whole stack from one node in one go.

Well.. After the installation, we can check the forms and reports services to ensure that they are running successfuly.

In order to do these tests, we use the following urls:

Forms: http://ip_adress:port/forms/frmservlet
Reports: http://ip_adress:port/reports/rwservlet

We do these test for both of the nodes and expect to see the following outputs in our browsers:

That 's it :)

Ohh, almost forgot.
Here is a handy script that I write for controlling this cluster and all the services accross nodes easily ->
FMW -- Starting/ Stopping a 2 Node Forms&Reports 12C Cluster with a single command. SCRIPTS.. Automated start/stop for High Available FMW environments