Tuesday, October 24, 2017

FMW -- 12C Forms and Reports Cluster installation (2 node, High Available)

Recently, installed a Forms & Reports 12C Cluster on Oracle Linux servers.
This installation was actually a pure FMW installation, rather than the FMW deployment that comes built in with EBS.
It was a little tricky, but at the end of the day; it completed successfully.


Here is the list of the components that were used along with their versions;

FMW 12.2.1.3 (latest version) for Solaris (FMW infra 12.2.1.3.0 )
Certified jdk: 1.8.0_131 or higher (64 bit)
Following Solaris packages (installed on Solaris servers):

SRU 11.3.3.6.0+
SUNWlibC
developer/assembler
libxp
motif

An Oracle database to be used for placing the RCU schemas. (11.2.0.4+)

Client Side - Browser: Microsoft Edge 40.*, Microsoft Internet Explorer 11.*, Google Chrome 60+,Mozilla Firefox 52+,Apple Safari 9.* vor Apple Safari 10.*

RCU -> Required for Forms and Reports RCU schemas Its version should be the same as, FMW infra version. In 12C, RCU comes built it with FMW.. (no need to download it seperately)

Forms_report_binary/installer : "Oracle Forms and Reports 12c (12.2.1.3.0)", for Solaris Sparc 64 bit

I did the installation in 6 steps..

Installation steps:

Install FMW INFRA on both of the nodes.
Install Forms and Reports on both of the nodes.
Create Forms and Reports related Database schemas using RCU. (only on 1 node, first node)
Configure Weblogic Domain using config.sh (only on one node, first node)
PACK and UNPACK the domain (Pack on node1, unpack on node2)
Start the services
Do the tests and fix the problems (if they exist)

Okay.. Let's take a closer look at the installation process ->

FMW infra installation:

We first install FMW Infra (on both of the nodes)

-bash-4.4$ export JAVA_HOME=/u01/java/jdk1.8.0
-bash-4.4$ export PATH=$JAVA_HOME/bin:$PATH

















Next, we install Form and Reports 12c (on both of the nodes)

Forms & Reports 12C installation:

We unzip the package in both of the nodes and run ./fmw_12.2.1.3.0_fr_solaris_sparc64.bin.. (this was a Solaris env)

Note: We first install motif and libXp packages to both of our Solaris nodes to not to get forms makefile errors during the installation.




















Next, we use RCU to place our Forms and Reports related schemas in to our database.. (RCU is executed on one of the nodes)

Using RCU to create the database schemas:

Note: We don't need to download RCU for FMW 12C.. RCU comes built in with FMW 12C.

Note: There is no need to create tablespaces for RCU schemas, beforehand.. RCU creates them during its run automatically.

cd /u01/FMWHOME/oracle_home/oracle_common/bin
./rcu

















Next, we config our Weblogic Domain..
This step is done only on node 1.

config.sh run:























































Note that, we have a bug to bypass..

FMW 12.2.1.3.0 ships with JDBC driver 12.2.0.1.0 where OMS error can occur after the driver FAN is auto-enabled.

The issue is reported in unpublished Bug 26045997 : ENABLING DRIVER FAN WITHOUT RUNNING ONS DAEMONS CAUSES CONNECT REQUEST ERROR.

In order to not to get these ONS errors we modify the config_internal.sh (set the fanEnabled to false)..

If we don't do this, we get ONS related errors as seen in the below screenshot
inside /u01/FMWHOME/oracle_home/oracle_common/common/bin/config.internal.sh ->

JVM_ARGS="-Dpython.cachedir=/tmp/cachedir ${JVM_D64} ${UTILS_MEM_ARGS} ${SECURITY_JVM_ARGS} ${CONFIG_JVM_ARGS}"
if [ -d "${JAVA_HOME}" ]; then
eval '"${JAVA_HOME}/bin/java"' -Doracle.jdbc.fanEnabled=false ${JVM_ARGS} com.oracle.cie.wizard.WizardController '"$@"' ${CAM_ARGUMENTS}
fi






Next we pack the domain from Node 1 and unpack it in Node 2 ..

PACK & UNPACK:

In NODE1 ->

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./pack.sh -managed=true -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain -template=/u01/FMWHOME/frsdomain.jar -template_name=frsdomainTemplate

<< read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
>> succeed: read domain from "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
<< set config option Managed to "true"
>> succeed: set config option Managed to "true"
<< write template to "/u01/FMWHOME/frsdomain.jar"
..............................
>> succeed: write template to "/u01/FMWHOME/frsdomain.jar"
<< close template
>> succeed: close template


In NODE2 ->

scp oracle@192.168.1.69:/u01/FMWHOME/frsdomain.jar /u01/FMWHOME
The authenticity of host '192.168.1.69 (192.168.1.69)' can't be established.
RSA key fingerprint is 9f:3d:b4:10:60:a7:f0:1f:ba:bb:da:42:6f:6e:2e:c2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.69' (RSA) to the list of known hosts.
Password:
frsdomain.jar 100% |******************************************************************************************************************************| 1232 KB 00:00

cd /u01/FMWHOME/oracle_home/oracle_common/common/bin
./unpack.sh -domain=/u01/FMWHOME/oracle_home/user_projects/domains/base_domain template=/u01/FMWHOME/frsdomain.jar -log_priority=DEBUG -log=/tmp/unpack.log -app_dir=/u01/FMWHOME/oracle_home/user_projects/applications/base_domain

<< read template from "/u01/FMWHOME/frsdomain.jar"
>> succeed: read template from "/u01/FMWHOME/frsdomain.jar"
<< set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
>> succeed: set config option AppDir to "/u01/FMWHOME/oracle_home/user_projects/applications/base_domain"
<< set config option DomainName to "base_domain"
>> succeed: set config option DomainName to "base_domain"
>> validateConfig "KeyStorePasswords"
>> succeed: validateConfig "KeyStorePasswords"
<< write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"
...........................................................................
>> succeed: write Domain to "/u01/FMWHOME/oracle_home/user_projects/domains/base_domain"

<< close template
>> succeed: close template

* At this point, we start our services.. (optionally, we may configure a load balancer in front of our Http Servers..)

After starting the services, we may get some errors actually. So, we may need to do some extra work to have a stable environment.

Here are my notes about these known issues:

Known Issues & Solutions:

Note 1:

Again: Bug 26045997 -- We disable the FAN/ONS in all the managed server nodes.



Note 2:

After pack and unpack operations, the config.xml is not built on node2. This can be a bug, but it is not documented.
In order to solve this, we start the managed servers using startManagedWeblogic.sh by specifying an ADMIN_URL.
After this move, the config.xml in node 2 gets created and subsequent start/stop operations can be done using the weblogic console.
In this step, we also do the FAN disabling thing in startManagedWeblogic.sh to disable ons..
Again: Bug 26045997

We write the following in startManagedWeblogic.sh->

export JAVA_OPTIONS=” -Doracle.jdbc.fanEnabled=false”:$JAVA_OPTIONS
Example of start commands: (“script <managed_server_adı> <admin_url>”)

Examples:

sh startManagedWebLogic.sh WLS_FORMS1 http://forms01:7001
sh startManagedWebLogic.sh WLS_REPORTS1 http://forms01:7001

Note 3:

In some cases, the health state of Managed servers in node 2 can be "not reachable".
However, if they are started using startmanagedweblogic.sh, their status seem okay.
In other words, their Health status become Not Reachable, if they are started from the weblogic console.

In order to fix this, I first tried setting Invocation timeout to 10..

Weblogic Console > base domain > Configuration > General > Advanced > Invocation Timeout Seconds = 10

However, this move didn't solve it.

As for the solution, we set the listen address for Admin server and restart everything.

Note 4:

Again: Bug 26045997
We disable FAN/ONS in startWeblogic.sh.. If we don't do this, admin server can not be started because of ONS errors.

JAVA_OPTIONS -Doracle.jdbc.fanEnabled=false

Note 5:

In order to be able to start the OHS components without supplying password everytime, the following script is executed once..

/u01/FMWHOME/oracle_home/user_projects/domains/base_domain/bin/startComponent.sh ohs1 storeUserConfig
--we give password when prompted and it is saved for the subsequent executions.

Note 6:

We just cant start OHS on node2 (OHS2) using startComponent.sh script. This is a restriction of Oracle. That's why, we need to use WLST or FMW console.

Here is an example of doing it via WLST ->

cd $FMW_HOME/oracle_common/common/bin
./wlst.sh

WLST> nmConnect('nodemanager','xxxxx','forms02.oracle.com’,'5556','base_domain','/u01/FMWHOME/oracle_home/user_projects/domains/base_domain','ssl'); ## at this point, we connect to thenodemanager ##of node 2.
nmStart(serverName='ohs2', serverType='OHS');

Note 7:

We just enable the ssh equivalency between our nodes to ease the scripting works that were done for starting and stopping the whole stack from one node in one go.

Well.. After the installation, we can check the forms and reports services to ensure that they are running successfuly.

In order to do these tests, we use the following urls:

Forms: http://ip_adress:port/forms/frmservlet
Reports: http://ip_adress:port/reports/rwservlet

We do these test for both of the nodes and expect to see the following outputs in our browsers:




That 's it :)

Ohh, almost forgot.
Here is a handy script that I write for controlling this cluster and all the services accross nodes easily ->
FMW -- Starting/ Stopping a 2 Node Forms&Reports 12C Cluster with a single command. SCRIPTS.. Automated start/stop for High Available FMW environments

10 comments :

  1. Excellent Post, congratullations!!

    ReplyDelete
  2. wls:/nm/APPS1> nmStart(serverName='ohs2', serverType='OHS');
    Starting server ohs2 ...
    Traceback (innermost last):
    File "", line 1, in ?
    File "", line 188, in nmStart
    File "", line 553, in raiseWLSTException
    WLSTException: Error occurred while performing nmStart : Error Starting server ohs2 : Received error message from Node Manager Server: [Server start command for OHS server 'ohs2' failed due to: [Failed to start the server ohs2
    Check log file /u01/Oracle/Middleware/Oracle_Home/user_projects/domains/APPS1/system_components/OHS/ohs_nm.log]. Please check Node Manager log and/or server 'ohs2' log for detailed information.]. Please check Node Manager log for details.
    Use dumpStack() to view the full stacktrace

    ReplyDelete
  3. Hussain,

    Did you connect to the node manager of node 2?

    Example:

    WLST> nmConnect('nodemanager','xxxxx','forms02.oracle.com’,'5556','base_domain','/u01/FMWHOME/oracle_home/user_projects/domains/base_domain','ssl'); ## at this point, we connect to thenodemanager ##of node 2.

    nmStart(serverName='ohs2', serverType='OHS');

    ReplyDelete
  4. Thanks for the update the issue is resolved

    ReplyDelete
  5. REP-52266: The in-process Reports Server rep_wls_reports_apps1 failed to start.oracle.reports.RWException: IDL:oracle/reports/RWException:1.0

    ReplyDelete
  6. Hussain,

    If you will ask a question, please don't comment here..

    For your questions, please create an issue into my forum.

    Forum Link: http://ermanarslan.blogspot.com.tr/p/forum.html

    Register and create an issue in the related category.
    I will support you from there.

    ReplyDelete
  7. HI Sir. The image of the blog don't appear.

    ReplyDelete
  8. Hi Can you please share FMW -- 12C Forms and Reports Cluster installation (2 node, High Available) document I'am unable to view the screen shot attached in this document .

    ReplyDelete
  9. Great job. But I have a question as I can raise the component reptools in node 2, since in node 1, through scripting tool you create it without problems, but in node 2, since you have only one console that of node1, you connect to it console and you get the error saying that it has been created before.

    regards
    Jorge Bonete

    ReplyDelete
  10. kindly send images to my email Qebraheem@hotmail.com
    or reupload good quality image please help

    ReplyDelete

If you will ask a question, please don't comment here..

For your questions, please create an issue into my forum.

Forum Link: http://ermanarslan.blogspot.com.tr/p/forum.html

Register and create an issue in the related category.
I will support you from there.