Weblogic -- Performance problem - Forms & Reports environment -- Unable to load performance pack / libmuxer.so

Recently, analyzed a problem on a Weblogic instance.

There was a Forms & Reports - based custom program, running o Weblogic 10.3.6 and the customer was complaining about the performance.

Every single form screen was working slowly..
The problem was obvious, it was on the application tier, as there were no real database activitiy.

Customer said that, this program was previously running with 1500 users on a Oracle Application Server 10g environment, and there were no performance problems encountered there.

Weblogic instance was running on a Solaris OS.

I first checked the configuration from the weblogic console, and concluded that it was all fine.
Just in case, I increased the heap sizes of the managed servers and restarted them..However;  this action didn't solve the problem.

So I jumped into the log files.

While analyzing the WLS_FORMS.log, I saw a strange error..
Weblogic managed server was complaning about the performance pack.. It was saying "Unable to load performance pack".
When this happens, Weblogic starts to use the Java I/O, rather than the native one.

These kinds of problems, are usually caused by the LD_LIBRARY_PATH.

In order to solve this; I first found the necessary shared library that is supposed to be used for enabling the native/IO - performance pack.

The name of the library libmuxer.so, and it was located in the directory named "/app01/weblogic/wlserver_10.3/server/native/solaris/sparc64/".

I modified the <DOMAIN_HOME>/bin/setDomainEnv.sh file and made the LD_LIBRARY_PATH (for the Forms server) include the location of the libmuxer.so file.

After this modification, I restarted the Weblogic Managed server and checked the WLS_FORMS.log  file once again to see whether the error was dissapeared or not.

The error wasn't there. So , I checked the application and saw those waits were gone.. The forms screens were working perfectly fine.. :)

This was like a surgical operation , so I liked the work that I have done, and wanted to share this with you :)

Hope , you will find it useful.

RDBMS / RAC / EXADATA-- switching Scan names -- removing the need for modifying the connection strings after a migration

Recently started a big " ECM ( Exadata Cloud Machine Migration) - OCM (Oracle Cloud Machine)" migration project..

In this exciting project, I will play the lead consultant role for both database and application migrations..

Databases in the scope are Oracle 11gR2 databases, and they all will be migrated to ECMs.

Applications, on the other hands, are running on WebSphere and they all will be migrated to the WebLogic instances running on OCMs.

My team is responsible the project as an whole. From analysis, to Planning & From planning to Execution..

As you may guess, the downtime is very important..

In other words, the methods that we should use for the migration of these databases, should require a minimum downtime..

The platforms of the databases, which are in the scope of this project, varies..

That is, some of these Oracle databases are running on AIX-IBM Power platforms and some of them on LINUX-INTEL Platforms.

As a result of these varying platforms & the need for having the minimum downtime; we are working on several migration plans.

For Linux-Intel Platform, we are mainly focused on the Dataguard based migration strategies, and this blog post will be based on a little method that we used while doing one of our migration POCs for proving Oracle's Dataguard technology.

Most of you may already know that, in order to migrate a database using Dataguard switchover or failover methods, we first create a standby database in the target site.

Then we use Oracle's managed recovery (Dataguard) to make it be sync with the primary.

Once the standby database becomes sync with the primary, we schedule a small downtime and do the switchover operation in this planned maintanence window.

After a successful switchover operation, our new primary starts running in the target site. (read-write)

This new primary will be running on a different platform, with different "management ip addresses, hostnames, virtual hostnames, virtual ip addresses, scan ip addresses and scan names".

As a result, we normally tell our clients & application owners to change their connections strings (java jdbc urls, tnsnames.ora and etc..) accordingly.

However, what we have done in one of our POCs, was a little different..

I mean, we did a switchover and make the primary database be activated on the target site.. But after that, we didn't tell our clients and application owners to change their connection strings..

There was no need for that..

Why? Because, we switched the scan names between the target and the source platform.

Here is the time flow of this operation:

t0: exax6 's scan name: exaxxx & exax7's scan name: exax7
t1: exax6 's scan name : exax6old & exax7's scan name: exax7
t2: exax6 's scan name : exax6old & exax7's scan name: exaxxx

So the new primary (its scan listeners actually) started listening on the same scan name, as the old primary. ( Although the scan IP addresses of the new primary is different than the old primary)

By doing this, we gained time & we didn't spend any extra efforts for changing the application connection strings..

Well... Let's look at the technical side of this operation;

Our source platform was an Exadata X6-2, and our target platform was an Exadata X7-2.

So, after the Dataguard-based switchover operation, we did the following ;

***We first changed the scan name of the source platform (Exa X6-2) and make it exax6old.. (it was exaxxx before this change)..

In order to do this, we took the following actions;

--Stopped the scan listeners and scan itself using Grid user (in our case it was oracle)

$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl stop scan

--We said the DNS team to update scan-related DNS definitions of Exa X6-2 and made them exax6old.

--Then, we checked our server (Exadata X6's db nodes) and confirmed that these new hostnames could be resolved .

-- modified the scan name as root user

$GRID_HOME/bin/srvctl modify scan -n exax6old

--made the modification for the scan resources & confirmed the change, -- as the grid user.

$GRID_HOME/bin/srvctl modify scan_listener -u
$GRID_HOME/bin/srvctl start scan_listener
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config scan_listener

*** After changing the scan name of our source platform, the old scan name of the source platform became available to be used for our target platform.

--So, this time in target (Exa X7-2), we stopped scan listener and scan itself using Grid user (in our case it was oracle)

$GRID_HOME/bin/srvctl stop scan_listener
$GRID_HOME/bin/srvctl stop scan

--We said the DNS team to add the former scan-related DNS definitions of the source platform and made them to be resolved using the target platform's scan IP addresses.

--Then, we checked our server (Exadata X7's db nodes) and confirmed that these new hostnames (scan names) could be resolved properly. We needed the clear the OS DNS cache (even reboot may be required) to make the target platform to resolve its scan ip address using newly mapped scan names.

-- modified the scan name as root user

$GRID_HOME/bin/srvctl modify scan -n exaxxx

--Lastly, made the modification for the scan resources & confirmed the change, -- as the grid user.

$GRID_HOME/bin/srvctl modify scan_listener -u
$GRID_HOME/bin/srvctl start scan_listener
$GRID_HOME/bin/srvctl config scan
$GRID_HOME/bin/srvctl config scan_listener

Reference for this operation: How to Modify SCAN Setting or SCAN Listener Port after Installation (Doc ID 972500.1)

Note that, this approach is applicable in environments where all the clients and applications are using scan names (scan listeners) to connect to the databases.

RDBMS -- ORA-38753 -- The effect of flashback_mode_clause (tablespace) to restore points & snapshot standby operations

Last year , I wrote a blog post about using guaranteed restore points.. In that blog post, I did a demo to show you the concept and tried to explain the Guaranteed Restore Points with or without Flashback Logging (database level), along with the prerequisites and restrictions.

Here is the link of that blog post -> http://ermanarslan.blogspot.com.tr/2017/08/rdbms-flashback-feature-demo-guranteed.html)

Today, I'm here to write about a very specific but also very important thing that you may face while restoring to a guaranteed restore point.

Although, this blog post seems to be related with the guaranteed restore points only, it is actually not .. That is, as the standby snapshot technology relies on the guaranteed restore point, this blog post is also related with the snapshot standby technology. So, you may face this issue while converting your snapshot standby to physical standby, as well.

I hope, you read this blog post, before facing with that thing, because it is a little shocking :)

In spite of the name "Guaranteed Restore Point", you need to be aware of the following fact in order to be able to restore to a guaranteed restore point! ->

You must not have any tablespaces which have Flashback_on set to NO.

If you have flashback_on set to OFF  for a tablespace-> then you may end of the following error stack while converting a standby snapshot to physical standby, or while doing a flashback to a restore point ->

ORA-38753: Cannot flashback data file XX; no flashback log data.
ORA-01110: data file YYY: 'XXX' 

Although, FLASHBACK_ON is by default set to YES, it can be changed to NO.. So if you do this, you won't be able to restore to a guaranteed restore point.

When FLASHBACK_ON is set to YES for a tablespace, Oracle Database will save Flashback log data for that tablespace and thus, the tablespace can participate in a FLASHBACK DATABASE operation.

However, when FLASHBACK_ON is set to OFF for a tablespace, then Oracle Database will not save any Flashback log data for that tablespace. That's why, if FLASHBACK_ON is set to OFF for a tablespace, you must take the datafiles of this tablespace offline (or put the tablespace offline) or drop them prior to any subsequent FLASHBACK DATABASE operation.

Relevant commands for disabling/enabling flashback for a tablespace; 

alter tablespace XXX flashback off; 

alter tablespace XXX flashback on;

So if you are planning to use guaranteed restore points, or snapshot standby technology, it is better to check v$tablespace to ensure that all the critical tablespaces are flashback enabled.

FLASHBACK_ON column in v$tablespace -> Indicates whether the tablespace participates in FLASHBACK DATABASE operations (YES) or not (NO)

Interesting right? Although, the database must not be in flashback mode, all the critical tablespaces must be in flashback mode, in order to be able to restore to a guaranteed restore point (or to be able to convert a snapshot standby to a physical standby)

So if, for any reason, you need to flashback your database to a restore point in a database environment where there are tablespaces for which the flashback modes are set to OFF, you need to follow the action plan documented in the following MOS note:

Flashback Database fails with ORA-38753 ORA-01110 with Tablespaces having Flashback off & RESETLOGS (Doc ID 1588027.1)