EBS R12 -- OAF NLS Character problem, Reviewing NLS related profiles

You may encounter a stange problem in OAF screens..
That is , you may save your new records with your language specific characters , but when you query these records you will see garbage characters instead of those special characters..
Like described in the following picture;

The issue is there for the scenarios;

When you reach the page Employee Self Service -> Personal information , insert a new empoloyee record with your special characters and press submit button..
When you reach the page Employee Self Service -> Personal information , update the employee info with some words with your special characters and press submit button..

Okay.. I will keep short . So , if that 's the case; and if your environment is setup properly to store those special characters; then I can say that the problem is caused by the ICX: Client IANA Encoding profile..

So, it is important to know the profiles but it is also important to know their usages..

In other words; it is important decided to which profiles to control in a problematic scenario..

For example : for this specific scenario, as we know the character is configured properly and as we know that the problem is in html based oaf screens, we suspect from the Client IANA Enconding and check that one..

So, this problem as I have mentioned about was caused by ICX: Client IANA Encoding profile.
It was set to Ascii , and the fix was setting it to "Turkish (ISO-8859-9)" .. (The environment was turkish, the problematic chars were turkish chars..)

Okay.. Lastly, I wil share bunch of EBS  profiles which may affect our clients..

You can check "Globalization Guide for Oracle Applications Release 12 (Doc ID 393861.1)" for more info..

Profile Option Name	Specifies	Comment
ICX: Client IANA Encoding	IANA character encoding used with displays for HTML-based Applications products	For a general description of IANA encoding, visit the IANA web site.
ICX: Date format mask	Date format
ICX: Date language	Date language	Not recommended for use in Release 12. Exists for backwards compatibility.
ICX: Language	Language preference for certain components such as Workflow
ICX: NLS Sort	Type of sort for character data	Sorting other than binary is not supported in Release 12.
ICX: Numeric characters	Decimal separator and grouping separator for number presentation. The characters are specified in the following format:	Commonly "10,000.00" for the US and Asia Commonly "10.000,00" for parts of Europe.
ICX: Preferred Currency	ISO Currency code
ICX: Territory	Territory setting, which determines cultural conventions such as local time, date, numeric, and monetary conventions
Server Timezone	Time zone on the database server.	This setting is available to administrators only. Time zones in Oracle Applications are handled by the database tier.
Client Timezone	Time zone on the client machine.	Time zones in Oracle Applications are handled by the database tier.
FND: Native Client Encoding	Native character encoding on the desktop client	This value is mainly used for file upload and download

EXADATA -- EBS 11i -- Migrating EBS 11i to EXADATA (12C RDBMS environment)

This was my 6th EBS on Exadata implementation..
This time, the source system' Application tier was 11i(11.5.10.2) running on Redhat Linux 32 bit and Source Db tier was 12c (12.1.0.2) running on Redhat Linux 64 bit.
It is certified..

The source database was Oracle Rdbms 10.2.0.4 64 bit ... Its size was approx 800GB and it was a heavy loaded one .. (hugepages were enabled with 90gb SGA and so on)

The target Database Tier was Exadata X3-2 equipped with 12c RDBMS software and Grid Infrastructure Homes.. It was a X3-2 Quarter Rack (High Performance)
Target Application tier was again Redhat Linux 32 bit, as 11i apps tier code is 32 bit..

The target environment was certified ..
Here is current certification of EBS 11i and Oracle Databases.

The migration operation was done for an EBS on Exadata POC, as before making the purchasing, Customer needed to see the performance improvement in standard EBS screens, processes, Discoverer reports and batch processes..

The documents used for there migration were as follows;

Export/Import Process for EBS 11i on RDBMS 12c (Doc ID 1585257.1)
Migrating Oracle E-Business Suite to Oracle Exadata Database Machine Using Oracle Data Pump Oracle Maximum Availability Architecture White Paper February 2013

The main document to follow was "Export/Import Process for EBS 11i on RDBMS 12c (Doc ID 1585257.1)" and the supplementary document was "Migrating Oracle E-Business Suite to Oracle Exadata Database Machine Using Oracle Data Pump Oracle Maximum Availability Architecture", which was actually used for EBS-RAC configurations.

Note that : there were other documents like EBS 11i-12c interoperability , EBS 11i 12c upgade and so on) referenced by these two main documents..

Okay, before we proceed, I need to say that this POC was a hard one , because some preparation steps were missing in the source system..
I mean autoconfig patches and 12c interop patches were not applied..
We realized this situation in the middle of the migration work, and we didnt want to go back and apply those patches and export the source database again..
Rather than doing the same things again, I choosed to try by continuing the migration altough the preparations were missing..

So It was a challange really, but I can say that it worths it..

There were a lot of thruobles on the way, but at the end of the day, if you survive all of those throubles, you feel yourself like you have done a big work really: )

Note that; I don't recommend such an operation .. Even for the POCs...

I will not give you the step by step instruction this time.
On the other hand; I will give the problems and solutions which may be encountered on the way;

MIGRATING EBS 11i to EXADATA 12C RAC database -- without applying 12c interop and autoconfig  patches in the source system preparation phase.. ->

The plan in this migration project was to make the database migration using expdp, and then make autoconfig in both the db nodes and the application node (Note that : apps tier autoconfig requires manual changes in the context file in this scenario)..
After running autoconfigs; once we would able to connect our target apps tier to its db tier; we planned to apply the autoconfig and 12c interop patches to make the System be supported again.
If we could achieve this; the next thing in our plan was to activate the second db node and make postclone in the Application tier to correct other things that we didnt while running autoconfig with manuel context file changes..  (that might be some other things that postclone does and auconfig does not..)

Lets take a look at the problem that we have encountered..

1)
The patch 20204035  could not be applied to the Exadata RDBMS Home 12.1.0.2..
It had a conflict patch , which belonged to a bundle patch (20075921)..
There should be a lack in the oracle document, because we could not rollback the bundle ,as it brought a lot of bugfixes.. So we just didnt apply the patch 20204035..

2)
While using opatch lsinventory to check the patches in RDBMS home; we have encountered an inventory lock problem.
We fixed it  by removing the patch_locked file located in $ORACLE_HOME/.patch_storage

3)
As I mentioned above the interop patch 16289505 and the autoconfig patches were not applied, which have made this operation more interesting :)
The reason was urgency.. For this time, we did not prepare the system.. We actually gave that work to the customer, and later we realized that the customer just left out the source system patching steps...
When I came to the customer environment and realized that the patches in the source system preparation phase were not applied , we were in the middle of importing the EBS database to Exadata , so I decided to give a try :)

I decided to continue without autoconfig and interop patches , and apply these patches at the end..
I mean , after the postclone operation -- if I could  achieve to reach the postclone steps..:)

4)
Installed Oracle Products from the 12C Database Examples CD into the Oracle RDBMS homes of the database nodes.. Altough, I didnt encounter any problems in this step, I had to write this, as this examples cd thing can be confusing sometimes.

5)
Copied the init.ora from the source system to the target node 1 and applied the 12c1 section of the 216205.1 on it.. Added some params, removed some params and so on..

The new parameter pga_aggregate_limit have attracted my attention, as it comes with 12c and it can be used to limit the pga usage..

Default value	By default, PGA_AGGREGATE_LIMIT is set to the greater of 2 GB, 200% of PGA_AGGREGATE_TARGET, and 3 MB times the PROCESSES parameter. It will be set below 200% of PGA_AGGREGATE_TARGET if it is larger than 90% of the physical memory size minus the total SGA size, but not below 100% ofPGA_AGGREGATE_TARGET.
Modifiable	ALTER SYSTEM
Range of values	Oracle recommends that you do not set PGA_AGGREGATE_LIMIT below its default value. PGA_AGGREGATE_LIMIT cannot be set below its default value except in a text initialization parameter file (pfile) or binary server parameter file (spfile). If a value of 0 is specified, it means there is no limit to the aggregate PGA memory consumed by the instance.
Basic	No

PGA_AGGREGATE_LIMIT specifies a limit on the aggregate PGA memory consumed by the instance.

There is no difference in behavior between PGA_AGGREGATE_LIMIT being explicitly set or being set to the default.

6)
 While modifying aucrdb.sql, actually while modifying the datafile and redolog file paths, we can use vi find and replace.. A command something like this may be used :
%s/ \/dbf/+DATAC1\/TEST70\/datafile/g

7)
While reviewing the aucrdb log file, we may see some errors related with default temp tablespace.. We can ignore them..
Anyways; after restarting our newly created db, we run database preparation scripts..
Note that: user pass information is as follows;
sys/change_on_install, system/manager

8)
After creating the db and running preparation scripts, it was good to check dba_registry..
At this point; we prepared a temporary listener  to make our DBAs be able to connect to the database directly from their PCs and make our check with our favorite tool (like toad)
After configuring listener.ora and starting the listener by making the local listener to be set to this temporary listener , the client could connect to the database..
alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=ermanhost)(PORT=1591))' scope=memory;
We have seen CATALOG and CATPROC as invalid in the dba_registry.

We could directly validate CATALOG using;
sqlplus "/as sysdba"
SQL> exec dbms_registry_sys.validate_catalog();
PL/SQL procedure successfully completed.

For validating te catproc, we just compiled the invalids and used validate_catproc procedure..
cd $ORACLE_HOME/rdbms/admin
sqlplus "/as sysdba"
SQL>@utlrp.sql
SQL>exec dbms_registry_sys.validate_catproc(),

9)

Could not disable automatic gathering of stats; as in 12C gather_stats_job does not exist..

declare
*
ERROR at line 1:
ORA-27476: "SYS"."GATHER_STATS_JOB" does not exist
ORA-06512: at "SYS.DBMS_ISCHED", line 4567
ORA-06512: at "SYS.DBMS_SCHEDULER", line 2920
ORA-06512: at line 2
ORA-06512: at line 23

743507.1 "Why Has the GATHER_STATS_JOB been removed in 11g?"

10)

At this point it was good to take a backup;
as our target db was in noarchivelog mode;
we mounted the database and took our backup;
rman target /
backup database format "/directory/TEST70_%U"

11)
We used parallel import; using impdp parallel parameter..
Index creation operations is in impdp are actually serial in OS level.. They are parallel in db tier.

That is ; there is only one index is created in a time, but that one index is created using parallel workers.. (Like Create index parallel 8)
Also ; even specifying parallel argument; oracle take the parallelization of db.. It takes the value of the parallel_max_servers,parallel_max_target and creates our indexes accordingly..

12)

While impdp is working; it may get resumable errors sometimes.. These resumable errors are generally encountered because of the insufficient spaces on the corresponding tablespaces.
When impdp gets these kind of errors, it waits and when we enlarge the problematic tablespaces, imdp continues automatically.But, there are also some errors which are not resumable.. For example: insufficient privileges error encountered during creation of indexes..

If such an error is encountered Imdp skips these problematic actions.. Thus, we need to take those actions manually after the import is finished.. For ex: We need to create the indexes using a database user which has needed grants..
In order to get rid of this risk, we can grant DBA privilege to the custom schemas just after starting our impdp process.. Ofcourse, after the impdp is finished, we need to revoke those privileges.

I say custom schemas because standard schemas are export/import compatible.. These kind of privilege errors are caused by the custom schemas most of the time..

13)
We may a suffix problem in the DB LINKS.
If that is the case, we just alter our global name and make it set to the db name with domain suffix.
For ex:
alter database rename global_name to "TEST70.ermanhost.com.tr"

14)

While importing the statistics , impdp had hard times.. It could take a long time.

That 's why we cancelled the import using imdpb ATTACH_JOB and kill_job..

Actually, firstly  we tried to skip the stats import job using START=SKIP_CURRENT, but it did not work.. Because SKIP CURRENT of impdp is for the sub jobs, not the main jobs.

I mean, you can skip importing of a table, but you cant skip importing the TABLE class action..

As , Importing statistics is a main job wihch does not have a sub job, it can not be skipped using SKIP_CURRENT option..

Anyways; we killed the import job and restored the db ..

We restored the db to the initial form of it, because there are some other tasks which impdp usually does after importing the statistics.. As we killed the impdp during the stats import, those other taks have become missing..

Before dropping the database for restore, we backed up the current control file. (default backup location is $ORACLE_HOME/dbs)

Then we dropped the db using rman..

To restore the db; we first restored the control file by specifying its path to rman, and then restored the database..

After restoring the db, we restarted the import.. This time we gave dba privilege to the custom schemas and enlarged the tablespaces.

15)

Because the interop and autoconfig patches were missing; we encounter Timezone problem while running autoconfig on target db nodes.

In order to fix this errors, we needed to modify the template files and

if conditions for db121..

if db121 -> use this timezone file..

16)

In postclone of appsTier , we encountered problems again.. It was caused by the missing interop and autoconfig patches as you may guess :)

That 's why we run autoconfig rahter than postclone.

But, autoconfig could not connect to db.. The cause was that autoconfig was trying to connect to the database using SID.. I mean its connection format was not right for the 12c RAC listeners..To solve this, I used a temporary listener in the db node 1 appropriate to the format that autoconfig uses, and made the autoconfig to connect to the db node using this listener..

Something like this;

listener_test70=
(description=
(address=(protocol=tcp)(host=exadb01-vip)(port=1555)


sid_list_listener_test70=
(sid_list=

(sid_desc=
(oracle-home=/u02/app/oracle/product/12.1.0/dbhome_2)
(sid_name=test701)))

This time, autoconfig encountered no available protocol errors.
I solved using this using SQLNET.ALLOWED_LOGON_VERSION=8 in sqlnet.ora files of the listener 's Oracle Home of theve problem giderildi.

Also granted inherit privileges to apps for the privilege errors.

About SQLNET.ALLOWED_LOGON_VERSION:

There's a common misconception about the sqlnet.ora parameter SQLNET.ALLOWED_LOGON_VERSION and how it affects the version of clients that are allowed to connect to a database for specific values of this parameter.

The parameter actually specifies the AUTHENTICATION PROTOCOL (for example SHA-1) that a client is allowed to use, NOT the actual VERSION of that client.

Example: SQLNET.ALLOWED_LOGON_VERSION=10

Even though the parameter value implies Oracle version 10 the internal check is really against the authentication protocol 'SHA-1'.

In earlier Oracle versions this was a one-on-one relation, ie. a certain Oracle client version used a specific authentication protocol.
Starting with Oracle 10g, this is no longer a one-on-one relation: Oracle 10g and 11g both use the SHA-1 protocol.
Oracle 12g will use the SHA-2 protocol.

This means that there's no way to prevent 10g client connections and allowing only 11g client connections through the use of this parameter, because both use the same authentication protocol SHA1. This also means that both client versions are 'equally secure' - there's no gain in security by trying to prevent 10g clients alone.
Important note:

The SQLNET.ALLOWED_LOGON_VERSION parameter is deprecated in Oracle Database 12c.
This parameter has been replaced with two new Oracle Net Services parameters:

SQLNET.ALLOWED_LOGON_VERSION_SERVER
SQLNET.ALLOWED_LOGON_VERSION_CLIENT

Okay.. It is actually not a big thing , it is written in the interop document, too :)

Start the new database listener (conditional)
If the Oracle Net listener for the database instance in the new Oracle home has not been started, you must start it now. Since AutoConfig has not yet been implemented, start the listener with the lsnrctl executable (UNIX/Linux) or Services (Windows). See the Oracle Database Net Services Administrator's Guide, 12c Release 1 (12.1) for more information.
Attention
Set the TNS_ADMIN environment variable to the directory where you created your listener.ora and tnsnames.ora files.
Attention
Ensure that your sqlnet_ifile.ora has the line:
SQLNET.ALLOWED_LOGON_VERSION_SERVER = 8

17)

Flexfields named JG_AP_INVOICES and JG_FA_BOOKS could not be compiled;

here is the explanation:

JG_FA_BOOKS descriptive flexfield is needed only for Latin American Localizations (JL). It is not needed when these localizations are not used. Therefore, it could remain as invalid just in case that the Latin American Localizations (JL) are not used.

JG_AP_INVOICES has problems in its declaration, if it is needed , those problems should be corrected..

Actually, these flexfields are not used and can not be compiled in the source environment , too..

That's why, this problems were ignored..


18) 

After making autoconfigs and connecting the appplication tier to the db node properly, we applied the autoconfig and 12c interop patches..

The patches 9535311 and 16289505 ...


16289505 encountered some errors..
sqlplus -s APPS/***** @/appydk/TEST65/test65appl/admin/TEST65/out/p001invok.sql checkfile(115.8=120.4):fnd:patch/115/sql:AFINSTLS.pls
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `sqlplus -s APPS/APPSPWEXA @/appydk/TEST65/test65appl/admin/TEST65/out/p001invok.sql
checkfile(115.8=120.4):fnd:patch/115/sql:AFINSTLS.pls'


This error was documented and Patch 12539637 solved it.
16289505 Failed With AFINSTLS.pls Error (Doc ID 1966820.1)
Please check if you have this patch applied:
12539637 : REQUEST NOT TO ERROR 11I FILES THAT HAVE CHECKFILE EQUIVALENCE SYNTAX

If NOT applied on your instance, please apply it first and then reapply patch 16289505.

---> At this point we became SUPPORTED again :)

19)

After the upgrade invalid OLAP objects may be dropped

Invalid OLAPSYS Objects After Upgrading TO 12C (Doc ID 1593666.1)

If needed, OLAP can be remove using;
How To Find Out If OLAP Is Being Used (Doc ID 739032.1)

20)

After making everying right, we run a postclone just to be sure that it is working properly..

Then changed the twotask context file parameters and run autoconfig to activate the changes..

We configured concurrent manager to be on db node 1 and everyting else to be load balance and failover..

21

)We have gather stats using fnd_stats.

22) 

We did not create any ACLs, because we saw that our patched appsTier already created one..
An ACL with HOST=* 

There were ORA-12504 in non-Oracle DBLinks.. (dblinks for MS SQL and DB2)

ORA-12504 When Using (HOSTNAME) Method For 11G Client/Database (Doc ID 556996.1)
On the other hand; the  HOSTNAME.DEFAULT_SERVICE_IS_HOST=1 parameter mentioned in the document was not the solution..

23)

At the end; we opened the system and tried to login..

Unfortuneatly, we could not...

When using the submit button Login page was throwing;
Oracle error &ERRNO: &REASON has been detected in &ROUTINE.

We analyzed and found that there was a invalid custom trigger in the fnd_nodes table.. Disabling it fixed the error.
Event Alerts on FND_USER, or FND Login/User Type Tables Cause Problems (Doc ID 353602.1)

After this step, the migration was finished ... We took a deep breath and delivered the system to the user acceptence and performance testing..

At the end of the day; I can say that those Interop patches are really needed to be in place before the migration..

Following Oracle documents for accomplishing such big taks is not the only way, but it is clearly the best way. If you dont follow, you become unsupported and you need to support the system yourself :)

Even if your system become unsupported during the way, you may find a way to be supported again, but note that: you will be on your own while you are unsupported..

Anyways, this POC was a successful one.. Migration that I have mentioned in this post was hard but successful ..

In the Exadata side, configured hugepages for the SGA, configured the database to use hugepages only, optimized the parallelization levels , kept some tables of flash cache and that's all..

Tests were successful and It is needles to say that the performance of Exadata was very good..

Without configuring almost anything , we have seen x10 increase in most of the batch jobs and reports..

In conclusion;

I can say that; EBS 11i works well with 12C database.

Interop and autoconfig patches make EBS 11i to work with 12c .. , the affects of these patches can be seen in the behaviours/capabilities of AD Tools, autoconfig and general application connections.. 

Exadata X3-2 with 12c database and grid software increases the performance of EBS in general . 

Ofcourse, some works which are designed to process the rows one by one, may be stay in the same performance, but in general EBS benefits from the processing power of Exadata..

In 12C we have pga_aggregate_limit parameter which let us limit the PGA usage.. This is good for consolidation environments.. (think several databases run on the same machine, same OS)

Using DBRM , we can limit maximum degree of parallelism for any operation within consumer groups..We use the PARALLEL_DEGREE_LIMIT_P1 directive attribute to specify the degree of parallelism for a consumer group. This is good for mixed environments such as EBS..

Lastly, migrating EBS to Exadata may be seen like following some documents and executing some scripts and so on, but in fact it is not :) It is a crucial work, and you encounter with new problem scenarios in every new Exadata migration projects..

I hope you will find this post useful..
I will post the performance reports when I will have them..

EBS R12 -- Hr Organization Chart redirects to the home page!

I have alrady written the article "Implementing Hr Organization in EBS"
Covered the steps , shared the screenshots and give some diagnostics info which may be used against the problems on the way..
http://ermanarslan.blogspot.com.tr/2014/05/ebs-122-implementing-hr-organization.html

But there are some problems, that can not be corrected easily..
Such problems make us configure our EBS environment according to the needs of Hr Org Chart application..
Yes! I m talking about the domain names..
Hr Org Chart want us to have the same domain name for all of the servers involved between the process for reaching Hr Org Chart from the EBS..

Load Balancer, Application server , SSO server, Hr Org Chart Server , you name it :) , they all need to have the same domain.. We need to see the same domain in fnd_nodes for all the nodes:)
Even if we dont have such an environment, we will end  with cookie problems.. Thus when we click on the Hr Org Chart link in EBS (For ex: Manager Self Service - > Talent Profile), the browser will refresh the page , it will go to the Hr Org Chart url but will come back to the same page again.. In short; Hr Org Chart will not open...
I have talked about this with Oracle Support, I have spoken with the associated developer .. This unfortunetaly is the fact.. 

EBS R12-- Problems in Enabling SSL on SSO 10g

In this post, I will point out some important aspects about enabling SSL on SSO 10g..
As you may know, support for SSO 10g is limited.. Thus, Oracle directs us to use Oracle Access Manager rather than SSO 10g.. So, the information that we can reach in the internet  is also limited ..
This become a problem when we hit some bug, or when we need a special configuration.. In such situations , unfortuneatly we stand alone.. That is; we need to solve the problems ourselves by making advanced diagnostics..

When we talk about SSL on SSO 10g. We actually talk about 3 tiers.
One of them is the Application that uses SSO to authenticate its users..
In this story; this application is EBS R12 ..
Ofcouse the midd tier is SSO 10g which is something like an  Oracle Application Server.
The 3rd tier is the OID and its database.. In this story , it is OID 11g which runs on Weblogic Server..
,
In order to diagnose  the integration problems, we need to know the following;

The txkrun.pl script stored in EBS .. This script makes the EBS to register itself with SSO through OID.
The scripts like ssocfg.sh, ssoreg.sh stored in SSO 10g server.. ssocfg.sh configures the SSO and ssoreg.sh register the SSO  and also updates the mod_osso registration record in osso.conf.
Owm in SSO 10g ( oracle wallet manager) which is used to store our SSL certificates.
OID 11g configuration tables to check the registered partner application configuration from the backend..
odsm in OID 11g to check and configure things like operationsUrl for such a configuration change..

Okay.. I will give two problems and their solutions to show what to check while dealing problems in an EBS integrated SSO10g-OID11g configuration after enabling SSL in SSO.

First of all, we can follow the steps described in the following link to enable SSL on SSO 10g.
http://ermanarslan.blogspot.com.tr/2015/02/sso-10g-enabling-ssl-on-ebs-integrated.html

This steps seems sufficient for such an operation, but still some problems may arise .

Possible problems:

1)
unable to logout from EBS after enabling SSL on SSO...
When we encounter this problem, we need to check the urls.. I mean the urls used in SSO login and SSO logout. I can say that : a consistency problem between these urls will create logout problem.
For example: if the SSO login url is https://ermanhost:4443/... and if te logout URL is https://ermanhost.ermandomain:4443/... , then we will have this logout problems.. That is, domain name is important..
To check the SSO urls used in EBS logout&login we can use  orasso.wwsso_papp_configuration_inf_t table in the OID database..
Browser http trace may also help..
If we see an inconsistency, then we need to use ssoreg.sh to update this records accordingly..
Lastly, we take the action to deregister and register EBS using txkrun.pl

2)
certificate warnings in Browsers..
If domain name used in login/logout urls  is different than the domain name in the ssl certificate , then we will have certificate not trusted warning in Browsers..
For example: our login url is ermanhost.ermandomain:4443 but the certificate we use in SSO 10g is signed for ermanhost..
In such a situation, the solution can be changing(using ssoreg.sh) the login/logout urls to match the info stored in ssl certificate. Lastly, we take the action to deregister and register EBS using txkrun.pl
 Alternatively , we can request a new certificate that matches our login/logout urls from the CA  and make the SSO use that certificate.
CN in the certificate should match the FQDN used in login/logout urls..
For example , if we have CN=ermanhost in the certificate , then we need to have https://ermanhost:4443.. written in our login/logout urls..

3)
The page can not be displayed.. Unable to reach the sso login page from the browsers using its https url even if everything seems ok.
In this situation, SSL may be disabled in our Browsers.. Security policy in our environment may want us to use TLS instead of SSL ..
If that 's the scenario, this is described in  TLS 1.0 Handshake Fails With "SSL call to NZ function nzos_Handshake failed with error 29014" ( Doc ID 470123.1 )
The fix is applying patch 6370967.. Ppply this patch in SSO Oracle Home and retest..


In conclusion, I can say that dealing with problems in SSO infrastructure is not an easy thing.. We cant get enough support when something strange happens on the way :)

However, I write this post to light the way for us to realize the facts in such a integrated configuration. Having the general knowledge about the configuration architecture as a whole let us to analyze the problems on our own, as  enabling SSL in SSO 10g affects the configurations stored in OID and EBS , too.. Even the browser congurations may affect our success in such an environment...

Weblogic -- a quick look at WLST and typical use case for addressing edit lock problems in EBS 12.2

WLST in Weblogic is a java based cli, that can be used to monitor and administer Weblogic servers and domains. We can also say that WLST is command line utility , which eases some of the administration works.

An example of the benefits of the WLS can be ability to access the Node Manager and starting,stopping or suspending both local or remote managed servers from the command line.. No need to have an admin server up & running, no need for a browser and so on.

Usages : Online and Offline

There are 2 usage for using WLST. Online and offline.When using WLST online , we can connect to Admin or Managed Servers . So we can access the Mbeans(Java objects which provides managed interfaces) .
In offline use, offcourse we have limited functionality.. Still, we can create a new domain or create/change some configuration files like config.xml..

Modes: 

There are 3 modes for using WLST.

Direct/Interactive/Command Line Mode, Script mode, Embedded Mode..

Interactive mode is a standard cli mode, like sqlplus, we execute commands and WLST returns back the response interactively..

We can also supply scripts to WLST. Scripts containing WLST commands in a text file with .py extension. This is called Script mode.

We can also use embedded mode for using WLST .. That is ;we can embed WLST interpreter in our Java code and use it to run WLST commands and scripts.

Okay.. After a general introduction, lets proceed with an example of the usage ;

Lets start by checking the status of a managed server;

First we set our domain env.. -> . $EBS_DOMAIN_HOME/bin/setDomainEnv.sh

java weblogic.WLST

wls:/offline> connect('weblogic','welcome1','t3://ermanhost:7001')

wls:/EBS_domain_ORATEST/serverConfig> cd ('Servers')

wls:/EBS_domain_ORATEST/serverConfig/Servers> ls()

dr--   AdminServer

dr--   forms-c4ws_server1

dr--   forms_server1

dr--   oacore_server1

dr--   oafm_server1

Note that ; we can use ls and cd commands :) ls() or even cd ('..') works well :)

Okay, we have 5 servers in this domain.. This is an EBS 12.2 instance bytheway..

Lets check the status of oacore_server1

wls:/EBS_domain_ORATEST/serverConfig>  print state('oacore_server1')

Current state of 'oacore_server1' : RUNNING

Okay. lets use some help(all) and see what more we can do with it.

 help('all') 

    help('activate')         Activate the changes.

    help('addListener')      Add a JMX listener to the specified MBean.

    help('addTemplate')      Extend the current domain.

    help('adminHome')        Administration MBeanHome.

    help('assign')           Assign resources to one or more destinations.

    help('assignAll')        (Deprecated) Assign all applications or services.

    help('cancelEdit')       Cancel an edit session.

    help('cd')               Navigate the hierarchy of beans.

    help('closeDomain')      Close the current domain.

    help('closeTemplate')    Close the current domain template.

    help('closestore')       Closes a store.

    help('cmo')              Current Management Object.

    help('compactstore')     Compacts and defragments the space occupied by a file store.

    help('config')           (Deprecated) Navigate to the last MBean in configuration hierarchy.

    help('configToScript')   Convert a domain configuration to WLST script.

    help('connect')          Connect WLST to a WebLogic Server instance.

    help('connected')        Variable indicating whether WLST is connected.

    help('create')           Create a configuration bean.

    help('createDomain')     Create a new domain.

    help('currentTree')      Return the current location in the hierarchy.

    help('custom')           Navigate to the root of custom MBeans.

    help('delete')           Delete a configuration bean.

    help('deploy')           Deploy an application.

    help('disconnect')       Disconnect WLST.

    help('distributeApplication')Copy the deployment bundle to targets.

    help('domainConfig')     Navigate to last domain configuration MBean or root.

    help('domainCustom')     Navigate to the root of domain custom MBeans.

    help('domainName')       Name of the domain to which WLST is connected.

    help('domainRuntime')    Navigate to last domain runtime MBean or root.

    help('domainRuntimeService')DomainRuntimeServiceMBean MBean.

    help('dumpStack')        Display stack trace from the last exception.

    help('dumpVariables')    Display all the variables used by WLST.

    help('dumpstore')        Dumps store contents in human-readable format to an XML file.

    help('edit')             Navigate to last edit configuration MBean or root.

    help('editService')      EditServiceMBean MBean.

    help('encrypt')          Encrypt the specified string.

    help('exit')             Exit WLST from the user session.

    help('exitonerror')      Variable indicating whether WLST exits on error.

    help('exportDiagnosticData')Execute a query (offline).

    help('exportDiagnosticDataFromServer')Execute a query (online).

    help('find')             Find MBeans and attributes.

    help('get')              Return the value of the specified attribute.

    help('getActivationTask')Return the latest ActivationTaskMBean.

    help('getAvailableCapturedImages')Returns the list of Diagnostic Images available for downloading.

    help('getConfigManager') Return the ConfigurationManagerMBean.

    help('getMBI')           Return the MBeanInfo.

    help('getMBean')         Return the MBean by browsing to path.

    help('getPath')          Return the MBean path.

    help('getWLDM')          Return the WebLogic DeploymentManager object.

    help('getopenstores')    Returns a list of opened stores (for script access).

    help('getstoreconns')    Returns a list of connections in the specified store (for script access).

    help('home')             Local MBeanHome.

    help('invoke')           Invoke a management operation on the current bean.

    help('isAdminServer')    Variable indicating whether WLST is connected to an Admin Server.

    help('isRestartRequired')Determine whether a server restart is required.

    help('jndi')             Navigate to the JNDI tree.

    help('listApplications') Lists all applications that are currently deployed to the domain.

    help('listChildTypes')   List all the child MBeans for the cmo.

    help('liststore')        Lists store names, open stores, or connections in a store.

    help('loadApplication')  Load an application and deployment plan.

    help('loadDB')           Load SQL files into a database.

    help('loadProperties')   Load property values from a file.

    help('lookup')           Look up the specified MBean.

    help('ls')               List all the child beans and/or attributes.

    help('man')              Display help from MBeanInfo.

    help('mbs')              MBeanServerConnection object.

    help('migrate')          Migrate JTA, JMS or Server services.

    help('nm')               Determine whether WLST is connected to Node Manager.

    help('nmConnect')        Connect WLST to Node Manager.

    help('nmDisconnect')     Disconnect WLST from a Node Manager session.

    help('nmEnroll')         Enroll the machine with Node Manager.

    help('nmGenBootStartupProps')Generate boot and startup properties

    help('nmKill')           Kill the specified server instance.

    help('nmLog')            Return the Node Manager log.

    help('nmServerLog')      Return the server output log.

    help('nmServerStatus')   Return the status of the server.

    help('nmStart')          Start a server using Node Manager.

    help('nmVersion')        Return the Node Manager server version.

    help('openfilestore')    Opens a file store.

    help('openjdbcstore')    Opens a JDBC store.

    help('prompt')           Toggle prompt information.

    help('pwd')              Display the current working directory.

    help('readDomain')       Open an existing domain for updating.

    help('readTemplate')     Open an existing domain template.

    help('recording')        Variable indicating whether WLST is recording.

    help('redeploy')         Reload classes and redeploys an application.

    help('redirect')         Redirect WLST output to the specified filename.

    help('removeListener')   Remove a listener that was previously defined.

    help('resume')           Resume a server instance.

    help('runtime')          (Deprecated) Navigates to the last MBean in runtime hierarchy

    help('runtimeService')   RuntimeServiceMBean MBean.

    help('save')             Save the edits that have been made.

    help('saveDiagnosticImageCaptureEntryFile')Downloads an entry from a captured Diagnostic Image.

    help('saveDiagnosticImageCaptureFile')Downloads a captured Diagnostic Image.

    help('serverConfig')     Navigate to the last configuration MBean or root.

    help('serverName')       Name of the server to which WLST is connected.

    help('serverRuntime')    Navigate to the last runtime MBean or root.

    help('set')              Set the specified attribute value.

    help('setDistDestType')  Sets the distributed destination type.

    help('setOption')        Set options related to a domain creation or update.

    help('showChanges')      Show the current changes that were made.

    help('showListeners')    Show all listeners that are currently defined.

    help('shutdown')         Gracefully shut down a server or a cluster.

    help('start')            Start a Managed Server or a cluster.

    help('startApplication') Start an application.

    help('startEdit')        Start a configuration edit session.

    help('startNodeManager') Start Node Manager.

    help('startRecording')   Record all user interactions with WLST.

    help('startServer')      Start the Administration Server.

    help('state')            Return a map of servers/clusters and their state.

    help('stopApplication')  Stop an application, making it unavailable to users.

    help('stopEdit')         Stop current edit session and release lock.

    help('stopNodeManager')  Stop Node Manager.

    help('stopRecording')    Stop recording WLST commands.

    help('stopRedirect')     Stop redirection of WLST output.

    help('storeUserConfig')  Create a user configuration and key file.

    help('suspend')          Suspend a running server.

    help('threadDump')       Display a thread dump for the specified server.

    help('typeService')      TypeServiceMBean MBean.

    help('unassign')         Unassign applications or resources.

    help('unassignAll')      (Deprecated) Unassign all applications or services.

    help('undeploy')         Undeploy an application.

    help('undo')             Revert all unsaved or unactivated edits.

    help('updateApplication')Update an application configuration using a new deployment plan.

    help('updateDomain')     Update and save the current domain.

    help('username')         Name of user currently connected to WLST.

    help('validate')         Validate the changes that have been made.

    help('validateConfig')   Validate a Domain configuration.

    help('version')          Current version of WLS.

    help('viewMBean')        Display information about an MBean.

    help('writeDomain')      Write the domain configuration information.

    help('writeIniFile')     Convert WLST definitions to Python file.

    help('writeTemplate')    Write the domain template.

Look we have stopEdit :) I

 help('stopEdit')         Stop current edit session and release lock.

So, lets put a domain lock on the admin console and leave it there..

Why do I give stopEdit() as an example? Because I want to address the problem in automated EBS 12.2 startups :)

Such as this one :

http://ermanarslan.blogspot.com.tr/2015/01/ebs-122-automatic-start-stop-scripts.html

START SCRIPT:

### ERMAN ARSLAN EBS 12.2 APPS START automated
### 12/30/2013
## supply appspassword and weblogicpassword in sequence as command line arguments
### reviewed and tested 1/15/2015
if [ `whoami` == "root" ]
then
echo you can not run this script with root!
exiting
else
. /u01/apps/EBSapps.env run
{ echo apps; echo $1; echo $2; } | sh $ADMIN_SCRIPTS_HOME/adstrtal.sh -nopromptmsg
fi

The problem is if we have domain lock in the Weblogic(which comes with the EBS 12.2) then our managed server start scripts hang... 

What we normally do to fix the problem is actually -> connecting to admin server and releasing the lock..

But now, we will release that lock using WLST :)

What we need to do is using edit() to reach the edit tree, and then using stopEdit to end the edit session..

wls:/EBS_domain_ORATEST/serverConfig> edit()

Sure you would like to stop your edit session? (y/n)

y

Edit session has been stopped successfully.

That 's all. Our edit lock is released.

When we check it using Admin console, we will see that there is no locks anymore..

As you see in the picture above, it is clean :)

Morover;if we can use this wlst command in our EBS start scripts , we can fix the problem even in there..

Okay lets use WLST in scriptized mode and release the lock by using the following script ..

We create a script named erman.py with the following contents;

connect('weblogic','welcome1','t3://ermanhost:7001')

edit()

stopEdit(defaultAnswer='y')

exit()

Then use this script for releasing the lock;

java weblogic.WLST erman.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://ermanhost:7001 with userid weblogic ...

Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_ORATEST'.

Warning: An insecure protocol was used to connect to the 

server. To ensure on-the-wire security, the SSL port or 

Admin port should be used instead.

Location changed to edit tree. This is a writable tree with 

DomainMBean as the root. To make changes you will need to start 

an edit session via startEdit(). 

For more help, use help(edit)

You already have an edit session in progress and hence WLST will 

continue with your edit session.

Edit session has been stopped successfully.

Exiting WebLogic Scripting Tool.

Script works well.

Then we put this script into our automatic start script and we are fixing the "hang" problem..

EBS 12.2 START SCRIPT:

### ERMAN ARSLAN EBS 12.2 APPS START automated
### 12/30/2013
## supply appspassword and weblogicpassword in sequence as command line arguments
### reviewed and tested 1/15/2015

### 02/20/2015 modified to relase the domain lock before starting the application services
if [ `whoami` == "root" ]
then
echo you can not run this script with root!
exiting
else

. /u01/apps/EBSapps.env run

. $EBS_DOMAIN_HOME/bin/setDomainEnv.sh

java weblogic.WLST erman.py

. /u01/apps/EBSapps.env run

{ echo apps; echo $1; echo $2; } | sh $ADMIN_SCRIPTS_HOME/adstrtal.sh -nopromptmsg
fi

Note that: If we dont have domain lock present at the time that we run this script; we may get the following exception:

WLSTException: Error occured while performing stopEdit : Cannot call stopEdit without an edit session in progress

But this is not a problem, it does not break our script . That 'is ; even if there are exception in WLST, the shell continues to execute the remaining lines of our shell script and able start our EBS services without any problems..

Here is the test ;

Contents of erman.sh: 

java weblogic.WLST erman.py

echo "I could execute commands here :)"

Lets run erman.sh and see if we could execute the commands after getting the exception caused by the absence of the edit lock.

sh erman.sh 

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Problem invoking WLST - java.io.FileNotFoundException: /u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain_ORATEST/erman.py (No such file or directory)

********I could execute commands here :)   --> So no problems :)

Okay.. In this post, I have tried to give some general information about WLST. I have given some examples for using the commands in this tool and also tried to consolidate our learnings by making a use case in EBS 12.2..

Hope you 'll find it useful.

RDBMS -- 11gR2 & Windows Single Node to Linux RAC(2 nodes) migration

Recently, migrated an Oracle Database 11gR2 from Single Node Windows 64 bit to Oracle Linux 64 2 node RAC system. Source database was on a NTFS filesystem. For the target database, as you may guess, we have used Oracle ASM .

Source and target database versions were 11.2.0.3 64 bit, so upgrading the database was not the point.
On the other hand,  the migration was already tricky, but the method that I have chosen for it, made the operation to be a clean work.
Besides, using rconfig reduces the effort that has to be taken in the post configuration phases.
There were other alternatives; but applied the following action plan;

• Use rman convert the source db to Linux 64 bit.
• Copy the converted database(like an image copy) to the RAC node 1.
• Recreate the controlfile in the target and invalidate then validate all the plsql objects .(using transport.sql) --this operation is for creating a database from the converted datafiles in the local filesystem of rac node 1 . it is an in-place operation.
• Use rconfig to physically move datafile from cooked filesystem(ext3) to ASM. 
• Make the post configuration and deliver.

Okay. Lets take a look at the steps;

We first take our source database into read only mode,

Then use the plsql in the following picture to ensure that the source database is convertable,

Next, we use rman to convert the source database to "Linux x86 64-bit" .. This operation creates the Linux type datafiles in the directory we speficied in the command.

This operation also creates a sql script (TRANSPORT.sql) for us.. This script will be used in the next steps while we ll be configuring the target instance..

At this point; our work with the source is done, so we can take it back to read write mode.. 

Note that : This was a test run, that's why we took the source database into readwrite mode just after finishing our rman convert operation.. This must not be done for the production instance.. Source Production should be closed at this point..

Next, we copy the converted datafiles to the target system .. Copy them to a local mount point in 1st node

We modify the init.ora in the target according to the target node configuration.

Then, we modify the transport.sql for the target environment. We may change the db sid if we want, also we delete the lines for adding the tempfiles. We leave only one add tempfile line, because of the bug

"Bug 13912880 - Rconfig fails with SQLFatalErrorException ORA-25152 "TEMPFILE cannot be dropped"

Next , we set our target database environment and execute the transport.sql

Once the transport.sql is completed successfully, we have our database up and running in the target Linux Rac node 1..

During the run of the transport.sql , we monitor the invalid objects to estimate the finish time of the operation.

We also check the database for compatible parameter comparing with the ASM diskgroup compatabilities accordingly.

Next we configure rman parallelism for the upcoming rconfig operation.. Here we are using 12 parallel processes.. This parallelism should be equal to the disk heads of the associated storage devices.
.

Next, we configure our rconfig configuration xm file, and run rconfig..
Note that we can also run the rconfig by editing the xml file .. The line "Convert Verify = ONLY " makes rconfig to check the possible errors, and reports them.. If we see any problems after running rconfig with "ONLY" argument, we fix the errors and then reexecute the rconfig..
Once all the errors dissapear, we make Convert Verify = YES and run rconfig again.. Rconfig makes the real conversion when "convert verify=Yes"  is used in xml file.

For example: we got the error in verify phase:
LOG ARCHIVE DEST does not exists on all nodes in the cluster.
Thus, we created the missing directories in node 2 before making the real conversion..

 Okay, here we run the rconfig...

Note that : If we dont have init.ora in default location ($ORACLE_HOME/dbs), we can get ORA-01078 error.
It is because rconfig wants to read the default parameter file during its run.

Altough, convert only completes with success we can still encounter errors during the real conversion..
In these situations, we check the log files under cfgtoollogs/rconfig/SID/sqlLog directory

Moreover; we may got the the tempfile error because of the bug 13912880... Workaround is to have only one tempfile before running rconfig.

When rconfig finishes its run, it sets its Result code to 0 , and reports "Operation Succeeded"

Lastly, we make the post configuration( checking FRA sizes, checking local and scan listener configuration etc, setting optimal process and memory related parameters) in our RAC database and we are done !
We dont have to srvctl add commands and anything else.. All comes configured.