Oracle Linux 6.6 - "kernel 3.8.13-98.1.1.el6uek.x86_64" "not using all available CPU cores"

We have recently seen a problem in Oracle Linux 6.6, as it was not using all the cpu cores available on that server. It was running on a VMWare, but the problem was not in VM actually.

The configuration was like below;

socket 1 = cpu0,cpu1,cpu2,cpu3 socket2= cpu4,cpu5,cpu6,cpu7

The problem was in the utilization.
That is, when using 3.8.13-98.1.1.el6uek.x86_64, Oracle Linux 6.6 was using only 4 cpu cores. We have analyzed cpu utilization properly and it didnt just not allocate the last 4 cpu cores.
Oracle Linux 6.6 was seeing all the 8 cpus on the other hand..

We have used taskset executable to force a process to run on a specific cpu core which Oracle Linux normally did not not utilize and seen that the process have started running on that cpu core without any problems and we could also see that cpu utilization of that cpu core have become %100, as expected.

[root@somehost opt]# taskset -c -p 6 2313
pid 2313's current affinity list: 0-7
pid 2313's new affinity list: 6

[root@somehost~]# top
top - 19:10:14 up 4 days, 6:02, 6 users, load average: 1.06, 0.63, 0.32
Tasks: 432 total, 2 running, 430 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.7%us, 0.7%sy, 0.0%ni, 98.0%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
..
..
Cpu6 : 99.7%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st


So, when forced, Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel was using all the cores, but normally the scheduler automatically did not utilize the 4 cores coming from the second cpu socket, even under a very loaded situation as seen below. (cpu4,5,6,7 is not used.. not utilized..)

op - 12:51:32 up 3 days, 23:43, 3 users, load average: 16.74, 9.82, 5.30
Tasks: 454 total, 18 running, 436 sleeping, 0 stopped, 0 zombie
Cpu0 : 92.2%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Cpu1 : 94.2%us, 4.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.1%si, 0.0%st
Cpu2 : 93.4%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.8%si, 0.0%st
Cpu3 : 92.7%us, 5.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32687204k total, 32521340k used, 165864k free, 104284k buffers
Swap: 33554428k total, 32992k used, 33521436k free, 21020212k cached


The strange thing was , the issue could not be reproduced in 3.8.13-44 el6uek kernel.
When booted with 3.8.13-44 el6uek kernel, Oracle Linux 6.6 have seen and utilized all the cpu cores without any problems, perfectly in balance.

So, the problem basically was   "Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel.

The problem was looking like the same that was discussed in the discussion that I have created in Oracle Community. Avi Miller from Oracle replied to that similar problem and stated that this is a known issue 3.8.13-98.2.1 (tracked by internal bug 21662) So , the workaround was downgrading to the previous UEK3 release or using the redhat compatible kernel for the time being.

Actually, a similar problem was there in 3.8.13-98.1.1, as well.

So, we are for now continuing with the older kernel 3.8.13-44 el6uek  and probably upgrade after the internal bug 21662 will be resolved.

RDBMS -- ORA-28750: unkown error in SSL web service call

If you are trying to do someting like the following and end up with ORA-28750 error, then the server that you are trying to connect, is probably using an SHA2 ssl certificate.

SQL> DECLARE
2 lo_req UTL_HTTP.req;
3 lo_resp UTL_HTTP.resp;
4 BEGIN
5 UTL_HTTP.SET_WALLET ('file:/wallet/','welcome1');
6 lo_req := UTL_HTTP.begin_request('https://ip_address/erman?wsdl');
7 lo_resp := UTL_HTTP.get_response(lo_req);
8 dbms_output.put_line(lo_resp.status_code);
9 UTL_HTTP.end_response(lo_resp);
10 END;
11 /

DECLARE
*
ERROR at line 1:

ORA-29273: HTTP request failed
ORA-06512: at "SYS.UTL_HTTP", line 1029
ORA-28750: unknown error
ORA-06512: at line

You may think that you wallet is problematic but it is not..
If that 's the case, then you are probably using an older version of Oracle Database :)
Something like 11.1.0.7 maybe... SHA2 is certified in Oracle Database 11.2.0.3 and above.
(To check the certificate you can just save it to your laptop, rename it to .cer then double click on it..)

So if that 's case, then you are two options:

1) upgrade your db ..
2) If the server that gives that web services is on your control, change the server side certificate to be 'GeoTrustSSLCA-G3'

We choose option 1 :) , as it is time.. it is even the time for an 12c upgrade..

EBS 12.2 -- what if we apply a patch with hotpatch option, if that patch is not suitable to be applied as hotpatch?

If not stated by an Oracle Support document or by any Oracle Document including Patch readme's, we just create online patching cycle and apply EBS 12.2 applicaiton patches online.
So what if we apply a patch with hotpatch option, if that patch was not suitable for this?

First of all, these patches are considered as unsafe patches. They are not tested by Oracle and probably not suitable for applying directly in to the run filesystem while the application services are running.

The answer for the question, on the other hand; is simple. We just take the risk and need to be prepared for the followings ;

• invalid objects, missing code dependencies 
• mismatches between code level of file system and database 
• missing column data, or other data integrity problems 
• out-of-date indexes and materialized view definitions 
• invalid data in runtime caches

:)

So, hotpatches was maybe more stable in EBS 12.1 or we thought that they were stable as maybe we didn't know the problems that users was expriencing while we were patching the system with an hopatch. 

But in EBS 12.2, the effects are on system-wide as you see in the above list.

Well..At the bottom line, that's why I wan to remind that it is not safe to apply a patch with hopatch option in EBS 12.2 -- unless it is stated by Oracle Support Documents or patch read me files.

EBS 12.2 -- why a downtime patch is faster than hotpatch?

Recently wondering, how can a downtime patch be faster than a hotpatch..
It was stated in all the Oracle Documents, but the reasons was not there..
I was trying to find a logical reason for this, eventough i couldnt see a change in the adop's behaviour while applying the patches with downtime option. adpatch was used again in the background, the online patch cycle was not used and it was just like applying a hotpatch from those point views.
Thus, I concluded that applying a patch with downtime option is considered faster, just because the server resources was not used by EBS users or application services.
Anyways, today I got an answer from an SR that I recently created for this question. The return that Oracle support did was just approved my idea.
So at the bottom line, applying a patch in downtime patch is faster than applying a patch with hotpatch option and it just because the system resources was not used by EBS users or application services when applying a patch with downtime option.

EBS 12.2 -- The new Simplified Home Page

EBS 12.2 has a UI feature that looks really nice. Discovered it recently. Don't ask why so late :) It is probably becausea we are too busy in the backend, so can't find time to navigate in the Application Screens :)

Anyways, the simplified Home page introduced in EBS 12.2 make you feel like you are in an Ipad application..

Here is how the home page looks after enabling the simplified home page in EBS 12.2:

Pretty cool right:) ? It just does not look like an EBS webpage :)

Well here is the key for enabling it:

That is;
In order to use this feature, The "Self Service Personal Home Page Mode" profile must be set to
"Framework Simplified"
After setting this profile option , you can relogin and start the use this simplified Homepage.

Note that : the profile option can be set at multiple levels by the system administrator, including at the individual user level.

ASM / Grid 11g -- Resync vs Rebalance

I wrote an article about Asm Resilvering vs Asm Rebalance in one of my previous post
http://ermanarslan.blogspot.com.tr/2015/05/exadata-asm-resilvering-vs-asm-rebalance.html

In this post, I will explain the difference between Asm Resync vs Asm Rebalance shortly.
Nowadays, I can't post several blog posts like I have done in the previous month, because I m working on a Book which will be released in Jan 2016 .
So that's why I can write to several blog articles and, when I m able to write them, I need to keep them short in these days:)

Anyways, "the difference between Asm resyncing and rebalancing" was a question posted by one of the followers in Exadata Facebook group; and here is the answer for that:

Their usages are different and can be compared by examining the fast disk resync and fast rebalance operations introduced in 11g
Resync is something like syncronizing the disks with the data that should reside on them. (can be used on transient failure ) (ASM 11g New Features - How ASM Disk Resync Works. (Doc ID 466326.1))
Rebalancing is something like spreading the data evenly across all  thedisks in a disk group. (can be used on a disk replace operation)

EBS 12.2 -- AD script start/stop hang issue in older AD TXK versions

Oracle recommends upgrading AD and TXK levels time to time.

Oracle Delevopment advises it as follows : 

Note: Oracle strongly recommends that you always apply the latest version of the AD and TXK release update packs at your earliest convenience.  

In addition to the performance enhancements and bug fixes there are some enhancements in the latest AD and TXK level that eases the administration and fixes some administration problems ,as well.

For instance; the AD and TXK Delta 6 brings an opportunity to start the WLS servers even if the admin server is done. 

This enhancement is introduced as an option named -msimode which brings the ability to use local copy of the configuration and skip the checks to identify the changes since last start.

Concerning the new -msimode above; also as per our practices, we have seen the following hang issue in AD startup scripts(for ex: adstrtal.sh) can not be reproduced with the lastest TXK and AD levels. 

The issue was documented in the following blog post;

AD startup script hangs, when the weblogic domain is locked: 

http://ermanarslan.blogspot.com.tr/2015/02/weblogic-wlst-general-information-and.html

It seemed the issue was due to the synchronization that was done from the generic domain config to a local copy of the configuration for the WLS server  and because the domain was locked the startup of the  WLS server was prevented.

The issue was seen in older code , probably in an environment with AD and TXK Level 3, which caused the AD startup scripts to hang where the WLS domain was LOCKED.

At the bottom line, upgrading AD and TXK levels are recommend and needed as they eases our administration works and saves us spending our for find the root causes of the problems caused by bugs in the older AD and TXK levels.

So , if you are available, upgrade to the latest AD and TXK levels.

Note that : The latest RUPs are R12.AD.C.Delta.6 (Patch 19197270) and R12.TXK.C.Delta.6 (Patch 19330775)

Reference : Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1)

EBS 12.2 - Adop Reports, adopreports

Recently found an utility when I was going around the EBS filesystem. The utility is called adopreports and can be executed as follows.
The utility can be used to do a list of thing without using any other utilities such as Toad and sqlplus.
You can check table details, indexes on a table and more. But, It is actually based on patching. So it is there for actually to be used during patching activities.
In short, It has several reports which can be used for diagnosing adop issues, or simply gaining knowledge about the status of your system.

Here is an example usage of it;
  [applmgr@somehost]$ adopreports

Enter the APPS username: apps
Enter the APPS Password:

    Online Patching Diagnostic Reports Main Menu
    --------------------------------------------

    1.  Run edition reports
    2.  Patch edition reports
    3.  Other generic reports
    4.  Exit

    Enter your choice [4]: 3




    Other Generic Reports Sub Menu
    ------------------------------

    1.  Editions summary
    2.  Editioned objects summary
    3.  Free space in important tablespaces
    4.  Status of critical AD_ZD objects
    5.  Actual objects in current edition
    6.  Objects dependencies
    7.  Objects dependency tree
    8.  Editioning views column mappings
    9.  Index details for a table
    10.  Inherited objects in the current edition
    11.  All log messages
    12.  Materialized view details
    13.  Database sessions by edition
    14.  Table details (Synonyms, EV, etc.)
    15.  Count and status of DDL execution by phase
    16.  Back to main menu

    Enter your choice [16]: 9
Enter the object name : FND_NODES

===============================================================================
=========================================================================
=  Indexes for FND_NODES
=========================================================================

OWNER           INDEX_NAME             COL# COLUMN_NAME            COLUMN_EXPRESSION
--------------- ------------------------------ ---- ------------------------------ ------------------------------------------------------------
APPLSYS         FND_NODES_N1                      1 NODE_ID
APPLSYS         FND_NODES_U1                      1 NODE_NAME


Above report is captured in file : /u01/install/APPSDEVEL/fs2/EBSapps/appl/admin/TEST/out/adzdshowindex.out.

EBS R12 , 12.2 -- MULTI_ORG, MO_GLOBAL.SET_ORG_ACCESS ORA-01031 insufficient privileges, APP-PO-14142, MO_GLOBAL, DBMS_SESSION Line 101

Recently encountered an error in EBS R12 where a Multiple Organization setup was done.
The Standard forms screens was encountering APP-PO-14142 errors.

The error message was saying that there was an ORA-01031 produced in the database level. This error is actually the insufficient privilege error and it is normally produced when there is lack of privileges detected in the database level. So, the apps session must have trying to use a database object without having the necessary grants.

After some investigation , and putting some debug logs into the standard code, we have seen the everyting was in place. The schema that was used by the session was APPS and all the grants of the APPS schema was proper  ( still, just in case, we have recreated the grants and synonyms , as well as executed adgrants.sql)

We have taken an FND debug and analyzed the trace file , using the following well known method;

select max(log_sequence) from fnd_log_messages
eg : log_sequence = 71666646
2) Go to the System Profile options and set the following parameters:
FND: Debug Log Enabled = YES
FND: Debug Log Filename = NULL
FND: Debug Log Level STATEMENT (most detailed log)
FND: Debug Log Module %
3) Re-login to the application and reproduce the bug.
4) select message_text from fnd_log_messages where log_sequence > XXX – value from step 1

During the analysis , we have also seen that the org_id , security group id, application id, responsibility id etc. were all proper.

Using oradebug and/or alter system set events ‘1031 trace name errorstack level 3; , we have also gathered the call trace  and it was like the following; didnt helped much as the names of the function were interrnal and they were not documented or known

Calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+36 call kgdsdst() 000000000 ? 000000000 ?
7FFF2A394E08 ? 000000001 ?
000000001 ? 000000002 ?
ksedst1()+98 call skdstdst() 000000000 ? 000000000 ?
7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
ksedst()+34 call ksedst1() 000000000 ? 000000001 ?
7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
dbkedDefDump()+2741 call ksedst() 000000000 ? 000000001 ?
7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
ksedmp()+36 call dbkedDefDump() 000000003 ? 000000000 ?
7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
dbkdaKsdActDriver() call ksedmp() 000000003 ? 000000000 ?
+1960 7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
dbgdaExecuteAction( call dbkdaKsdActDriver() 7F01CAED9710 ? 7FFF2A39BA90 ?
)+1065 7FFF2A394E08 ? 000000001 ?
000000000 ? 000000002 ?
dbgdaRunAction()+81 call dbgdaExecuteAction( 7F01CAED9710 ? 00A184340 ?
0 ) 0020C0003 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbgdRunActions()+59 call dbgdaRunAction() 7F01CAED9710 ? 000000005 ?
0020C0003 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbgdProcessEventAct call dbgdRunActions() 7F01CAED9710 ? 000000005 ?
ions()+651 0020C0003 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbgdChkEventKgErr() call dbgdProcessEventAct 7F01CAED9710 ? 00BBC01C0 ?
+1653 ions() 7F01CA9BF698 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbkdChkEventRdbmsEr call dbgdChkEventKgErr() 7F01CAED9710 ? 00BBC01C0 ?
r()+56 7FFF2A391640 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
ksfpec()+61 call dbkdChkEventRdbmsEr 7FFF2A391640 ? 00BBC01C0 ?
r() 7FFF2A391640 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbgePostErrorKGE()+ call ksfpec() 7FFF2A391640 ? 7FFF2A391640 ?
1129 7FFF2A391640 ? 7FFF2A39BA90 ?
000000001 ? 000000002 ?
dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 00BBC01C0 ? 7F01CA990040 ?
66 000000407 ? 000000000 ?
100000000 ? 000000002 ?
kgeade()+351 call dbkePostKGE_kgsf() 00BBC01C0 ? 7F01CA990040 ?
000000407 ? 000000000 ?
100000000 ? 000000002 ?
kgerelv()+135 call kgeade() 00BBC01C0 ? 00BBC0370 ?
7F01CA990040 ? 000000407 ?
100000000 ? 000000002 ?
kserecl0()+157 call kgerelv() 00BBC01C0 ? 7F01CA990040 ?
000000407 ? 009D95D14 ?
7FFF2A39C900 ? 000000000 ?
kzctxChkTyp()+402 call kserecl0() 00BBC01C0 ? 7F01CA990040 ?
000000407 ? 000A39118 ?
074F6DB00 ? 074F6DB38 ?
kzctxesc()+341 call kzctxChkTyp() 7F01CA9DE998 ? 000000000 ?
000000009 ? 000A39118 ?
074F6DB00 ? 074F6DB38 ?
pevm_icd_call_commo call kzctxesc() 7F01CA9DE998 ? 000000005 ?
n()+867 7F01CA9A3DE0 ? 000A39118 ?
000000000 ? 074F6DB38 ?
pfrinstr_ICAL()+163 call pevm_icd_call_commo 00A5C6D80 ? 000000000 ?
n() 000000001 ? 000000007 ?
000000005 ? 7FFF00000000 ?
pfrrun_no_tool()+63 call pfrinstr_ICAL() 7F01CA9DE998 ? 24856BD10 ?
7F01CA9DEA08 ? 000000007 ?
000000005 ? 7FFF00000000 ?
pfrrun()+622 call pfrrun_no_tool() 7F01CA9DE998 ? 24856BD10 ?
7F01CA9DEA08 ? 000000007 ?
000000005 ? 7FFF00000000 ?
plsql_run()+644 call pfrrun() 7F01CA9DE998 ? 000000000 ?
7F01CA9DEA08 ? 7FFF2A39DD80 ?
000000005 ? 069D984CC ?

We have also reverse engineered the code (MO_GLOBAL) and seen that it was failing at DBMS_SESSION Line 101.

Line 101 of DBMS_SESSION was something like the following;

KZCTXESC(attribute1,attribute2.....);

the KZCTXESC was an internal C function  and Oracle was using it through a gateway interface probably declared with the following PRAGMA definition in the DBMS_SESSION;

PROCEDURE SOMEFUCTION(attribute1,attribute2.....);
PRAGMA INTERFACE (C, SOMEFUNCTION);

So , this was an Internal mechanism and it was closed to us , thus the issue can not be investigated more after this level.

Also look : undocumented pragmas http://www.petefinnigan.com/weblog/archives/00000835.htm

Also, the problem in the DBMS_SESSION.. It just could not be tested directly and it was misleading to test the dbms_session by using a direct plsql approach, as to set the attribute of a context it is necessary to use the package/procedure declared when the context was created.

That is ;

begin
dbms_session.set_context('multi_org', 'access_mode', 'S');
end;
--> Can not be done, it will produce ORA-01031 in any cases, as MO_GLOBAL should be used to set this context..
->
select type,schema, package from dba_context where namespace = 'MULTI_ORG';
ACCESSED LOCALLY APPS MO_GLOBAL

Anyways, altough the investigations and finding the solution took 4 hours of mine ; I ll keep it short :)

Then, we have took a Forms Trace with binds and analyzed the trace file.

In that trace file, we saw a very interesting thing.

There was a very interesting line, as the standard context name was there , I mean multi_org, but it was written as MULTİ_ORG.. An uppercase "İ" was there, so the database session must have been using the word MULTİ_ORG for using the multi_org context and failing because the MULTİ_ORG with an uppercase İ was not equal with multi_org actually.

By keeping it in mind, checked the recently modified profile option values in the system and saw ICX_NLS_SORT, ICX: NLS Sort was changed to XTURKISH , which was actually the extended Turkish.

Then we connected to the database with sqlplus and set my session's nls_sort to XTURKISH, and used the following query to check the situation;

select nls_upper(multi_org) from dual;

The output of the query was MULTİ_ORG as I was expected.

So at that point we concluded that there must be some uppercase function used in the EBS code and that was changing the name of the context that was trying to be set by the database session from multi_org to MULTİ_ORG , which was actually a different word than multi_org.

We also concluded that, DBMS_SESSION Line 101 could have getting ORA-01031 insufficient privileges because it could not find the context that the session was trying set , as there was actually  no context named MULTİ_ORG in the database.

As for the solution;

We changed the the ICX NLS SORT profile from XTURKISH to BINARY and solved the problem ..

It was a very interesting story , that's why I shared this with you.

It means, chaging a EBS profile, actually changes the behave and output the code and I didnt like it at all. It seem like a bug actually, as an internal APPS context should not be dependent on this kind of profile change..  Anayways, we know the solution for that now :)

The system that we have encountered this problem was the latest Version, 12.2.4.

So, in any case, I ll report that to Oracle, as well.

EBS R12 , 11i -- Changing output file before printing

In this post, I will mention about a little trick that can be used for customizing the printing process in EBS.

As you may already know, EBS prepares the output and send it to printers using Os tools, such as "lp" in Linux.
So, if we can manipulate the output files before sending it to the printers , we can accomplish customizing the output file before sending it to the printers.
Let's explain this process this by a little explain.
Suppose we want to replace all the 0 (zero characters) with the O ( The letter O) in our printings and suppose we want to do this in all EBS prints.
In this kind of scenario, we have to build a mechanism that is tightly integrated with EBS printing sub system, as we want EBS modify the output files generated by EBS and we want it to be transparent to EBS.

So, the things that need to be done are pretty simple actually.
First, we create a script in the application node of EBS environment, where the printing is done.
The script will include a one liner which will replace the characters in the outpout file and send it to the printer. Note that, its first argument will be the printer name and the second argument will be the output file.

/home/applmgr/erman.script
sed -i 's/0/O/g' $2
lp -d $1 $2

Then , we go to EBS screens and modify the printer definition (to the driver actually) as follows;

Change in Arguments text box: /home/applmgr/erman.script $PROFILES$.PRINTER $PROFILES$.FILENAME

I will not go in details about these arguments and script, as you have found this blog and reading this article with curiosity :) That is, I suppose you can understand the purpose of the script and the changes we have done in the arguments textbox above.

Note that: The arguments may be changed, extra arguments may be added.
So, this is the way, and I hope you like it.