EBS 12.2 -- Configuring SSL in Oracle E-Business Suite Release 12.2

Lets demonstrate a SSL implementation process in EBS 12.2.
What I mean by SSL implementation is configuring Clients Browser and EBS Web Services communicate through SSL.  EBS uses FMW to communicate over ssl between its components.

Info:

Oracle Fusion Middleware supports SSL version 3, TLS version 1 and JKS-based keystores for components running under Java and Oracle Wallets for other components, such as the Oracle HTTP Server.

In the Web tier, We still use Oracle Wallet to store and manage our certification.. On the other hand; Java keystores can be used for building a secure communication to and from weblogic servers.

Info:

Oracle Fusion Middleware uses JKS keystore (the default JDK implementation of Java keystores used by Oracle WebLogic Server) to store keys and certificates.

Other components (such as Oracle HTTP Server) continue to use the Oracle wallet as their storage mechanism.

Info:

Using Widcard is now certified with EBS 12.2.

Okay lets start;

• We start by creating a Wallet.We open Oracle Wallet Manager by using the Applications's OS user account and ovm command; set your environment first, -- dont use owm from the Oracle 10.1.2 home..We need to use the owm in FMW home.

applmgr> export PATH=$FMW_HOME/webtier/bin:$FMW_HOME/oracle_common/bin:$PATH
applmgr> owm & (open the wallet manager in background .. note that you must have X window privileges , I mean with root xclock + ,and you must have necessary rpms installed on your system for example: LibXrender 32 bit :))

On the Oracle Wallet Manager Menu navigate to Wallet -> New.

Answer No to: “Your default wallet directory doesn't exist. Do you wish to create it now?” 

The new wallet screen will now prompt you to enter a password for your wallet. Be sure to make the password something you will remember. You will need to use the password whenever you open the wallet with Oracle Wallet Manager, or perform operations on the wallet using the Command Line Interface. With auto login enabled processes submitted by the OS user who created the wallet, there is no need to supply the password to access the wallet. Click YES when prompted: “A new empty wallet has been created. Do you wish to create a certificate request at this time?”

• Once we created our wallet, the OWM asks us if we want to create a certificate request.. We answer yes, and fill the form accordingly. ( We use the information that our customer provides, I mean OU, State,Country, Key Size etc..)

• After creating our certificate request, we need to export it.

• While exporting, note the directory name when prompted. This is the default directory of our wallet..

• Declare a name for your csr , for example: server.crs and save.. It will create a server.csr file in your wallet directory: For example under : /apps/fs2/EBSapps/10.1.2/owm/wallets/applmgr. It will report the file as saved as below..

• Okay, now we should send this to our customer , as they should submit this Certificate Authority to request a Server Certificate.

The process should be like this. It is documented like this, but sometimes we need to go beyond the documents..

I mean, if the customer has already acquired certification in .P12 file / format; we need to convert it.

We need convert P12 to JKS and then convert JKS to Wallet ..

Here is the method to do that:

keytool -v -importkeystore -srckeystore yourcert.p12 -srcstoretype PKCS12 -destkeystore yournewkeystore.jks -deststoretype JKS

--You must use the same password for the new jks and the private key = "mypassword"

keytool -import -alias Root -keystore yournewkeystore.jks -trustcacerts -file root.cer
keytool -import -alias Intermediate -keystore yournewkeystore.jks -trustcacerts -file intermediate.cer

mw_home\oracle_common\bin\orapki wallet create -wallet ./ -pwd "mypassword"

mw_home\oracle_common\bin\orapki wallet jks_to_pkcs12 -wallet ./ -pwd "mypassword" -keystore ./yournewkeystore.jks -jkspwd "mypassword"

--Make sure the private key password and the wallet password match = "mypassword"

open the newly created ewallet.p12 with Oracle wallet manager and in wallet menu, tick "autologin", then save. This creates cwallet.sso along side ewallet.p12.

• So once we have a walid wallet (ewallet.p12 and an autologin file (cwallet.sso) ), we can continue with the SSL configuration.
• While we are making our wallet autologin using Wallet Manager,  we should also save our wallet in to the directory -> {s_web_ssl_directory}/Apache
• Note that : curly braces are for the Autoconfig variables. You can find the value of them using ;cat $CONTEXT_FILE |grep variable_name
• Note: If all trusted certificates that make up the chain of Server Certificate are not present in the wallet, adding the certificate will fail. When the wallet was created only the certificates for the most common CA’s were included automatically. Contact your Certificate Authority if you need to add their certificate, and save the provided file (for example as ca.crt) in the wallet directory. If your Certificate Authority provided an intermediate certificate (to complete the chain) then save the provided file (for example as intca.crt), this will need to be imported into Oracle Wallet Manager prior to importing the Server Certificate (server.crt if you used the example name).Okay, then we import our root and intermediate certificates in to the b64InternetCertificate.txt file..
• So in this case; we add our certificates to the related file..
cat ca.crt >> <10.1.2 ORACLE_HOME>/sysman/config/b64InternetCertificate.txt
cat intca.crt >> <10.1.2 ORACLE_HOME>/sysman/config/b64InternetCertificate.txt
• Then we copy cwallet.sso file from {s_web_ssl_directory}/Apache directory(where we saved)
to the following directories ;

--note that : we take a backup of cwallet.sso in that directories before copying/overwriting it..

{s_ohs_instance_loc}/config/OHS/{s_ohs_component}/keystores/default

{s_ohs_instance_loc}/config/OPMN/opmn/wallet
$EBS_DOMAIN_HOME/opmn/{s_ohs_instance}/{s_ohs_component}/wallet
$EBS_DOMAIN_HOME/opmn/{s_ohs_instance}/wallet
$FMW_HOME/webtier/instances/{s_ohs_instance}/config/OHS/{s_ohs_component}/proxy-wallet

• Then, we update cacert file;(Oracle WebLogic Server, Oracle Web Services etc uses this file..)

cd {s_fmw_jdktop}/jre/lib/security

chmod u+w cacerts

keytool -import -alias OHSRootCA -file ca.crt -trustcacerts -v -keystore cacerts
keytool -import -alias OHSIntCA -file intca.crt -trustcacerts -v -keystore cacerts
keytool -import -alias OHSServer -file server.crt -trustcacerts -v -keystore cacerts

On a UNIX system the TCP/IP port numbers below 1024 are special in that only processes with root privileges are allowed to listen on those ports..

So, opening a privileged port requires extra steps, but we dont use privileged port normally.. We choose to have a unknown port which is above port 1024..

So , our cacerts and wallet files are configured. We continue with application configuration to use this files to supply SSL communication;

• We first, start with the Web tier configurations;

Login to Oracle Fusion Middleware Control Console (for example http://<hostname>.<domain>:<AdminServer Port>/em)

Select Web Tier Target under EBS Domain
Select Administration > Advanced Configuration
Select ssl.conf file for edit
Update the Listen <port> and the VirtualHost _default_:<port> directives to SSL port, for example Listen 4443   --> your choice -- ssl port
Click Apply

Then run following command by Application Owner OS 'user with Application environment set..

perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE
Enter the APPS user password:
Enter the WebLogic AdminServer password:

adSyncContext.pl will propogate the changes, we just did from the Oracle Fusion Middleware Control Console to the context file variables.

• Secondly , we open OAM(it can be done using vi editor from the OS) and control & change the following autoconfig variables if necessary;

SSL Related Variables in the Context File
Variable	Non-SSL Value	SSL Value
s_url_protocol	http	https
s_local_url_protocol	http	https
s_webentryurlprotocol	http	https
s_active_webport	same as s_webport	Verify the port, correct if required.
s_webssl_port	not applicable	Verify the port, correct if required.
s_https_listen_parameter	not applicable	Verify the port, correct if required.
s_login_page	url constructed with http protocol and s_webport	Verify the protocol and port, correct if required.
s_external_url	url constructed with http protocol and s_webport	Verify the protocol and port, correct if required.

• Lastly, we shutdown our application services, and run autoconfig to make the system configured by considering the new variable values.. Autoconfig makes the SSL configurations System wide.. When the autoconfig is completed, we start our services..

So that's it.. We configured our EBS 12.2 environment to work with SSL.

Some importants notes about the SSL configuration in EBS 12.2:

After this configuration , we will reach our login page using https://hostname:ssl_port

• Even if we use http://hostname:non_ssl_port to reach our login page, it will be redirected to the ssl enabled url https://hostname:ssl_port..
• That is , Our non-ssl tcp port will be open. Note that : We can close it. We can change httpd.conf and comment out line starting with Listen , and restart Apache to do that.. On the other hand, I dont recommend it , because that port is kept open for those products that need to access some of their pages via the HTTP, and for the Oracle E-Business Suite Help System. 
• So in short , you will not be able to use EBS Help pages, if you close that non-ssl tcp port.
• One last thing; With this configuration, our forms will be working in ssl, too.. It is becuase EBS uses servlet architecture in Forms communication layer .. So we dont need to do anything about the Forms. Forms will be communicating in https...
• Wildcards can be used in the Certificates! .. Tested & Verified. A certificate with a servername *.ermandom.com works very well..
• Oracle E-Business Suite Release 12.2 currently supports securing the communication between the end users browser and the data center. Securing OPMN wallet with Fusion Middleware Control Console is detailed in this document but securing the internal communication (Oracle Weblogic Managed Server and other components with Oracle HTTP Server) will be supported in a later release and detailed in this section when they become available.

EBS R12 -- 12.2 -- ABILITY TO USE CAPTCHA

One of our clients requested use captcha in a new relase EBS ( 12.2 ).

The vulnerability was critical for the customer, the severity was high . So they wanted us to provide the information about captaha ability in EBS.

Captcha was needed for the Login  and for the forgot password pages of EBS.

According to my research, I conclude that Captcha is not availabe in EBS..
Never was it, never will be in near future..
if it is a Must to use Captcha in EBS, a custom Captcha funcionality must be deleveped and deployed.

To prove my findings and SR was opened , and the response  was :

This was considered at length by development in Bug 8274552 - ABILITY TO USE CAPTCHA IN IREC DURING USER REGISTRATION. It's confusing that this is called a bug as this is not actually a product defect but in fact an enhancement request. This enhancement request was considered but has been dropped by development. So sorry, no, at this time there are no plans for EBS 12.2 to have CAPTCHA functionality on login page.

EBS 11i&R12 -- Google Chrome problem (Watch out for Google Chrome updates!)

We all know , Oracle doesnt support Google Chrome , but we still use it. In most of the cases, it works well and doesnt create any problems.
We also use Google Chrome for connecting ancient EBS environments :), like connecting to a EBS 11.5.8 :)
Chrome with its EBS enablement works well in such environments too.. At least it have worked well..
Today we have seen a strange behaviour in Chrome.. That is It just could not open Forms in an environment, which was working well yesterday..
The problem was that EBS could not interact with java plugin while the forms are opening.. The little EBS page that loads the client java applet was concerning the lack of a java plugin in the browser.
After some analysis, we have seen that the Chrome browsers in these problematic clients were updated.. They were working well, so the problem must have been started after the upgrade..
So, for the solution, we downgraded Google chrome and disabled its auto updates. After downgrading everyting went back to normal:)

Here is what we have done to accomplish that;

• We uninstall our current Google Chrome using Windows Add/Remove programs

• We gather an older version of Chrome using Google Standalone setup. You can check the version using the following url.. http://www.oldapps.com/google_chrome.php
• We download the required version 's Standalone setup (ChromeStandaloneSetup.exe)

• Then we disable auto update of Chrome using windows registry;

We put it into manual update mode:
Here is the reqistry for that:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Update]
"Update{8A69D345-D564-463C-AFF1-A69D9E530F96}"=dword:00000002

• Then , we install older version of Chrome that we downloaded in the first step. (before installation we disable our internet connection --just in case)
• Lastly, we enable the Chrome EBS enablement for our newly installed Chrome using following url:

https://chrome.google.com/webstore/detail/oracle-ebs-r1211i enablem/ekkagabmggbmpmncofhgkfigmeldifnc

Following document explains related Windows operations..

Discoverer 11g Installation Document (Weblogic installation, RCU activities, Discoverer installation and upgrade activities)

In this post, I m sharing a Discoverer 11g installation document.. It covers Weblogic installation, RCU activities, Discoverer installation and upgrade activities. This document can be considered as a filtered information .. The information provided in here is gathered from Oracle Support , but the installation screens were taken from a real life example. Thanks my collegue Hülya Yılmaz for sharing this document with us.

Note : This is the first PDF that I embeded into this site. :)

Note that:
After installing your Discoverer 11g environment, you need to create a EUL using Discoverer Admin tool. Discoverer Admin tools should be in appropriate version. Also if you have an EUL with earlier version , again you need to upgrade it using Disco admin tool.

Oracle ADF-Browser Certification

It seems ADF is delicate when it comes to the Browsers :)

Nowadays, these ADF applications are popular in EBS world.. On the other hand, even if your EBS runs smoothly in your Browser, the integrate ADF application may produce errors..These errorrs/problems may vary .. Your ADF application may not able to bring any data in to your screens, or it may report an error, or it may not even function at all.
So in order to prevent these kind of problems, you need to know the ADF version which has support for your Browser, and install the proper version of ADF in order to be in safe side..
I share ADF - Browser certification in this post, because I didnt see any ADF releated requirements in Oracle Installation documents.. For example: Hr Organization Chart.. In HR organization chart implementation document , Oracle does not mention any browser requirements related with ADF.. In that document, you only see that ADF 11.1.1.4 or later is required for HR Org Chart application..
So when it comes to the real life, you may end up with errors while using your ADF applications..
The errors may arise because of your ADF release.. Actually the lack of the needed Browser support in your ADF release.
ADF Faces is important component, and it can be used as a reference for ADF-Browser Support matches ..

The Oracle ADF Faces rich client components supported with the following browsers.

Supported Browsers	ADF Faces
Firefox 2.x, 3.x 2, 4+5	Supported
Internet Explorer 7, 8 2, 9 5, 107, 117	Supported
Safari (desktop) 3.2, 4.x 2, 5.x 4, , 6 7	Supported
Safari (iOS) 4.32, 56, 67, 77	Supported
Chrome 1+ 1	Supported
Chrome (Android) 18+ 7	Supported

 Notes:

• IE 11 support added as a post-release certification on 11.1.1.7.0. IE 11 support requires a patch from Oracle Support (patch #18277370 for 11.1.1.7.0)
• The minimum required Firefox version is 2.0.0.20
• For the ADF Faces components that render in Flash, Flash 9 and Flash 10 are certified until 11.1.1.4.0 from which point only Flash 10 is certified.
• For Internet Explorer 8, 9, and 10, only Native mode is supported. View Compatibility mode should be disabled.

-------------------------------
1. Support or certification added in 11g Release 1 (11.1.1.1.0). Does not apply to 11g versions prior to 11.1.1.1.0.
2. Support or certification added in 11g Release 1 (11.1.1.2.0). Does not apply to 11g Release 1 versions prior to 11.1.1.2.0.
3. Support or certification added in 11g Release 1 (11.1.1.3.0). Does not apply to 11g Release 1 versions prior to 11.1.1.3.0.
4. Support or certification added in 11g Release 1 (11.1.1.4.0). Does not apply to 11g Release 1 versions prior to 11.1.1.4.0.
5. Support or certification added in 11g Release 1 (11.1.1.5.0). Does not apply to 11g Release 1 versions prior to 11.1.1.5.0.
6. Support or certification added in 11g Release 1 (11.1.1.6.0). Does not apply to 11g Release 1 versions prior to 11.1.1.6.0.
7. Support or certification added in 11g Release 1 (11.1.1.7.0). Does not apply to 11g Release 1 versions prior to 11.1.1.7.0.

Also, You should check Oracle Support's Certification tab.. Look for Oracle Application Development Runtime..

Here is an example of certification info related with ADF 11.1.1.6.0

Note that: There is no Browser certification information of ADF 11.1.1.7.0 on Oracle Support.
But I can say that , especially for IE 10, ADF 11.1.1.7.0 should be used.

EBS 12.2 / Rdbms -- Listener Poisoning -- Oracle Security Alert CVE-2012-1675

This post will be about a vulnerability, that was discovered in 2008 and was fixed in 2012.
The vulnerability is present in EBS 12.2 , which comes with an 11.2.0.3 Oracle Database.
The problem is addressed in  Oracle Security Alert CVE-2012-1675, and following document explains the fix for that:
Using Class of Secure Transport (COST) to Restrict Instance Registration (Doc ID 1453883.1)
The reason which makes me write this post,is, that I find the document not clear , especially actions mentioned for testing the fix is not clear.

Here is what you need to ;

If you dont use IPC(EBS 12.2 listener does not use it by default)

Apply the patch 12880299 to the affected Oracle Homes.

Set SECURE_REGISTER_listener_name parameter to (TCP) .. (if your listener name is test then set SECURE_REGISTER_TEST=(TCP) in your listener.ora) .. It is better to set it in listener ifile to prevent autoconfig from overwriting it.

Restart your listener and Test..

To test: login to another Oracle Database which resides on a different server. Set remote listener parameter on that listener , and make it try to register to the database listener which you have fixed above.. You will see an error message in the target listener 's log file (TNS-01194: The listener command did not arrive in a secure transport) This means your fix is in place...

Note that: With this configuration, your listener will not accept any registration request except from Local and TCP.

Tested and verified .. I have implemented these steps, and we could pass the Security Tests in a Customer Environment..

EBS 12.2 -- Notification mailer SSL configuration

In this post, I will explain how to use SSL configuration in Workflow Notification mailer briefly.

As you know, Notification mailer has the ability to work with smtp and imap. Also Notification mailer can use these protocols with SSL configuration. So if you enable SSL in your configuration, your notification mailer will speak smtp and imap securely..

The logic is the same as EBS Web server SSL implementation.
We export the certificate, the Autority signs it , we store it and lastly we configure our applications (in this case notification mailer) to use it and work in SSL. The export phase is somewhat different, because in ssl implementation of Notification mailer, to certificate is exported from the mail server..

There is video on Oracle Support to be followed. -> Workflow Mailer Configuration SSL Implementation (1324799.1)

Here is the action plan for enable SSL in Notification mailer:

We export the certificate ..Following video explains how to export a certificate in details ;

https://support.oracle.com/epmos/main/downloadattachmentprocessor?attachid=1324799.1:EXPORT_CERTIFICATE&width=100%&height=100%

A. Access your main web page https://<host.domain:port>.
B. Double-click on the padlock at the bottom of the page to view the Certificates.
If there is no padlock, then on the top toolbar:
select File->Properties->Certificates
a) click on the first line and then View Certificate. - This will be the certificate for the root
Certifying Authority (CA).
b) On Details tab click Copy to File, this will start the export wizard.
c) Click Next to continue.
d) Select Base-64 encoded X.509 (.CER) and click next.
e) Enter ca1 as the name and click ok to export the certificate.
f) Repeat steps a thru e for each line on the Certification Path tab incrementing the file name each
time by 1, i.e. ca2, ca3.

Next, we create a keystore and import our certificate to it.

a. Create a keystore that holds the certificate, use the default location (i.e. /home/applmgr)
keytool -genkey -alias -keystore -storetype JKS
Pass : ermanpass
Note: Answers for the questions like CN , OU  can be gathered from the information written in the certificate

b. Upload the Certificate to the Concurrent Manager tire using a ftp tool then import the certificate into the
created keystore:
keytool -import -alias my_cert -file server.cer -keystore .keystore -storepass ermanpass
c.Verify the cetificate
keytool -list -keystore .keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries
my_cert, Jul 1, 2014, trustedCertEntry
Certificate fingerprint (MD5): 12:7A:55:50:6C:76:54:38:13:1D:EC:BF:FF:32:40:95
-keystore, Jul 1, 2014, PrivateKeyEntry,
Certificate fingerprint (MD5): 4F:F6:43:44:FD:C0:98:FE:DF:68:7D:C8:F8:0A:67:BE

Then we set MAILER_SSL_TRUSTSTORE (i.e. /home/applmgr/.keystore )

From sqlplus:
$FND_TOP/sql/afsvcpup.sql
Enter Component Id: 10006 ( Workflow Notification Mailer)
Enter the Comp Param Id to update : 10229
You have selected parameter : SSL Trust store
Current value of parameter : NONE
Enter a value for the parameter : /home/applmgr/.keystore

Lastly, we configure the notification mailer to work in SSL..
We configure the mailer through OAM.. We just check the checkboxes(outbound ssl enabled &Inbound SSL enabled) for SSL (IMAP and SMTP) and restart our mailer..

Weblogic -- Increasing Memory Size / Managed Server High Cpu usage/ Weblogic Console hang

Weblogic managed servers can spin in the Cpu if the memory they may use is not enough for their work.
You can monitor situation using Linux top command.
Just after starting your managed server, issue top command.
Look for RES column , this will display the physical memory that managed server uses at a time.. Also look for the CPU column, for the percentage of the CPU resource that your managed server uses..
In a memory leak situation  , you will see that , after starting your managed server, its RES column will increase .. It will increase but stop increasing at a limit.. This limit is actually the limit that your managed server's java process is configured.. I am talkin about Xms Xmx XX:PermSize and XXMaxPermSize.. In some situation you will see your weblogic managed server consumes all the CPU cycles in the system and a hang situation arises because of it.. You will not able to navigate even in your weblogic admin console, when this happens.

The solution for this problem is to increasse the memory limits of the problematic Managed server..

You can increase the limit by using Admin console before starting your managed servrer..
You should choose your managed server and put following parameters in to the argument section which resides the Server Start tab..

-Xms2048m -Xmx4096m -XX:PermSize=1024m -XXMaxPermSize=1024m

For example :

Thanks to http://itbitsolutions.files.wordpress.com for sharing this screenshot.

When you set these memory parameters properly and start your managed weblogic server, you will see that it will use CPU and it will increase the memory usage (RES column) in the first start, but then after a while it will finish its works and release the CPU, as well as stabilize its Memory usage.

Lastly, I m sharing the definitions of java memory parameters used in this document as follows;

Thanks to http://www.avricot.com/ for the info below..

Permanent Space is  used by stored classes, methods etc. It is controlled by ;
-XX:PermSize: initial value
-XX:MaxPermSize: max value

Java Heap is a continous memory region where all Objects data will be stored (by data, we mean instance of class, primitive and references). controlled by,

-Xmx : max heap size
-Xms : min heap size.
-Xmn : the size of the heap for the young generation
Young generation represents all the objects which have a short life of time. Young generation objects are in a specific location into the heap, where the garbage collector will pass often. All new objects are created into the young generation region (called "eden"). When an object survive is still "alive" after more than 2-3 gc cleaning, then it will be swap has an "old generation" : they are "survivor" .
Good size is 33%
-XX:NewRatio : the same as Wmn, but using a % (dynamic fs static -Xmn option). -XX:NewRatio=3 means that the ratio between the old and young generation is 1:3
-XX:NewSize - Size of the young generation at JVM init. Calculated automatically if you specify -XX:NewRatio
-XX:MaxNewSize - The largest size the young generation can grow to (unlimited if this value is not specified at command line)
-XX:SurvivorRatio : "old generation" called tenured generation, ratio, in %. For example, -XX:SurvivorRatio=6 sets the ratio between each survivor space and eden to be 1:6 (eden is where new objects are created)
-XX:MinHeapFreeRatio: default is 40%. JVM will allocate memory to always have as minimum 40% of free memory. When -Xmx = -Xms, it's useless.
-XX:MaxHeapFreeRatio: default is 70%. The same as Min, to avoid unecessary memory allocation.

Weblogic -- Weblogic Java processes hang- Admin Console may work slow -- Watch out /dev/random!

I have faced with this incident in a Weblogic 10.3.6 environment running with 64 bit jdk 7 upgdate 5 on a Redhat Linux 6.3 64 bit system.
Okay, before characterizing the problem and explaining the workarounds, Lets first start with /dev/random..
/dev/random is a special file .. It is a blocking number generator. /dev/random is based on the entropy pool, which is compromised by  random bits that are assumed to be unknown..  The bits are calculated from user-triggered events (keystroke, disk I/O, mouse clicks etc
The mechanism is like;
When random bits are requested, return that many bits derived from the entropy pool (by a cryptographic hash function, say) and decrement the estimate of the number of random bits remaining in the pool. If not enough unknown bits are available, wait until enough are available.
So it is blocking.

Okay, lets take a look to the /dev/urandom..
/dev/urandom is an unlimited, non-blocking random source, which reuses the internal pool to produce more pseudo-random bits.

The key difference above is that /dev/random is a blocking number generator, so it may block the processes which are trying get random number from it.

So, this blocking situation may arise if the remaining bits decrease in the entropy pool..

But how this affectes Weblogic processes?

We have /dev/urandom set in our java.security files by default, (java.security file is in the JRE folder of the JDK that our WEblogic server uses -- jre\lib\security\java.security) ,so we should not get affected from it..

Actually , yes we may be affected. Our weblogic processes may hang and these hang situation can make us feel the slowness even in the weblogic console screens..

So lets look at this this bug : 

JDK-6202721 : SHA1PRNG reads from /dev/random even if /dev/urandom selected

If you do

import java.security.SecureRandom;

class JRand {

  public static void main(String args[]) throws Exception {

    System.out.println("Ok: " +

      SecureRandom.getInstance("SHA1PRNG").nextLong());

  }

}

then SecureRandom will read from /dev/random even if securerandom.source is configured to use /dev/urandom. This is a problem if /dev/urandom was chosen because /dev/random is not working properly.

The root cause is that 4705093 assigned special meaning to the string "/dev/urandom".

So it is magic. When you say /dev/urandom, it goes to /dev/urandom :)

Workaround 1)

The workaround is specfiying securerandom.source=file:/dev/./urandom rahter than securerandom.source=file:/dev/urandom to eliminate the magic. :)

Workaround 2)

Using Oracle Linux 's "rngd" which can generate entropy data for the /dev/random pool following certain rules that can defined in a configuration file.

Workaround 3)

In setDomainEnv.sh
if [ "${USER_MEM_ARGS}" != "" ] ; then
MEM_ARGS="${USER_MEM_ARGS}
export MEM_ARGS
fi
MEM_ARGS="${MEM_ARGS} -Djava.security.egd=file:/dev/./urandom"
Workaround 4)
While configuring your domain;
export CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"
/u01/app/oracle/product/fmw/wlserver_12.1/common/bin/config.sh

So, the behaviour of java for /dev/urandom is confusing but JDK 8 seems promising for clearing this confusion..

If you call: 

 o new SecureRandom() on Linux and the default values are used, it will read from /dev/urandom and not block. (By default on Solaris, the PKCS11 SecureRandom is used, and also calls into /dev/urandom.) 

 o SecureRandom.getInstance("SHA1PRNG") and do not specify a seed, *OR* new SecureRandom() but have specified an alternate java.security.egd besides "file:/dev/urandom", it will use the SHA1PRNG which calls into /dev/random and may potentially block. 

 o SecureRandom.getInstance("NativePRNG"), it will depend on what java.security.egd is pointing to.

EBS 12.2 -- Securing EBS

There are some documented configurations in order to have a secure EBS R12 environment.
Lets explain them by walking through the "Secure Configuration Guide for Oracle E-Business Suite Release 12".

• Ensure the ownerships of the binaries in ORACLE_HOME belong to Rdbms Software Owner.
• Check that all the files under $APPL_TOP belong to the Application Software Owner.
• Disable remote logins to the Rdbms and Application Software owner account .. Instead, create one account for each System Admins , and make them use their own accounts to login to the servers, and make them use "su" command to connect to the Rdbms or the Application Software accounts. (sudo can be implemented, too)
• Set the permissions on $ORACLE_HOME/bin to 0751 or less. Set all other directories in $ORACLE_HOME to 0750 or less. Note, this limits access to the Oracle user and its group (probably DBA).
• The file permission of  listener.ora and sqlnet.ora to 0600.
• The file permission of  tnsnames.ora to 0644.
• The file permission of  database data files*.dbf to 0640.
• chown -R <oracle> $ORACLE_HOME
• chgrp -R <dba> $ORACLE_HOME
• Make the necessary permission configurations in order to ensure that the files under $ORACLE_HOME/rdbms/admin directory can not be deleted..
• The file permission of the dbc file located in $FND_SECURE/ should be 600set userid (SUID) and set group id (SGID) shoud not be set on the Oracle Binaries.

Note that : 
Oracle recommends that the SUID and SGID bits to be removed from binaries shipped by Oracle.
Warning: If Concurrent Manager runs on the Database tier and using the BEQ adapter to avoid TCP cost,the SUID and/or SGID bit must be set on the Oracle database executable in $ORACLE_HOME/bin. This may also apply for any third party products running on the db tier.

• X server is not required after the installation, it can be closed.
• Open printer ports if necessary but restrict access to these printers..
• Ssh should be used for remote connections .(not telnet,rsh,rlogin,rcp or ftp)

• XDB support is not required in Oracle E-Business Suite. So the XDB service that is serviced from the listener should be disabled.To disable XDB, remove or comment out the line in init.ora that reads *.dispatchers='(PROTOCOL=TCP) (SERVICE=sidXDB)'

• Database Links should be reviewed.. The unnecessary db links should be dropped. Public database links should not be created.

• Database parameter REMOTE_OS_AUTHENT should be set to FALSE in init.ora REMOTE_OS_AUTHENT specifies whether remote clients will be authenticated with the value of the OS_AUTHENT_PREFIX parameter.

• There should be 2 password profiles in the database. One for the Applications and one for Administrators. Password management profile should be configured as follows;

• Database user password should  be changed with "alter user" and Application passwords should be changed with FNDCPASS

For example :
alter user <SCHEMA> identified by <NEW_PASSWORD>;
$ FNDCPASS APPS/<apps_pwd> 0 Y SYSTEM/<system_pwd> ALLORACLE <NEW_PWD>
(AFPASSWD can be used if the EBS version >=12.1.2)

Note that: Application users can be listed using adutconf.sql

DB schemas are as follows;

Note that : Patch 4926128 can be used to list  all open accounts with default password in the database. DBA_USERS_WITH_DEFPWD view also lists these accounts. It is present in 10g&11g.

• The init.ora parameter _TRACE_FILES_PUBLIC should be set to FALSE. This will grants file system read access to anyone who has activated SQL tracing.

• Set REMOTE_OS_ROLES init.ora parameter to FALSE.

• Limit the filesystem accesses from the database through the application. set utl_file_dir.. Avoid using *. UTL_FILE_DIR = <dir1>,<dir2>,<dir3>...

• Prevent select any privilege from reading the data dictionary tables using O7_DICTIONARY_ACCESSIBILITY = FALSE.

• Revoke unnecesary grants from Applsyspub account..

To check permissions, login as SYSTEM and issue the following query:

SELECT * FROM dba_tab_privs WHERE grantee ='APPLSYSPUB';

To revoke unnecessary privileges granted to APPLSYSPUB schema, see patch 3763612.

Applsyspub shoule have the following grants; any extra grants should be revoked.

The grants of applsyspub should be listed with the following sql: 

SELECT * FROM dba_tab_privs WHERE grantee ='APPLSYSPUB';

Patch 3763612 can be used to revoke the unnecessary grants of APPLSYSPUB.

Applsyspub proper grants:

EXECUTE ON FND_PUB_MESSAGE

EXECUTE ON FND_SECURITY_PKG

EXECUTE ON FND_SIGNON

EXECUTE ON FND_WEBFILEPUB

INSERT ON FND_SESSIONS

INSERT ON FND_UNSUCCESSFUL_LOGINS

SELECT ON FND_APPLICATION

SELECT ON FND_APPLICATION_TL

SELECT ON FND_APPLICATION_VL

SELECT ON FND_LANGUAGES_TL

SELECT ON FND_LANGUAGES_VL

SELECT ON FND_LOOKUPS

SELECT ON FND_PRODUCT_GROUPS

SELECT ON FND_PRODUCT_INSTALLATIONS

• Enable Audit trail .. Enable auditing to the database or OS.

AUDIT_TRAIL = OS or DB or TRUE

Audit the operations:

SQL>audit session;

SQL>audit user; -- any changes to the standard Oracle E-Business Suite database schemas or creation of new schemas

SQL> AUDIT DATABASE LINK; -- Audit create or drop database links

SQL> AUDIT PUBLIC DATABASE LINK; -- Audit create or drop public database links

SQL> AUDIT SYSTEM AUDIT; -- Audit statements themselves

SQL> AUDIT ALTER ANY ROLE by ACCESS; -- Audit alter any role statements

SQL> AUDIT ALTER DATABASE by ACCESS; -- Audit alter database statements

SQL> AUDIT ALTER SYSTEM by ACCESS; -- Audit alter system statements

SQL> AUDIT CREATE ROLE by ACCESS; -- Audit create role statements

SQL> AUDIT DROP ANY ROLE by ACCESS; -- Audit drop any role statements

SQL> AUDIT PROFILE by ACCESS; -- Audit changes to profiles

SQL> AUDIT PUBLIC SYNONYM by ACCESS; -- Audit public synonyms statements

SQL> AUDIT SYSDBA by ACCESS; -- Audit SYSDBA privileges

SQL> AUDIT SYSOPER by ACCESS; -- Audit SYSOPER privileges

SQL> AUDIT SYSTEM GRANT by ACCESS; -- Audit System grant privileges

If audit trail is set to DB, the records can be analyzed using the following views;

DBA_AUDIT_EXISTS

DBA_AUDIT_OBJECT

DBA_AUDIT_SESSION

DBA_AUDIT_STATEMENT

DBA_AUDIT_TRAIL

DBA_OBJ_AUDIT_OPTS

DBA_PRIV_AUDIT_OPTS

DBA_STMT_AUDIT_OPTS

Column Definitions:

Username: Oracle Username.

Terminal: Machine from which the user originated.

Timestamp: Time the action occurred.

Object Owner: The owner of the object that the user touched.

Object Name: The name of the object that the user touched.

Action Name: The action that occurred against the object (INSERT, UPDATE, DELETE, SELECT,

EXECUTE

Note that: These audit records should be archived and purged periodically.. Period should be at least 90 days. Restrict access to audit records accordingly.

• Use Apache trusted.conf to limit the web page access as follows;

The <list of TRUSTED IPs> is being replaced with the value of the AutoConfig variable

s_admin_ui_access_nodes which you should set to the list of host machines from which administrators connect.

<Location "uri-to-protect">

Order deny,allow

Deny from all

Allow from localhost <list of TRUSTED IPs>

</Location>

• Stop adpatch from logging passwords;

adpatch flags=hidepw

• set WORKFLOWNOTIFICATION MAILER SEND_ACCESS_KEY TO "N" . When this is set to "N", an unauthenticated user who clicks on the notification link must sign on before accessing the Notification Details web page.

• Forms users should be prevented from using the enter-query feature on a production system.In default.env file ;  /x/inst/apps/VIS12_dbs01/ora/10.1.2/forms/server/default.env , set FORMS_RESTRICT_ENTER_QUERY to TRUE.

• Set the following profiles to restrict the attachment operations;

Attachment File Upload Restriction Default | FND_SECURITY_FILETYPE_RESTRICT

_DFLT -> "N (whitelist)" 

When enabled it will by default restrict using a blacklist of the file extensions that Windows consider

"executable" such as .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC.

Upload File Size Limit|  UPLOAD_FILE_SIZE_LIMIT -> "as needed"

While you are considering attachments, you may want to check the setting of the maximum allowed size of an

uploaded attachment, see the profile option UPLOAD_FILE_SIZE_LIMIT (kb).

• Disable the Antisany using following profile option;

FND: Disable Antisamy Filter | FND_DISABLE_ANTISAMY_FILTER -> N

The Antisamy filter ensures that the HTML document only contains an allowed subset of HTML (no scripting or other “dangerous” tags are allowed).

• SSL should be used between Browser and Web Server..

376700.1 “Enabling SSL for Oracle Applications Release 12”.

Avoid SSLv2 and all ciphers with a key size less than 128 bit.
SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
Note that:

If you are concerned about the BEAST attack (CVE-2011-3389), you can change the SSLCipherSuites setting to
use the RC4 stream cipher instead of a block cipher.
The BEAST attack exposes a client side weakness in SSL v3 and TLS 1.0 when used with Block Ciphers in
CBC mode.
The following line will make the server prefer RC4 stream cipher
SSLCipherSuite RC4-SHA:MEDIUM:!aNULL::!MD5:!ADH
You will have to leave SSLv3 enabled as OHS 10.1.3.x uses nzlibs version 10 which do not support TLS.
The UTL_HTTP PL/SQL client in the database also does not currently support TLS.

• If EBS will be open to the external sites , an External WebTier should be used. (DMZ configuration)--more secure..

• Terminal Services should be used for direct client connections. (secure trusted servers rahter than desktop clients) -- Client-Server , for example : Toad connections should be from a Terminal server.

• Passwords of Seeded Application  accounts should be changed.

If necessary, the accounts can be disabled. You can disable an application user account by setting the END_DATE for the account.

Note that :
Do not disable the GUEST user account
Do not disable the SYSADMIN user account until you have created other accounts with similar privilege
fnddefpw.sql executed as Apps will display  the seeded accounts that still have the default password.
In the out put of the script;
an ‘x’ in the EndDT column means the account ship end-dated.
an ‘x’ in the NoPwd column indicates that the account ships with an “impossible password”, this
means that the password column in FND_USER contains a clear text string that is never a valid encrypted or
hashed password. Thus it is not possible to login as this user, - unless you change the password!
The “impossible” value can be “DUMMY”, “INVALID” or “INTERNAL USER-NOLOGIN“.
You can easily identify the users with an impossible password as the length of the impossible password is
shorter than the encrypted or hashed password. For example this SQL statement will list users with in impossible
password:
select USER_NAME,END_DATE,ENCRYPTED_USER_PASSWORD from FND_USER
where length(ENCRYPTED_USER_PASSWORD) < 30 order by 1;

• Hashed password should be used instead of encrypted password stored in FND_USER table;

MOS Note 457166.1 “FNDCPASS Utility New Feature: Enhance Security With Non-Reversible
Hash Password”

• Strong passwords, account lockouts and session inactivity timeouts shoud  be configured using followind profile options:

SIGNON_PASSWORD_LENGTH 8
SIGNON_PASSWORD_HARD_TO_GUESS YES
SIGNON_PASSWORD_NO_REUSE 180
SIGNON_PASSWORD_CASE Sensitive
SIGNON_PASSWORD_FAILURE_LIMIT 5
ICX_SESSION_TIMEOUT 30
SIGNON_PASSWORD_CUSTOM implement (optional -- if standards are not enough , you can impelement a custom validation function and register it with EBS.)

• Single Sign On (SSO) should  be used..

Note 376811.1 “Integrating Oracle E-Business Suite Release 12 with Oracle Internet Directory and Oracle Single Sign-On”.

• Make the user creation operations through UMX (Oracle User Management)
• Use shared responsibilites rahter than shared accounts.
• Configure Concurrent Manager for safe authentication. That is, concurrent managers pass apps schema password to the concurrent programs on the command line.. Something that can interpret the command line can steal the passwords.. To prevent this define the concurrent program executable as a HOST program in the Concurrent Program Executable form. Enter ENCRYPT in the Execution Options field of the Concurrent Programs window when defining a concurrent program using this executable. ENCRYPT signals Concurrent Manager to pass the username/password in the environment variable FCP_LOGIN. Concurrent Manager leaves argument $1 blank.

To prevent username/password from being passed, enter SECURE in the Execution Options field. With this change, Concurrent Manager does not pass the username/password to the program. This of cause means that the program will have to get the database credentials some other way if it needs to connect to the database.

• Configure Start/Stop scripts of the Concurrent Manager to able to work without apps credentials..

To implement this;
Create an application user named CONCOPER, and assign Concurrent Manager Operator responsibility to this user.
Then set following autoconfig variables in the application tier and run autoconfig in the application tier.
s_cp_user ->CONCOPER
s_cp_password_type -> AppsUser
s_cp_resp_shortname -> FND
s_cp_resp_name -> Concurrent Manager Operator

After this change we will use -secureapps argument when starting the control scripts..

For Ex:
sh adstrtal.sh -secureapps
enter application username:CONCOPER
Enter applications password:

• Check server security status.. It should be secure..

select NODE_NAME,SERVER_ID,SERVER_ADDRESS from FND_NODES
where SERVER_ADDRESS = '*'
NODE_NAME SERVER_ID SERVER_ADDRESS
------------------  ------------------ ----------------------------
AUTHENTICATION SECURE  *

-- Note that : SECURE Recommended; only registered application servers and trusted code modules may connect.

• Review document 1334930.1 "Sensitive Administrative Pages in Oracle E-Business Suite".. The access to these pages should be minimized, eliminated. Only System Administrator and some trusted users should access these pages.

• Set Sign-on Audit level profile option  to "form" to  enable as much auditing possible. With this setting operations like sign-on , responsibility selections, form accesses are audited and saved in the fnd_logins,fnd_login_responsibilities and fnd_login_resp_forms tables.

End user access data stored in tables below :
APPLSYS.FND_LOGINS
 APPLSYS.FND_LOGIN_RESPONSIBILITIES
APPLSYS.FND_LOGIN_RESP_FORMS
APPLSYS.FND_UNSUCCESSFUL_LOGINS
FND_CONCURRENT_REQUESTS
ICX.ICX_FAILURES

FND_LOGIN table should be archived and truncated periodically. Purge Signon Audit Data concurrent program  should be scheduled to run once a week to retain 30 to 90 days of records.. You can report audit records using seeded report programs..

• Track page accesses using OAM. Note : 402116.1 Page Access Tracking.

• Make the necessary configuration in database listener to restict the connections.

Enable validnode_checking in sqlnet.ora  to restrict the serves which can be connected to the database..
tcp.validnode_checking = YES
tcp.invited_nodes = ( X.X.X.X, hostname, ... )
--Note that: servers should be Application server , database server and if needed some thrid party application servers, or tool such as Toad -- not recommended for Production.)
These validnode checking configurations can be done using Autoconfig also..
If the profile option “SQLNet Access” (FND_SQLNET_ACCESS) is set to “ALLOW_RESTRICTED” at the Site level when AutoConfig is run on the database server, AutoConfig will add IP restrictions to sqlnet.ora.

• Listener connection timeout should be configured in listener.ora CONNECT_TIMEOUT_$ORACLE_SID = 10  --> $ORACLE_SID is ORACLE_SID not the the listener's name)

• Tns listener traffic should be encrypted.. (This requires Advanced Security Option!)

To enable encryption;

In $TNS_ADMIN/sqlnet_ifile.ora;
SQLNET.ENCRYPTION_SERVER = REQUIRED
SQLNET.ENCRYPTION_TYPES_SERVER= (AES256, AES192, 3DES168)
SQLNET.CRYPTO_SEED = somelongandrandomstringforyourdeploymentUpTo70characters

Note that : We need to allow both the AES and the triple DES ciphers as the OCI and JDBC client side code varies in their cipher support.

• The Listener should be configured to ask for passwords;

Normally, listener uses OS authentication.. That is the user that owns the software can control the listener.
To enable the authentication, simply set s_enable_listener_password to ON and run Autoconfig on the database server.

Set the current listener, then change the password.
LSNRCTL> set current_listener VIS12
Current Listener is VIS12
LSNRCTL> change_password
Old password: -- just hit return
New password: -- new, secure password
Reenter new password: -- new, secure password
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbs01)(PORT=1541)))
Password changed for VIS12
The command completed successfully
LSNRCTL> set password
Password:
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbs01)(PORT=1541)))
Saved DBLSNR configuration parameters.
Listener Parameter File /x/db/tech_st/10.2.0/network/admin/VIS12_dbs01/listener.ora
The command completed successfully
This added the following lines to listener.ora:
#----ADDED BY TNSLSNR 13-JAN-2007 11:47:56---
PASSWORDS_VIS12 = D911537D50B15546
#--------------------------------------------

• Set ADMIN_RESTRICTIONS_<listener>=ON in listener.ora

Alternatively, autoConfig can set this if you set the AutoConfig variable s_admin_restrictions to ON and run AutoConfig on the database server.

ADMIN_RESTRICTIONS disables lsnrctl usage. So after enabling it ; only way to change the configuration is to edit the listener.ora file.

• Enable logging in listener by setting the following parameters in listener.ora;

LOG_STATUS = ON
LOG_DIRECTORY_$ORACLE_SID = $TNS_ADMIN
LOG_FILE_$ORACLE_SID = $ORACLE_SID

• In the client site;

Configure browser settings: 389422.1 . Update browser with the security enhancements (only to the supported level)

Turn off autocomplete.

Use password locked screensavers.. Lock the screen through a corporate policy.

• In the network site;

Use firewall or router machine, filter unused port/services out.Block all rpc ports on the router.(unless nfs is used between the network)

In /etc/ssh/sshd.conf limit which users can connect  to the machine.

Turn of source routing and filter packets from outside the network that have source IP address from inside the network.

Automatically disable OS users that have failed login attempts.. Use hard-to-guess OS user passwords. Change the password on a reqular basis.

Okay, That 's it.  I have try to show up whats need to be done to have a secure EBS R12 environment. I tried to write the actions which seem necessary even in the first sight.

Note that : Of course there are some extra configurations and/or some advanced security configurations which can be implemented to enhance the security .

 Please refer "Extras for Expert" section of "Secure Configuration Guide for Oracle E-Business Suite Release 12" for those configurations.

Important Note: Weblogic is a big part of EBS 12.2.. So securing Weblogic is an essential subject to have a secure EBS environment.  

The document named : 

Oracle® Fusion Middleware Securing Oracle WebLogic Server should be analyzed for this. The possible effects that may change the way that EBS operates,  should be considered before making any changes to the Weblogic.

http://docs.oracle.com/cd/E14571_01/web.1111/e13707/toc.htm

EBS 12.2 -- Recovering Cutover Problems, Adop phase=cutover, Flashback

We know that we can abort and cleanup the patching works till we run the cutover phase. (adop phase=cutover)
This document will be about recovering and aborting a patch cycle even during a cutover.
The methods described in here can be implemented in problematic scenarios such as event of a failed cutover, and as a last resort
If a cutover error occurs, you should first check the error message and try to determine if the problem can be fixed easily, or (as is true in many cases) cutover can be made to succeed simply by running the command again. Restoring to a point before cutover via Flashback recovery should only be done when the error cannot easily be fixed, and continues to fail on subsequent cutover attempts.

So, to rollback the system to a point before the patching cycle was started, we can use Oracle Database's Flashback feature. We can go back to our restore point that we created just before we run the cutover phase. Note that: While creating a restore point, it is recommended to stop application services..

We must create this restore point when; 

we are ready to perform cutover.
All concurrent managers have been shut down cleanly.
There are no current database transactions being performed by any third-party applications.

In order to be able to use flashback restore points we must enable the flashback in the database tier..
We must be in archivelog mode.
We must set our retention to a sufficient value. alter system set db_flashback_retention_target=120 (in minutes)

Activate flashback . alter database flashback on;

Okay, once we activate the flashback, we can create restore point

So just before cutover phase;

we create our restore point as follows;

SQL>alter system switch logfile;
System altered.
SQL>create restore point BEFORE_CUTOVER guarantee flashback database;
Restore point created.
SQL>alter system switch logfile;
System altered.

So we created our Restore point ..

Now suppose we have encountered a problem in the cutover phase, and want to restore our database just before the cutover;

To able to do this;

we first shutdown our database and startup in mount mode as follows;

SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.

Then we issue the following command to restore our database to the restore point;

SQL>flashback database to restore point BEFORE_CUTOVER;
Flashback complete.

Then we open our database in read only mode and check everyting is clear.

alter database open read only;

Lastly, we shutdown and startup our database with openresetlogs option..

SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
Total System Global Area 2142679040 bytes
Fixed Size 1346140 bytes
Variable Size 520095140 bytes
Database Buffers 1593835520 bytes
Redo Buffers 27402240 bytes
Database mounted.
SQL>alter database open resetlogs;
Database altered.

Once the database is altered, we can disable the flashback and drop the restore point as follows;

SQL>alter database flashback off;
Database altered.
SQL>drop restore point BEFORE_CUTOVER;
Restore point dropped.
SQL>alter system set db_recovery_file_dest='';
System altered.
SQL>select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------
NO

Okay we have seen how to revert our database back before the cutover phase , but we may also need to restore our applications file system before the cutover phase, as well..

We can understand if there is a need for doing by looking to the cutover logs...

$NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.
Case 1 - If the log messages indicate that cutover failed before the file systems were switched

If it is the case;

We just run adstpall.sh and clean the running services.. Then we restart them using adstrtal.sh.

Then we continue with aborting the patch cycle and doing a cleanup.

Case 2 - If the log messages indicate that cutover failed after the file systems were switched..

If this is the case; 

We shutdown the application services.

We switch the filesystems..

perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>

Lastly start up all services from the old run file system (using adstrtal.sh on UNIX).