EBS -- Patching Tips

 In this post, you can find my patching tips.These tips can be evaluated for Oracle EBS 11i and R12.. These tips are written according to my experiences gathered from various patching and upgrade projects I have done. I hope you will find these tips helpful...

• Record your invalid objects, concurrent manager informations, notification mailer configurations and etc.
• Check you tablespaces. Ensure they have enough free space.
• Check you filesystem. Ensure you have enough free space.
• Disable your custom login triggers.
• Disable your password verify functions.
• Stop your database jobs. 
• Cancel your scheduled backup or maintanence jobs.
• Analyze all the patches carefully and prepare a patch list with the columns.
• . module name: The module that the patch is relevant.
• . merge name : The name of the merge patch ( if you will merge the patch with the other patches)
• . patch number : self explanatory
• . prereq/actions: prereq patch numbers and prereq actions that need to be taken prior applying patch
• . postreq/actions : postreq patch numbers and postreq actions that need to be taken prior applying patch
• . Nls language patch: specify if NLS version of the patch is available.
• . Description: Description of the patch elapsed time
• . Problems&Solutions: If you encounter an error while applying the patch, note it here.. Write the solution also. Give details or link to the detailed documents.
• . Reference Doc: The Oracle Support document number that points to this patch.. If it s prereq or postreg, write the relevant patch number here.
• . Elapsed Time: Write down your timings for applying the patch. Consider the overhead of NLS language patches and manual actions too.
• Download all the patches in to a folder and unzip them beforehand. It s better to have a seperate/specific foler for some patches(like db psu) , as some kind of patches have multiple directory structures.. (like two directories ...)
• While applying the patches(especially the big ones), always work with a VNC server..
• Merge the patches if you can.. Remember AD patches can be merged with AD patches only.. Altough it s not recommended, you can merge patches with their NLS patches.. You can do this if you dont have long downtime.. Otherwise; merge US patches and NLS patches seperately.
• You dont need to create only one merged patch.. Analyze your stylesheet, look the manual actions. If a patch has important manual action, dont merge it with other patches. Create a merge till you come to that patch, and create another merge after that patch. 
• By doing so, you can take your manual actions, and continue your process without question marks.
• Analyze your system. Your actual cpu counts, your IO power, and decide adworker count that you will use for applying the patches.(Consider your operating system limits too.)
• Dont use high parallelism for small patches..
• Always monitor your archive log dest and tablespace sizes..
• Always check you environment.. For example, if you are applying a patch using opatch for 10.1.3 home, set your environment to reflect 10.1.3 home.For adpatch you need to be on the default environment. Dont forget to take your environment back to the original, after you applied a patch on 10.1.3 Oracle home.
• If you have custom partitions or custom modifications on seeded tables, you can encounter error during some patch applications. Control them carefully, and if you believe you can skip that particular work, make the adworker skip that failed job, and take manual actions later.(for example, if adworker can not create index because of custom subpartitions)
• Use nocompiledb and noautoconfig options if needed.. Analyze and determine this kind of patches in your first iteration.. These options may save you significant time..
• While applying some patches, you will find adsstats.sql running.. If running this sql takes a long time and prevent you to continue; kill the db session of the adworker and skip the job using adadmin 8th option.. (adsstats.sql: this sql is used for gathering the statistics for relevant tables..)
• If you have multiple application nodes with Shared Apps filesystem, apply the patches from the admin node only, but if you see patch running autoconfig,manually run autoconfig on other nodes too..
• Use the defaultsfile option of adpatch, if there are a lot of application patches that you need to apply. By using defaultsfile, you will save time and energy.
• If you need to stop a patch application and adworkers are running; use adctrl to stop the patch application. In such a scenario, always check the database too.. By stopping a patch application, you can leave some worker sessions working in the database..Kill them too..
• Always take backups when you reach the milestones..
• When you finished you patching/upgrade process-> check the invalids and correct them taking necessary actions (apps compile, flexfiled compile and etc), check the file versions, check bug records, check application patch levels..
• Once you ensure that everyting is okay, enable all the custom logon triggers and password profile functions, that you disabled before. Take any changes that you have done before the upgrade back to the original.
• Before opening the applcation services, contact the necessary persons, and ask their approval to open the services. This is necessary for your own sake. EBS systems are mission critic systems, there are integrations and sensitive data on them.. Opening the services without controlling them functionally, can leave you in an irrecoverable state.. 
• Before opening the services, you can take your important and unrestartable concurrent programs on hold. 
• After you open the application services, check concurrent managers, notification mailers, oaf screens and forms screens.

These are the things that comes to my mind at this moment..
I will update this post from time to time.. 

You can add other statements too.. It will be helpful for all of us.