EBS new release 12.2 have started to be used in the new implementations. It is still quite new, but eventhough it is new, it is preferred in the new EBS projects, as it brings a quite new technological architecture and big advancements. In the last 2 months, we have done 4 different EBS 12.2 installations&upgrades(including 1 VM template import operation) and I can say that we have met with the new technology and we are glad for the new features altough getting used to the new EBS Apps Dba capabilities was a little challenging.
Almost the entire middleware architecture is different, its installation capabilities are advanced, and patching on EBS 12.2 is a lot different. We use different tools for several administration activities and we have new tools for monitoring them.. Also we dont use a lot of things, which were mandatory in the previous releases, because they are obsoleted now.
Today, I will write an article about one of these new features in EBS, which is Online Patching. Online Patching is a big advancement in EBS, and it brings a completely new patching architecture to us. It is a bit tricky and complex, but it brings a lot of advantages..
In one of my previous post, I already wrote about Online patching in EBS 12.2, and I have mentioned the new patching tool adop..
http://ermanarslan.blogspot.com.tr/2014/03/ebs-122-applications-dba-online.html
In that post, I didnt go in to the details about Online patching. On the other hand; as the days went by, I felt the need to go in the details about it, because online patching is the process that we often execute as apps dba's and that's why I think, we need to have a detailed knowledge about what's going on behind the scenes while we use adop to patch our EBS environments..
Okay, Lets start with it.
Online patching is a big advancement. Actually, it s revolutionary advancement addressed to be the solution for the patch downtime problem, which is a major concern for EBS customers.
It brings a new feature called patching cycle to provide its functionality. Maintanence mode is obsoleted now, as all patching activities are done online. Patching any module does not cause the entire system to be down, anymore.
We patch our environment online without affecting the running code and when we need to switch our patched environment with the running environment, we allocate our downtime ,which is measured in minutes and commit our changes..
So, still we will have a little downtime but it s just to little to cutover the things, it is measured in minutes, and it can be anytime we want.
We can apply our patches during the day without affecting the application and then cutover at the evening during our maintanence window.. That means, we can apply a big legislative patch during our payroll is running..
So how is it possible?
To provide online patching, Oracle uses a bunch of technologies.. Edition Based Redefinition is the biggest one of these.. Note that, Weblogic is also contributed..
Basically, patches are applied to the copy of the production system, actually copy of the production code. Thus, users are unaware about patching operations, as patches are applied to the copy.
Okay, when we say copy of the production, we only mean copy of the code, not the copy of the data..
We have codes on Filesystem and in the Database, so in online patching any code objects changed in a patch is copied.. I repeat, Application data is not copied on online patching operations..
As I said in the beginning, we switch our environments to commit the changes done during a patch application.. The Cutover phase is the switch.. It is actually the time to switch from production to newly patched copy, and that's where we have the downtime...(measured in minutes)
We have a downtime in cutover phase, because our middleware services are restarted from the patch environment . They are restarted to be able use the new code.. So you can consider this downtime as the time need to restart the middle tier. The downtime is felt like a logoff, like a disconnection;
I mean, when we cutover, the user are logged off and when they reconnect, they are directed to newly patched system.
Okay, lets explore patching architecture a little bit deeper;
In EBS 12.2 we have a secondary filesystem. It is actually a copy and it is kept syncronized by the patching tools.
On the other hand, in the database tier; we have a single database..
We dont have a copy of the database to support the online patching.. However ; copies are made in terms of code objects in the databsae.. That'is ; a seperate copy is maintained for all of the database code objects which are changed by a patch.. This feature supply the online patching in the database tier, and it is the actual revolution.
In online patching, when we start to apply a patch, patching tools and online users start to share the system resources..That's why patching can take longer time when you compare it with traditional patching. However; in online patching the system remains online .. That's why , patching time is not important..
Note that : In 12.1, we had an option named staged appl top. With this option, the system could be online while patching.. On the other hand, when it comes to database updates, the system should be offline again..
Note that : Staged Appl top was the basis for EBS 12.2's online patching..
In 12.2, we have actually 3 filesystems, not 2. We have fs1, fs2 and fs_ne
fs1 is prod, fs2 is patch filesystem(they change the roles after cutover)
fs_ne is the non editioned filesystem.. It contains logfiles, reports outputs and data import/export files.Fs_ne is the non editioned filesystem, and it is not syncronized during patching activities.. So, as all the data that is to be written intothe filesystem is written to fs_ne, we dont need to worry about copying fs data, as it is already in fs_ne.
So we have 3 filesystem and they connect to same database. Run edition filesystem (lets say fs1) only reads code and data definition.. Patch filesystem read and writes data into the both fs and db..
Okay, lets examine the db tier by starting from the Edition Based Redefition (EBR)..
EBR is an isolation mechanism that supplies the online upgrades of db tier. It allows an application to store its application definition in different editions. So we can have 2 copies for the same object code in a single database. Changes to the database objects are made in the isolation of an edition , and the editions define what you view.. Client code choses the edition.(run or patch)
In EBS 12.2 , we always have a run edition, which is used by production.
We have a patch edition which is only present while a patching cycle is active. Patching tools use it..
Also we can have one or more old editions. Old editions are defined after a patch edition becomes the run edition.. Old editions are removed when we execute cleanup.
Patching cycle consists of 5 processes.
Prepare->Apply->Finalize>""!!Cutover!!""->Cleanup
<------Abort----------------->
These are the phases of ADOP tool, actually.
Basically; prepare phase is the copy phase, apply phase is the phase where one or more patches are applied, finalize phase is where we finalize things , where we compile objects and etc..
Cutover phase is where we switch and cleanup phase is where we clean the mess.
Also we have abort phase, which is an option to abort a patch application.. We can abort one or patch application till we perform cutover..
Lets continue with the details about adop phases:
Prepare:
Syncronization of patch and run on the filesystem.( Oracle doesnt blindly copy everyting.. It is an incremental sync) Only files changed after the last patch are copied. Patch FS= Run FS --> they become synced.
A new patch edition in the db is created. Copies made for every code object. They are virtual copies though..The copied objects are actually pointers. They dont consume space.. Only, when an object is patched, it become real/actualized, and only then it consumes space.
For ex: When you change a view or package , it become real and occupy space inside the patch edition..
Note that: Objects like Tables and Indexes are not copied in online patching, as they are non-editioned objects!
Apply:
Adop runs adpatch to apply one more patches.
Patches are applied to the Patch edition. Normally, in a EBS patch, we have a fs patch driver and db patch driver in the backend.So, Fs patch driver uses patch filesystem(lets say fs2), and db driver uses patch edition created in prepare phase.
Finalize:
It is just a staging point. We compile invalid objects, generate derived objects, pre-compute ddl to be run at cutover and getting ready.
--We can pause at this stage as long as we want. We can wait for the appropriate downtime for the cutover in this stage.
Cutover: (downtime in minutes)
It restarts the application on the patched copy.
Oracle cutovers to the patch edition of the FS , and cutovers to the patch edition of the db.
Oracle(adop) restarts the middle tier services.
Users are logged off when the middle tier is stopped. Run FS becomes the Patch FS (vice versa) and Run Db edition becomes the Patch Db edition (vice versa)
In this stage, a final maintanence is performed, too.
At the end, the users are brought back online on the patched System..
Cleanup:
We do the cleanup the keep the System tablespace clean. In this phase, Adop deletes the unnecessary editioned objects on the db tier.
Once all objects are deleted from an old edition. The old edition itself is deleted, too.
Filesystems are resyncronized..(Oracle doesnt delete any files on FS actually )
Abort:
It is basically aborts the patching cycle. We can abort during and after prepare,apply or finalize phases. On the other hand; we cant abort once we cutover ..
Okay lets go a little deeper and mention about the EBR techniques that Oracle uses to maintain the new online patching feature;
Basically, we have editioned and non-editioned objects in our databases.
Editioned objects can be: PLSQL,Spec,Body,Procedure,View,Trigger,Editioning Views,Types and Synonyms..
Not-editioned objects are : Tables, Indexes, Materialized Views and Sequences .. These objects are non-editioned because they are storage objects..
(Remember : we only copy the application defitinion)
Altough, we have non-editioned objects in our databases , Oracle supplies this Online Patching... So how?
Adop uses Advanced features of EBR to operate on non-editioned objects.. So we use editioning views, cross edition triggers and editioned data storage in the backend..
Note that: Non-editioned objects cant reference editioned objects.. I mean; for example: a table that has a type object defined to one of its columns.. This kind of references can not be tolerated in 12.2. So we dont have such references in 12.2..
So in the case of tables, by using Editioning views, we can see the tables in different shapes acorrding to the editions.. So it is important to know that all the code which access EBS, should use this editioning views, because this views are the key for online patching.. So we dont copy the tables, but alter them in a different ways to not to interrupt the running applications.
Suppose a patch wants to make an alter to a table; for example patch wants to alter a column;
In online patching architecture, the patch doesnt alter the actual columns, it just adds a new column with desired attributes.. This is because we dont want our applications to be affected from this change... So the running application still uses the old format/run format of the table, that's why it is not getting affected.. Note that :Application can use the old format without any user intervention, because it uses an editioning view to reach that table.
What are those editioning views in APPS?
We have now synonyms defined in APPS for the tables . These synonyms are pointing to the Editioning views, and the editioning views are pointing to the actual tables.
So, for example:
WF_ITEMS is the synonym -> WF_ITEMS# is the editioning view -> WF_ITEMS is the actual table ..
That's why all the codes reaching the EBS should use those synonyms..
If we reach the table directly(physically), we may end up with a wrong structure for the table we want to operate on.. I mean, we may still see the old columns which are obsoleted by a patch.
Following describes what I tried to explain more clearly;
So suppose we want to alter a column named ID, and we want to change its column type from varchar2(10) to varchar2(50);
Oracle adds the change as a new column to supply the online patching.. So during the patch cycle, Patch edition views the table's columns as name,surname and ,id2, but Run edition still views the table's columns as name,surname and id.. This makes Run edition to be online, while applying a patch which changes the tables definition.
Also, It is worth to explain the corss edition triggers , as they are one of the major features that contributes the Online patching evolution in EBS. This triggers works somehow integrated with EBR..
Basically, during a patch application; if a row is inserted or updated in the run edition; these triggers are triggered. When they are triggered, they write down the data in the columns defined on the patch edition..
Consider the following scenario for the above table, what happens if a new ID is inserted to the ID column by the running application while we are patching ?
So by using cross edition triggers we save this change in the newly added ID2 column as well.
Furthemore, there is another feature that is used behind the scenes that makes online patching to be possible. It is Editioned Data Storage.. Editioned Data Storage allow an online patch to modify the seed data. The seed data means seeded data, which is the data for the Self Service UI for example..
So altough, we have said that only the application code is copied during an online patching operation, the seed data is an exception to that...So it s copied , too
That is; if we have a seed data in the table, Oracle copies the data in to the same table, and the patch operate on that copy.. This copy is retained till we cleanup.
To supply this function, Oracle is using VPD policies and similar technologies together..
In the meanwhile; when the application changes a seed data that is used by the run filesystem, the change is synced to the copy portion, as well.
For example: a profile option is changed during a patch that is operating on the relevant seed table, then this change is synced to the copy portion, which the patch is operating on.
Note that : Ofcourse, this is a one way syncronization.. So there is no syncronization from patch/copy portion to run portion.
Okay, till this point; I have tried to explain the online patching architecture in general and given some details about the underlying technologies which contribute to the online patching feature of EBS 12.2
Lastly, I will mention about some facts and benefits of the online patching..
All the features that I have mentioned in this post come with the installation of 12.2. Also, Oracle supplies these features even for the custom application sif we register them properly.
These features/online patching comes with the 12.2 upgrade, too..
As we have learned that we have a patch edition in Online Patching; It is important to remind that : The patch edition is not a testing environment! So you will still need to be make your tests on your test system.
As for the benefits; Online patching removes the bariers for the upgrade, because it is online.
It gives us the ability to make our downtime negotiations easier, because using online patching, we have downtime in minutes.
Also it is easier to identify the downtime if using Online patching, because the downtime means the restarting of Midtier in general.
My favorite benefit is that, It is online . Application is online ! :)
That's it for now. I hope you will find this document useful.
Please feel free to contact me, in case you have any questions or objections about the things that I have written in this post..
References: Oracle E-Business Suite Technology Webcasts and Training
Almost the entire middleware architecture is different, its installation capabilities are advanced, and patching on EBS 12.2 is a lot different. We use different tools for several administration activities and we have new tools for monitoring them.. Also we dont use a lot of things, which were mandatory in the previous releases, because they are obsoleted now.
Today, I will write an article about one of these new features in EBS, which is Online Patching. Online Patching is a big advancement in EBS, and it brings a completely new patching architecture to us. It is a bit tricky and complex, but it brings a lot of advantages..
In one of my previous post, I already wrote about Online patching in EBS 12.2, and I have mentioned the new patching tool adop..
http://ermanarslan.blogspot.com.tr/2014/03/ebs-122-applications-dba-online.html
In that post, I didnt go in to the details about Online patching. On the other hand; as the days went by, I felt the need to go in the details about it, because online patching is the process that we often execute as apps dba's and that's why I think, we need to have a detailed knowledge about what's going on behind the scenes while we use adop to patch our EBS environments..
Okay, Lets start with it.
Online patching is a big advancement. Actually, it s revolutionary advancement addressed to be the solution for the patch downtime problem, which is a major concern for EBS customers.
It brings a new feature called patching cycle to provide its functionality. Maintanence mode is obsoleted now, as all patching activities are done online. Patching any module does not cause the entire system to be down, anymore.
We patch our environment online without affecting the running code and when we need to switch our patched environment with the running environment, we allocate our downtime ,which is measured in minutes and commit our changes..
So, still we will have a little downtime but it s just to little to cutover the things, it is measured in minutes, and it can be anytime we want.
We can apply our patches during the day without affecting the application and then cutover at the evening during our maintanence window.. That means, we can apply a big legislative patch during our payroll is running..
So how is it possible?
To provide online patching, Oracle uses a bunch of technologies.. Edition Based Redefinition is the biggest one of these.. Note that, Weblogic is also contributed..
Basically, patches are applied to the copy of the production system, actually copy of the production code. Thus, users are unaware about patching operations, as patches are applied to the copy.
Okay, when we say copy of the production, we only mean copy of the code, not the copy of the data..
We have codes on Filesystem and in the Database, so in online patching any code objects changed in a patch is copied.. I repeat, Application data is not copied on online patching operations..
As I said in the beginning, we switch our environments to commit the changes done during a patch application.. The Cutover phase is the switch.. It is actually the time to switch from production to newly patched copy, and that's where we have the downtime...(measured in minutes)
We have a downtime in cutover phase, because our middleware services are restarted from the patch environment . They are restarted to be able use the new code.. So you can consider this downtime as the time need to restart the middle tier. The downtime is felt like a logoff, like a disconnection;
I mean, when we cutover, the user are logged off and when they reconnect, they are directed to newly patched system.
Okay, lets explore patching architecture a little bit deeper;
In EBS 12.2 we have a secondary filesystem. It is actually a copy and it is kept syncronized by the patching tools.
On the other hand, in the database tier; we have a single database..
We dont have a copy of the database to support the online patching.. However ; copies are made in terms of code objects in the databsae.. That'is ; a seperate copy is maintained for all of the database code objects which are changed by a patch.. This feature supply the online patching in the database tier, and it is the actual revolution.
In online patching, when we start to apply a patch, patching tools and online users start to share the system resources..That's why patching can take longer time when you compare it with traditional patching. However; in online patching the system remains online .. That's why , patching time is not important..
Note that : In 12.1, we had an option named staged appl top. With this option, the system could be online while patching.. On the other hand, when it comes to database updates, the system should be offline again..
Note that : Staged Appl top was the basis for EBS 12.2's online patching..
In 12.2, we have actually 3 filesystems, not 2. We have fs1, fs2 and fs_ne
fs1 is prod, fs2 is patch filesystem(they change the roles after cutover)
fs_ne is the non editioned filesystem.. It contains logfiles, reports outputs and data import/export files.Fs_ne is the non editioned filesystem, and it is not syncronized during patching activities.. So, as all the data that is to be written intothe filesystem is written to fs_ne, we dont need to worry about copying fs data, as it is already in fs_ne.
So we have 3 filesystem and they connect to same database. Run edition filesystem (lets say fs1) only reads code and data definition.. Patch filesystem read and writes data into the both fs and db..
Okay, lets examine the db tier by starting from the Edition Based Redefition (EBR)..
EBR is an isolation mechanism that supplies the online upgrades of db tier. It allows an application to store its application definition in different editions. So we can have 2 copies for the same object code in a single database. Changes to the database objects are made in the isolation of an edition , and the editions define what you view.. Client code choses the edition.(run or patch)
In EBS 12.2 , we always have a run edition, which is used by production.
We have a patch edition which is only present while a patching cycle is active. Patching tools use it..
Also we can have one or more old editions. Old editions are defined after a patch edition becomes the run edition.. Old editions are removed when we execute cleanup.
Patching cycle consists of 5 processes.
Prepare->Apply->Finalize>""!!Cutover!!""->Cleanup
<------Abort----------------->
These are the phases of ADOP tool, actually.
Basically; prepare phase is the copy phase, apply phase is the phase where one or more patches are applied, finalize phase is where we finalize things , where we compile objects and etc..
Cutover phase is where we switch and cleanup phase is where we clean the mess.
Also we have abort phase, which is an option to abort a patch application.. We can abort one or patch application till we perform cutover..
Lets continue with the details about adop phases:
Prepare:
Syncronization of patch and run on the filesystem.( Oracle doesnt blindly copy everyting.. It is an incremental sync) Only files changed after the last patch are copied. Patch FS= Run FS --> they become synced.
A new patch edition in the db is created. Copies made for every code object. They are virtual copies though..The copied objects are actually pointers. They dont consume space.. Only, when an object is patched, it become real/actualized, and only then it consumes space.
For ex: When you change a view or package , it become real and occupy space inside the patch edition..
Note that: Objects like Tables and Indexes are not copied in online patching, as they are non-editioned objects!
Apply:
Adop runs adpatch to apply one more patches.
Patches are applied to the Patch edition. Normally, in a EBS patch, we have a fs patch driver and db patch driver in the backend.So, Fs patch driver uses patch filesystem(lets say fs2), and db driver uses patch edition created in prepare phase.
Finalize:
It is just a staging point. We compile invalid objects, generate derived objects, pre-compute ddl to be run at cutover and getting ready.
--We can pause at this stage as long as we want. We can wait for the appropriate downtime for the cutover in this stage.
Cutover: (downtime in minutes)
It restarts the application on the patched copy.
Oracle cutovers to the patch edition of the FS , and cutovers to the patch edition of the db.
Oracle(adop) restarts the middle tier services.
Users are logged off when the middle tier is stopped. Run FS becomes the Patch FS (vice versa) and Run Db edition becomes the Patch Db edition (vice versa)
In this stage, a final maintanence is performed, too.
At the end, the users are brought back online on the patched System..
Cleanup:
We do the cleanup the keep the System tablespace clean. In this phase, Adop deletes the unnecessary editioned objects on the db tier.
Once all objects are deleted from an old edition. The old edition itself is deleted, too.
Filesystems are resyncronized..(Oracle doesnt delete any files on FS actually )
Abort:
It is basically aborts the patching cycle. We can abort during and after prepare,apply or finalize phases. On the other hand; we cant abort once we cutover ..
Okay lets go a little deeper and mention about the EBR techniques that Oracle uses to maintain the new online patching feature;
Basically, we have editioned and non-editioned objects in our databases.
Editioned objects can be: PLSQL,Spec,Body,Procedure,View,Trigger,Editioning Views,Types and Synonyms..
Not-editioned objects are : Tables, Indexes, Materialized Views and Sequences .. These objects are non-editioned because they are storage objects..
(Remember : we only copy the application defitinion)
Altough, we have non-editioned objects in our databases , Oracle supplies this Online Patching... So how?
Adop uses Advanced features of EBR to operate on non-editioned objects.. So we use editioning views, cross edition triggers and editioned data storage in the backend..
Note that: Non-editioned objects cant reference editioned objects.. I mean; for example: a table that has a type object defined to one of its columns.. This kind of references can not be tolerated in 12.2. So we dont have such references in 12.2..
So in the case of tables, by using Editioning views, we can see the tables in different shapes acorrding to the editions.. So it is important to know that all the code which access EBS, should use this editioning views, because this views are the key for online patching.. So we dont copy the tables, but alter them in a different ways to not to interrupt the running applications.
Suppose a patch wants to make an alter to a table; for example patch wants to alter a column;
In online patching architecture, the patch doesnt alter the actual columns, it just adds a new column with desired attributes.. This is because we dont want our applications to be affected from this change... So the running application still uses the old format/run format of the table, that's why it is not getting affected.. Note that :Application can use the old format without any user intervention, because it uses an editioning view to reach that table.
What are those editioning views in APPS?
We have now synonyms defined in APPS for the tables . These synonyms are pointing to the Editioning views, and the editioning views are pointing to the actual tables.
So, for example:
WF_ITEMS is the synonym -> WF_ITEMS# is the editioning view -> WF_ITEMS is the actual table ..
That's why all the codes reaching the EBS should use those synonyms..
If we reach the table directly(physically), we may end up with a wrong structure for the table we want to operate on.. I mean, we may still see the old columns which are obsoleted by a patch.
Following describes what I tried to explain more clearly;
So suppose we want to alter a column named ID, and we want to change its column type from varchar2(10) to varchar2(50);
Oracle adds the change as a new column to supply the online patching.. So during the patch cycle, Patch edition views the table's columns as name,surname and ,id2, but Run edition still views the table's columns as name,surname and id.. This makes Run edition to be online, while applying a patch which changes the tables definition.
Also, It is worth to explain the corss edition triggers , as they are one of the major features that contributes the Online patching evolution in EBS. This triggers works somehow integrated with EBR..
Basically, during a patch application; if a row is inserted or updated in the run edition; these triggers are triggered. When they are triggered, they write down the data in the columns defined on the patch edition..
Consider the following scenario for the above table, what happens if a new ID is inserted to the ID column by the running application while we are patching ?
So by using cross edition triggers we save this change in the newly added ID2 column as well.
Furthemore, there is another feature that is used behind the scenes that makes online patching to be possible. It is Editioned Data Storage.. Editioned Data Storage allow an online patch to modify the seed data. The seed data means seeded data, which is the data for the Self Service UI for example..
So altough, we have said that only the application code is copied during an online patching operation, the seed data is an exception to that...So it s copied , too
That is; if we have a seed data in the table, Oracle copies the data in to the same table, and the patch operate on that copy.. This copy is retained till we cleanup.
To supply this function, Oracle is using VPD policies and similar technologies together..
In the meanwhile; when the application changes a seed data that is used by the run filesystem, the change is synced to the copy portion, as well.
For example: a profile option is changed during a patch that is operating on the relevant seed table, then this change is synced to the copy portion, which the patch is operating on.
Note that : Ofcourse, this is a one way syncronization.. So there is no syncronization from patch/copy portion to run portion.
Okay, till this point; I have tried to explain the online patching architecture in general and given some details about the underlying technologies which contribute to the online patching feature of EBS 12.2
Lastly, I will mention about some facts and benefits of the online patching..
All the features that I have mentioned in this post come with the installation of 12.2. Also, Oracle supplies these features even for the custom application sif we register them properly.
These features/online patching comes with the 12.2 upgrade, too..
As we have learned that we have a patch edition in Online Patching; It is important to remind that : The patch edition is not a testing environment! So you will still need to be make your tests on your test system.
As for the benefits; Online patching removes the bariers for the upgrade, because it is online.
It gives us the ability to make our downtime negotiations easier, because using online patching, we have downtime in minutes.
Also it is easier to identify the downtime if using Online patching, because the downtime means the restarting of Midtier in general.
My favorite benefit is that, It is online . Application is online ! :)
That's it for now. I hope you will find this document useful.
Please feel free to contact me, in case you have any questions or objections about the things that I have written in this post..
No comments :
Post a Comment
If you will ask a question, please don't comment here..
For your questions, please create an issue into my forum.
Forum Link: http://ermanarslan.blogspot.com.tr/p/forum.html
Register and create an issue in the related category.
I will support you from there.