Weblogic -- Single Weblogic Server & Multiple Domains

Here is the tip of the day and again I will keep it short, so directly jumping into the topic;
We don't need to install a seperate Weblogic Server in each time we want to create a new domain.
Suppose, we need to install OID + OAM.
We first a Weblogic Server, and then install OID on it. OID will have its own weblogic domain on top of this Weblogic Server, as it is a part of the installation.
So , when we come to OAM installation, we can use the same Weblogic Server to place our OAM software and create a specific domain for OAM as well.
At the end of the day, we install OID and OAM in to the same Weblogic Server but in to different Weblogic domains.

Well, Multiple domains can be created on single Weblogic Server. That's right. We tested it. We actually installed OID + OAM in to the same Weblogic Server and I can say that it is stable and it is not so hard to administrate.
However, there are 2 things that I will warn you if you want to do a similar configuration;

1) Ensure there is no conflict between Admin server or Managed Server ports, when we you have 1 WL Server running on one machine with multiple domains.
Altough Weblogic can listen on all avaiable interfaces(That is, while , Admin Server on domain 1 can listen from ip_address:7001, Admin Server on domain 2 can listen on localhost:7001) , having the same ports across domains will cause problems. Nodemanger will not work properly. There should be no admin server, or managed server sharing the same port accross domains.
2) Change the cookie name of the domains to be different from each other.
For example : FOR OID domain, the cookie name can be WLSOIDCOOKIE, and for OAM domain , the cookie name can be WLSOAMCOOKIE. Cookie change requires all the WLS server components in a domain to be restarted. If you don't do this change and restart, then you will not be able to administrate multiple domains in a single Browser, that is the cookie will conflict.

So, that 's all I need to mention about this topic, I hope it will help you.

Exadata- X6-2 1/8, its specs and its readiness for EBS migrations

We have done several EBS migrations and most of the time we do these migration using datapump method. The reason we choose this method was always a need for an upgrade. That is , ofcourse standby switchover/failover based migration is faster , and smoother then a datapump based one, but once you need to upgrade the database tier in the source instance, the things become complicated and you most probaby face with more downtime in total.

As for the new generation Exadata X6-2 , the migration of EBS is quite similar. We just create our database in the target Exadata patch EBS and EXA(if necessary) and then use datapump to exp-imp EBS database tier to Exadata. After the migration of the database, we do couple of configurations using post clone and autoconfig sometimes and that's it.
This general procedure still applies to Exadata with its new generation hardware and software.

In this post, I will give some quick overview of Exadata X6, a eight rack version of it and then give some instructions for the readiness of the Exadata environment to be used as an EBS database tier.

I will give the explanation with a Question an Answer list and try to keep it simple.

What do we have in Exadata X6-2?

Exadata X6-2 Eight(1/8) Rack:
--------------------------------------------------------------------------------------------------------
-2 database servers:
Each server has:
*2 x 22-core processors, each with 11 cores enabled (Xeon E5-2699 v4 processors)
*256 default memory minimum(default) -->  256 GB (8 x 32 GB) RAM expandable to 512 GB (16 x 32 GB) or 768 GB (24 x 32 GB) with memory expansion kit
*4 x 600 GB 10K RPM SAS disks, hot swappable, expandable to 8x
*Disk controller HBA with 1 GB cache (no more batteries)
*2 x InfiniBand 4X QDR (40 Gb/s) ports (PCIe 3.0), both ports active
*4 x 1 GbE/10GbE Base-T Ethernet ports
*2 x 10 GbE Ethernet SFP+ ports (1 dual-port 10GbE PCIe 2.0 network card based on the Intel 82599 10 GbE controller technology)
*1 Ethernet port for Integrated Lights Out Manager (ILOM) for remote management
*Oracle Linux 6 Update 7 with Unbreakable Enterprise Kernel 2 or Oracle VM Server 3.2.9

-3 Exadata Storage Server X6-2 Servers:
Each server has:
*2 x 10-core processors(10-core Xeon E5-2630 v4 processors), each with 5 cores enabled
*128GB memory
*2 PCI flash cards (each 3.2 TB)+ 6x8 7200 rpm disk OR 8 PCI flash Drives(4 enabled, each 3.2 TB)

-2 Sun Datacenter InfiniBand Switch
-2 redundant PDUs (single phase or three phase, high voltage or low voltage)
-1 48-port Cisco Catalyst 4948E-F, model number WS-C4948E-F-S Ethernet switch
--
Raw PCI flash capacity: 38.4 TB for EF, or 19.2 TB for HC --> for EF(Extreme Flash) Calculation: 3.2 X 4 x 3 = 36 TB.
Raw hard disk capacity: 144 TB for high capacity disks --> for HC Calculation : 6 x 8 x 3 = 144TB.

Why is it called Extreme Flash (I mean Exadata Extreme Flash) ?

If we look at the spare flash drives of Exadata, we see the "SSD" clause on them.
This is because; actually SSD drives are Flash-based.

There is no difference between SSD and Flash. So altough it is called Extreme Flash, Exadata EF uses SSD disks, which actually use flash in the background.

How do we(Actually Oracle Field engineer) deploy Exadata for the first time, after all the cabling and other hardware related stuff is done?

The deployment of Exadata X6 is done using the script called install.sh.
With the latest version, following software components are deployed by the installer:

Database Software Version: 12.1.0.2
Grid Software Version: 12.1.0.2
Os version: OEL 6.7

Kernel: 2.6.38-400.277 UEK

If we want to use NFS exports from Exadata to store our EBS database exp dump files, what do we do?

NFS configuration from Exadata to the source system can be done by following:

Exadata: How to Create a NFS Mount on a Database Node (Doc ID 1900335.1)

Do we use Oracle Home that is deployed with Exadata to provision our EBS database?

Actually, no. We leave that home as is, as an example for us.

We create/clone a new home on Exadata and use it for our EBS database tier.

How do we clone our database home in Exadata? (as it is RAC and as it is an appliance, it may be question?)

A new Oracle Home is created using cloning technique, for this the following actions are taken.
In each node
-> Create a new  oracle home by copying the existing one, use cp -rp to preverse the permissions.
-> Register Oracle home to inventory
    for example:
    cd $NEW_ORACLE_HOME
    cd clone
    cd bin
    perl ./clone.pl ORACLE_BASE=$ORACLE_BASE \
    ORACLE_HOME=<NEW_ORACLE_HOME_PATH> \
    ORACLE_HOME_NAME=<NEW_ORACLE_HOME_NAME> \
    '-O"CLUSTER_NODES={exa01,exa02}"'\
    '-O"LOCAL_NODE=exa01"'      --> we change it to second node when we are executing it in the second node.

Note that, this script should be run on all the nodes(by modifying the LOCAL_NODE appropriately) one by one after copying the oracle homes and lastly root.sh should be executed after these runs, as shown in the following example;

Here is a demo run for you:

[oracle@exadb01 bin]$ perl clone.pl ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_prod ORACLE_HOME_NAME=OraDB12HomeProd '-O"CLUSTER_NODES={exadb01,exadb02}"' '-O"LOCAL_NODE=exadb01"'
./runInstaller -clone -waitForCompletion  "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_prod" "ORACLE_HOME_NAME=OraDB12HomeProd" "CLUSTER_NODES={exadb01,exadb02}" "LOCAL_NODE=exadb01" -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_prod/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 16660 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 23123 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-09-07_11-12-23AM. Please wait ...You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2016-09-07_11-12-23AM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12HomeProd was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2016-09-07_11-12-23AM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0.2/dbhome_prod/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_prod/root.sh on the following nodes:
[exadb01]

..................................................   100% Done.

[root@exadb01 ~]# sh /u01/app/oracle/product/12.1.0.2/dbhome_prod/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_prod/install/root_erman_2016-09-07_11-13-30.log for the output of root script

What is next?

The next thing is to the migration :) by following Oracle Support documents (export import process of EBS, EBS - 12 C RDBMS interoperability and EBS on Exadata whitepapers) and ofcourse reading my blog posts which are about real life example of EBS - Exadata migrations:)

So that 's it for now. I will write a seperate blog post for migration EBS on Exadata X6 (altough it will not be so different than migrating to earlier releases of Exadata ;)

RMAN-- Commvault restore, controlfile autobackup and a little more especially about the controlfile

In this post, I will give a quick overview of Commvault-Rman based database cloning..
This post is based on a customer site story, where we wanted to implement Commvault based Rman restores and even if it was not our duty or responsibility at all, we still needed to get our hands dirty with this 3rd party tool . (3rd party from Oracle Site)
Basically, Commvault uses rman to do online backups and also the restores of this backups.
Restoring a backup a to different server, can be considered as cloning and here in this post, we will talk about that.
Ofcourse a Commvault admin is also required to administrate the Commvault side and to initiate backups and restore from there, but  what needed in the database side are as follows;

• init.ora must be in place.
• commvault seees the database from oratab , so the database entry must be there in oratab.
• Database should be opened in nomount mode for the commvault discovery.
• If controlfile is not supposed to restored by Commvault , then Database should be opened in mount mode before the commvault restore job.
• To be able restore rman wants an autobackup controlfile why?

With a control file autobackup, RMAN can recover the database even if the current control file, recovery catalog, and server parameter file are inaccessible. Because the path used to store the autobackup follows a well-known format, RMAN can search for and restore the server parameter file from that autobackup. Also -> If you have configured control file autobackups, then the server parameter file is backed up with the control file whenever an autobackup is taken. 

However, Rman does not want the autobackup of controlfile, but we make rman to want it. We tell rman to use the autobackup for controlfile restores with the command "restore controlfile from autobackup", and when we give that command, rman just tries to do the spfile and controlfile restore using the autobackups.

Commvault for instance; does use the "restore controlfile from autobackup" command , and because of that rman wants autobackups in place to be able to restore the controlfie.

Well, alternatively, we can use a a traditional controlfile backup(non autobackup) by specifying its backup piece (restore controlfile from '/BACKUP/09I594QI_1_1';) as well. Check: How To Restore Controlfile From A Backupset Without A Catalog Or Autobackup (Doc ID 403883.1)

When we are talking about the backups of controlfile, let's give some basic but important  information about the controlfiles.

Controlfiles contain information about the database, redolog files and other files that are required open an Oracle database.
All the structural database changes are stored in the controlfile (adding a datafile, renaming a datafile, dropping a database , or redolog files etc)
Without controlfile, we can't open the database, and that's why in the first place, we need to have a controlfile to provision our clone database environments.
The following information is stored in a Controlfile.

The database name 
The timestamp of database creation
The names and locations of associated datafiles and redo log files
Tablespace information
Datafile offline ranges
The log history
Archived log information
Backup set and backup piece information
Backup datafile and redo log information
Datafile copy information
The current log sequence number
Checkpoint information

Some of the information types in the controlfile may be overwritten when there is no space left in their region. 

These types of information are listed  below; 

Note that, the parameter named CONTROL_FILE_RECORD_KEEP_TIME plays a big role in this overwriting, as it specifies the minimum number of days before a reusable record in the control file can be reused. In the event a new record needs to be added to a reusable section and the oldest record has not aged enough, the record section expands. If this parameter is set to 0, then reusable sections never expand, and records are reused as needed.

Note that, the size of the database control file growth may depend on the number of: Backups taken and stored in the controlfile, the count of archived redo logs that our database generates, the number of days that the information is configured to be stored in the control file .

LOG HISTORY

OFFLINE RANGE

ARCHIVED LOG

BACKUP SET

BACKUP PIECE

BACKUP DATAFILE

BACKUP REDOLOG

DATAFILE COPY

BACKUP CORRUPTION

COPY CORRUPTION

DELETED OBJECT

PROXY COP

The other information types which are not listed above, such as the database name or the data file names can not be overwritten, so when there is a shortage on their region, the controlfile basically expands for storing the new records (such as when adding a new datafile and when there is no space left in the related area of the controlfile)

When there is no controlfile, or there is no autobackup or normal backup of controlfile , the controlfile can be created using CREATE CONTROLFILE command. However, this command will only a create the controlfile and will not load it with any information like archivelogs, deleted object, log history and so on. Similarly, using the "alter database backup controlfile to trace" command, we can create a controlfile creation statement for ourselves. This might be used in cloning. That is, we can restore our database without our controlfile and issue the command backup controlfile to trace in our source environment and then execute the created controlfile creation script in our target environment to have a brand new controlfile for it, but the same type of information will not be captured using this controlfile creation method. 

The I/O(read write) that is done from or to the controlfile, is done through the PGA.

Processes(foreground and background) write to the controlfile and when they write , they wait in Controlfile file parallel write wait.

The reads or done using the control file sequential read event, and the wait occurs here is the control file sequential read wait.

Control file is read in incidents such as when making a backup controlfiles, sharing info from the controlfile between instances (RAC), so in other words; the controlfile sequential read will be there for the admin activities.

These IOs are different than the DB block I/Os and they are done with the physical  block size of the underlying OS (For Linux it is 512 bytes)

Controlfiles are also written in single writes sometimes.. In those times, the wait event Control file single write wait occurs. This wait is signaled while the control file's shared information is written to disk.

These operations are under the control of Controlfile Enqueue (CF)

So when we see a high CF in our database, then there is a problem with the Controlfile transactions. The IO subsystem may be operating very slowly, or we may produce a high activity that will cause the controlfile is continously and aggressively updated.(such as producing lots lots lots of archivelogs)

Here are some recorded cause for that:

Very slow I/O subsystem where the Control files are stored. 

Frequent log switching - redo logs too small or insufficient logs 

Async I/O issue or multiple db_writers, you can't use both of them, back out one.

OS / Hardware issues

Why did I get into these details? I don't know:) maybe I wanted to give some solutions for the cloning problems that may arise because of lacking the controlfile backups..

Anyways, let's get back to our main topic:

• db_create_file_dest should be set to the new path, that becausee: "ORA-01119 error in creating file +data" (415579.1 )

How commvault does the restore, an example:

-->
It connects with rman target / to the target environment, and then restore the controlfile(from autobackup) , database and lastly recovers the database and opens it in resetlogs mode.

run {

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

restore controlfile ;

sql 'alter database mount';

}

exit;

RESTORE & RECOVER the DATABASE: 

run {

set newname for database to '/u02/ERMAN/Data/%U';

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

allocate channel ch2 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)"

TRACE 0;

allocate channel ch3 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch3)"

TRACE 0;

allocate channel ch4 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch4)"

TRACE 0;

allocate channel ch5 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch5)"

TRACE 0;

## send " -jm 14 -a 2:0 -cl 2320 -ins 234 -at 0 -j 309830 -bal 0 -rcp 0 -ms 5 -p 2 -PREVIEW";

restore database  until time = "TO_DATE('06/22/2016 01:25:42','MM/DD/YYYY HH24:MI:SS')" ;

switch datafile all;

}

exit;

run {

allocate channel ch1 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)"

TRACE 0;

allocate channel ch2 type 'sbt_tape'

PARMS="SBT_LIBRARY=/opt/simpana/Base/libobk.so,BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)"

...

.....

.....

recover database  until time = "TO_DATE('06/22/2016 01:25:42','MM/DD/YYYY HH24:MI:SS')" ;

alter database open resetlogs;

}

exit;

Linux -- dpkgd , ssh attack

Recently dealed with a SSH attack and want to share the experience with you.
The problem was reported to me by one of my customers, as they have seen a high traffic from one of their EBS R12 servers to their firewall , causing CPU saturation on the Firewall machine itself.
One of the interesting thing was, this EBS environment was fully open to the world, so all of its ports including ssh was publically available to the world :) Besides, the root password was a very simple one. In other words; the situation was an expected one :)
Anyways; when I logged into the EBS server, I directly saw that it was hacked, as the standard commands such as ps , lsof and netstat was not working, and also there were weird processes and files scattered on the system.
It seemed like the server is hacked from SSH using a brute force. The attacker should have been crack the root password then logged into system using the root account and  deployed his/or her malwares to the filesystem(probably using scp). We actually approved this theory by seeing the nmap port scans and the ssh login attempts using the network monitor tools.
The name of the malware was dpkgd and it replaced the ss,ps,netstat and lsof binaries with his own copies , cleared the syslogd and directed the auth.log, audit.log and utx.laslogin to /dev/null) So it tried to leave no traces behind.
Actually, altough I didn't check this on my own, the malwares were used to create an RCP tunnel between the attacker and the system to make the system be under control and obey the attackers commands. Thus, the malwares could be used to do DOS attacks against the various servers which are available on internet. The attacker also reset the root password couple of times.

Not: the same attack and the affected files were also recorded in the blog post below; "https://blog.smarthoneypot.com/in-depth-analysis-of-ssh-attacks-on-amazon-ec2".

The things that I ve did to clear and secure the system were,

• changed the root password
• found and deleted all the malware files
• cleared the malicious crontab entries
• deleted the log files which were directed to /dev/null and recreated them 
• Recreated the affected standard programs like lsof,ps,ss and netstat by copying them from another unaffected system (from another machine, which have the same OS and arch (32 bit/64 bit) with the affected node)
• reinstalled the rsyslog
• rebooted the server and saw that eveything was come back to normal
• Requested the Network team to close all the port in the firewall, except the Oracle's HTTP listener active port, which was ssl. (also disabled the non-ssl port for the http server by implementing: "Enabling TLS in Oracle E-Business Suite Release 12.1 (Doc ID 376700.1)" - Section 6.2 Disabling the HTTP Port)

What should be do for the future? (In case, the SSH port will need to be opened )

• More secure passwords should be used.
• SSH should be configured to be serving from a non-default port.
• SSH should only accept specific Ip addresses which are included in a whitelist.
• User pass authentication, a more secure authentication mechanism should be used.

Recommend you to read this blog post, as well:

"https://blog.smarthoneypot.com/in-depth-analysis-of-ssh-attacks-on-amazon-ec2"

RDBMS/Linux -- Installing Oracle 11gR2 on CentOS 6

Altough we say CentOS is not certified with Oracle at all, sometimes customers insist on using CentOS as for the Operating System of their Oracle Database Server.

For Oracle RDBMS, here is the supported list of Linux Distributions:  Oracle RDBMS ile certified olan OS lerden; SLES, Asianux, Redhat and Oracle Linux .
Note that; CentOS, on the other hand can be considered as Redhat  (from Oracle perspective)

Inspite of this lack of certification, I m here to say that, Oracle Database can be installed and used with CentOS. We tested and saw it .

Here is the installation work that we did for installing an Oracle11gR2 on Centos 6 64 bit Operating System.

1. Download repository
cd /etc/yum.repos.d
wget https://public-yum.oracle.com/public-yum-ol6.repo

2. Install Pre-Install Database rpm
yum install --nogpgcheck oracle-rdbms-server-11gR2-preinstall

3. Configure the files below
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=ermanserver.ermandomain.com
GATEWAY=89.106.24.88
# oracle-rdbms-server-11gR2-preinstall : Add NOZEROCONF=yes
NOZEROCONF=yes

/etc/hosts
127.0.0.1       localhost localhost.localdomain
89.106.24.139   ermanserver.ermandomain.com ermanserver

/etc/selinux/config
SELINUX=disabled
SELINUXTYPE=targeted

4. Create Oracle Home folders
mkdir -p /erm/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /erm/
chmod -R 775 /erm/

5. Download Oracle Software
http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_database_1of2.zip
http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_database_2of2.zip

6. Unzip Software
unzip linux.x64_11gR2_database_1of2.zip
unzip linux.x64_11gR2_database_2of2.zip
7. Set environments in the .bash_profile

[oracle@ermanserver ~]$ cat .bash_profile
PATH=$PATH:$HOME/bin
export PATH
export ORACLE_BASE=/erm/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=ERMAN
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export CLASSPATH=$ORACLE_HOME/jlib

8. Install Software
cd database
./runInstaller

9.    Follow the screens

10.    Run netca to setup the listener

     11.       Listener is created so It should be started automatically. (you can start it with “ lsnrctl start LISTENER_NAME”)

     

     12.       Create database by following the screens shown below; (this is a standard thing..)

      Actually all the things are standard , we are installing Oracle Database on CentOS, like we do it on an Oracle Linux..

      Well, that is it, no ORA-600 or 7445. Our database is ready to be used.

      At the end of this post, I want to mention one thing; that is giving this kind of standard installation procedure is actually not my style, but in this post, I did this to show that Oracle Database (at least 11G) can be installed to CentOS Operating System without problems.

      Hope this will help.

Linux/Scripting - drawing dialog boxes in shell/tui/terminal

This post is a cookie . I m writing this in my limited free time : ) but it is interesting , as I will explain how to draw dialog boxes in standard linux shells. (note that I m not talking about X windows)

Let's start;

In order to draw a dialog box, we install dialog utility using yum. (my demo enviroment is Oracle Linux)

[root@demoorcl ~]#yum install dialog
...
.....
........
Installed:
  dialog.x86_64 0:1.1-9.20080819.1.el6                            

Then we look at the definition: via "man dialog" command.

manual says " dialog - display dialog boxes from shell scripts" :)

Next, we write a shell script to use the features of this dialog utility. 

A shell script like the one below;

#!/bin/sh

DIALOG=${DIALOG=dialog}

tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$

$DIALOG  --ok-label "Do the selected!" --cancel-label Exit --clear --title "Admin Interface" \

        --menu "Choose the action to be taken:" 20 70 15 \

        "1"  "Check the integration" \

        "2"  "Shut down the integration Services" \

        "3"  "Startup the integration Services" \

        "4"  "Check the other services" \

        "5"  "Shut down the other services" \

        "6"  "Check the processes"  2> $tempfile

retval=$?

Next, we execute the script, and here our dialog is displayed... ->

Well, that is it.. We display a standard dialog. 

In this simple dialog, when we hit exit button, the dialog is closed, but naturally;  there is no event handler for "the selections"/"Do the selected" buttons, as our script did not include those events.

But if we want to go further and put all the event handlers, we can enhance our script and make it production ready .

An example script in this context is like the following;

Note that: This script is used in a production system, so I changed some paths and commented out some action lines. However,  when you analyze it you will get idea and use the info for your own benefit.

#!/bin/sh

script=/root

DIALOG=${DIALOG=dialog}

tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$

$DIALOG  --ok-label "Do the selected!" --cancel-label Exit --clear --title "Admin Interface" \

        --menu "Choose the action to be taken:" 20 70 15 \

        "1"  "Check the integration" \

        "2"  "Shut down the integration Services" \

        "3"  "Startup the integration Services" \

        "4"  "Check the other services" \

        "5"  "Shut down the other services" \

        "6"  "Check the processes"  2> $tempfile

retval=$?

choice=`cat $tempfile`

case $retval in

  0)

    if [ $choice == "1" ]

    then

    ps -aux | grep eventprocessor | grep -v grep | grep -v Warning > /tmp/kontrol1

    cat /tmp/kontrol1 | grep eventprocessor

    if [ $? = "0" ]

     then

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KONTROL  " --textbox  $script/arayuz_durum1.txt 22 70

     sh $script/main

    else

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KONTROL  " --textbox  $script/arayuz_durum2.txt 22 70

    sh $script/main

     fi

    elif [ $choice == "4" ]

    then

    ps -aux | grep tomcat | grep -v grep | grep -v Warning > /tmp/kontrol2

    cat /tmp/kontrol2 | grep tomcat

     if [ $? = "0" ]

     then

     dialog --exit-label "Back to Main Menu" --title "  ARAYUZ KONTROL  " --textbox  $script/tomcat_durum1.txt 22 70

     sh $script/main

    else

     dialog --exit-label "Back to Main Menu" --title "  ARAYUZ KONTROL  " --textbox  $script/tomcat_durum2.txt 22 70

    sh $script/main

     fi

    elif [ $choice == "2" ]

    then

    ps -aux | grep eventprocessor | grep -v grep | grep -v Warning| awk '{print $2}'  > /tmp/arayuz_pid

     arayuz_pid=`cat /tmp/arayuz_pid|grep -v Warning`

     sudo kill $arayuz_pid

    dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KAPATILDI  " --textbox  $script/arayuz_kapatildi.txt 22 70

     sh $script/main

     elif [ $choice == "3" ]

    then

    sudo nohup sh /erm/eventprocessor/bin/run &

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON ACILDI  " --textbox  $script/arayuz_acildi.txt 22 70

     sh $script/main

     elif [ $choice == "5" ]

    then

   #sudo sh /erm/tomcat/apache-tomcat-5.5.17/bin/shutdown.sh &>/dev/null

 sh /erm/tomcat/apache-tomcat-5.5.17/bin/shutdown.sh &>/dev/null

dialog --exit-label "Back to Main Menu" --title "  ARAYUZ  KAPATILDI  " --textbox  $script/tomcat_kapatildi.txt 22 70

    sh $script/main

   elif [ $choice == "6" ]

    then

sudo nohup sh /erm/tomcat/apache-tomcat-5.5.17/bin/startup.sh

#sudo sh  /erm/tomcat/apache-tomcat-5.5.17/bin/startup.sh &>/dev/null

   # sudo sh /home/helpdesk/arayuz_script/tomcat_start.sh

dialog --exit-label "Back to Main Menu" --title "  ARAYUZ  ACILDI  " --textbox  $script/tomcat_acildi.txt 22 70

sh $script/main

     else

     echo ...

    fi

   ;;

   1)

    echo "Program Exited"

;;

  255)

    echo "ESC pressed."

;;

esac

Okay, here we go..

Let's execute this script (named Main)

sh Main

We press enter(equivalent to "Do the selected") while our selection line is on "Check the integration";

Well, the action is taken in the backround to check the integration control, and the status of the integration is reported in the dialog :) .. Actually, I modified the script to not to show what is done there, as it is a production code :)

Okay. Lets press enter while our selection line is on "Back to Main Menu"

Here we are in Main Menu again .. 

So that is it. You can go further and create your own dialogs, you can  even add control mechanisms and enhance your dialogs accordingly. By the help of this admin dialogs, you can ease the job of Linux and Oracle admins and even offload some works to the Operators.

That's enough for today.

See you in next blog posts :)

RDBMS -- how to rollforward a database using incremental backups after a failed duplicate/recovery session

In this post, I will give a quick tip about the incremental backup based recovery.
I will explain the concept by giving an example "rman duplicate" scenario.
Actually, this type of incremental backup based recovery is automatically done  by the rman duplicate command.  However, I will show you a failed automatic recovery that is supposed to be done by a rman duplicate and then, I will give you the command to roll forward the database using the incremental backups.

Suppose you started your rman duplicate work;

Rman output:

LEVEL0_DB

channel ORA_AUX_DISK_5: restore complete, elapsed time: 02:10:28

channel ORA_AUX_DISK_4: restored backup piece 12

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42248_12.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_4: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42248_13.bck

channel ORA_AUX_DISK_7: restored backup piece 13

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42245_13.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_7: restore complete, elapsed time: 02:12:08

channel ORA_AUX_DISK_8: restored backup piece 14

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_14.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_15.bck

channel ORA_AUX_DISK_4: restored backup piece 13

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42248_13.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_4: restore complete, elapsed time: 02:13:06

channel ORA_AUX_DISK_8: restored backup piece 15

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_15.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_16.bck

channel ORA_AUX_DISK_8: restored backup piece 16

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_16.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_17.bck

channel ORA_AUX_DISK_8: restored backup piece 17

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_17.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_18.bck

channel ORA_AUX_DISK_8: restored backup piece 18

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_18.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: restore complete, elapsed time: 02:39:26

Finished restore at 15-JUL-16

..................

.........................

..............................

Now suppose: after restoring the level0 backups, "rman duplicate" started to restore the incremental backups and it encountered an error , so that your duplicate db could be successfully created.

Rman output:

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_1/PROD_LEVEL_1_

42328_1.bck

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 07/15/2016 01:14:54

RMAN-03015: error occurred in stored script Memory Script

ORA-19870: error reading backup piece /BACKUP1/LEVEL_1/PROD_LEVEL_1_42327_1.bck

ORA-19505: failed to identify file "/BACKUP1/LEVEL_1/PROD_LEVEL_1_42327_1.bck"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 2

The error in this example is so obvious. That is, it is a Linux permission denied error and what rman clearly says with this error is "I can't read the backup files(incremental in this case) , because I don't have the necessary filesystem permission"


At this point, if we summarize the situation we can say that:
we restored our database (level 0 backup), but it is not up-to-date.
So, we have 2 choices to roll forward the database to current time
1) archivelog apply
2) incremental backup + archivelog

In fact, if we have incremental backups for the level0 we have restored, it is more reasonable to recover our database using those incremental backup.

This is because; if the level0 was taken a long time ago and if, we choose applying all the archivelogs generated since the level0 was taken, we ll probably have to apply lots of archivelog..

So, in this case, the things that we need do for makin rman use the incremental backups for recovering our newly restored database, are as follows;

1) We fix the permission issues for the backup files (chmod or chown)
2) We connect our mountented duplicate database using rman target /
3) We register the incremental backups using catalog start with (catalog start with 'BACKUP1/LEVEL_1')
4) We execute "recover database" command.!! So here is the surprise :))) we just simple execute recover database command. So, in conclusion, Rman already knows that, it is better to recover a database using incremental backups if they are available :)
5)We apply the archivelogs, which could be created since the time that we have taken our level1 backup, to make our database consistent and up-to-date. (Remember all the backups -- "including rman backups", which are taken while the databases are open, are inconsistent)