Thursday, May 21, 2015

EBS/RDBMS -- Preventing Discoverer Logons Using a After Logon Trigger

Here is a after logon trigger that can be used to prevent the Discoverer clients logins.
This might come in handy in a saturated environment where heavy Discoverer reports consume all the CPU and IO resources.. Instead of killing the Discoverer sessions or telling the clients to not to use their Discoverer reports during the peak hours, we can use this trigger to prevent the Discoverer client logins .. We can even let some domain users to use Discoverer while preventing the others..
We were using it in the year 2009, it was saving us those days.. Then we invented the limiter..(http://ermanarslan.blogspot.com.tr/p/self-developed-software.html#limiter) Later , we have used this trigger in conjuction with the storage replication.. We were redirecting some discoverer users to the replicated environment... Nowadays, we dont need this kind of solutions as the speed of the systems has increased . On the other hand; things like Resource Manager still works for the moderate hardware.. In Exadata  we use IORM(although we did not need)..
Everyting aside, Sql tuning is always the best solution for this..
Anyways, just came to my mind to share it :)

/*Here we get the Os username and Program name from SYS CONTEXT and checking the  allowed_users table, which has only one column (osuser) using the osuer we get from the sys_context.. If there is a row there, we let the user in.. If there is no rows, we reject the Discoverer client coming to the database from that Domain user... If we need to give an access right to a new user, we just add a row with the value of the relevant Osuser name , to the "allowed_users" table..*/

CREATE TABLE ALLOWED_USERS
(
OSUSER VARCHARR2(500)
);

/*Here we get the Os username and Program name from SYS CONTEXT and checking the  allowed_users table, which has only one column (osuser) using the osuer we get from the sys_context.. If there is a row there, we let the user in.. If there is no rows, we reject the Discoverer client coming to the database from that Domain user... If we need to give an access right to a new user, we just add a row with the value of the relevant Osuser name , to the "allowed_users" table..*/
/*Just using the SYSTEM user create the allowed_users table and then create this trigger */
/*You can disable this trigger if you want to let all the users use Discoverer Client to connect to the database*/
/*Discover client version used by the clients in this example has a program name as dis51.exe, you can change it according to your environment*/

CREATE OR REPLACE TRIGGER DISCO_PREVENT_TRG
   AFTER LOGON
   ON DATABASE
DECLARE
   v_osuser    v$session.osuser%TYPE;
   v_program   v$session.program%TYPE;
   v_count     NUMBER;

   CURSOR user_prog
   IS
      SELECT osuser, program
        FROM v$session
       WHERE audsid = SYS_CONTEXT ('USERENV', 'SESSIONID');
BEGIN
   OPEN user_prog;
   FETCH user_prog
   INTO v_osuser, v_program;
   CLOSE user_prog;

   SELECT COUNT (1)
     INTO v_count
     FROM APPS.allowed_users
    WHERE UPPER (TRANSLATE (v_osuser, 'iiI', 'III')) =
             UPPER (TRANSLATE (osuser, 'iiI', 'III'));

   IF LOWER (v_program) LIKE ('%dis51%') AND v_count = 0
   THEN
      raise_application_error (-20001,'<<<<< You do not have the access right for using Discoverer in this environment.>>>>>');
   END IF;
END;

Thursday, May 14, 2015

ODA X4-2 -- ODA 2.* , Virtualization -- How to add vdisk to a Virtual Machine

If you are using ODA X4-2 , you are likely to be using ODA version 2.x..
Then you will have no command to add a vdisk to your Virtual Machines..
There is no option in oakcli..New oakcli interface comes with ODA Version 12 ..It has commands to create, show and delete vdisks, but at the moment it is not our concern :)Note that : there is no opportunity to use Oracle VM Manager..

So, in ODA Version 2.x , you need to make it manually..

What you have to do is creating a file with your desired size using dd command and then modifying your vm 's vm.cfg file manually in order to reflect this new file as a disk. Lastly, shutdown and startup your vm machine..(note that: you can hot plug the disk if you need to do so..)

The procedure is explained in DOC ID 1951279.1..

Lets make an example:

cd /u01/app/sharedrepo/vmrepo1/VirtualMachines/EBS_12_2_3_TEST_DB
dd if=/dev/zero of=cgcloud.img bs=1M count=200000    (creating an approx. 200 GB file/disk)
vi vm.cfg
Add the newly disk/file:

vif = ['']
name = 'EBS_12_2_3_TEST_DB'
extra = 'NODENAME=EBS_12_2_3_TEST_DB'
builder = 'hvm'
cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'
vcpus = 8
memory = 65536
vnc = 1
serial = 'pty'
disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_TEST_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w', 'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_TEST_DB/cgcloud.img,xvdb,w']
maxvcpus = 8
maxmem = 65536

Dont bother the u'file..
'u' in the disk entry, i.e ['u'file:/disk/location] means unicode, as in character encoding method.
Normally, when we create a VM through OVM manager it adds virtual disks using disk=[file:/...."] entry, however this template is built for the disk to be used as unicode method, which will work perfectly as well.


So dont use "u" for your new disk..

Shutdown and Start your vm
poweroff vm machine..
Start the vm machine : oakcli start vm EBS_12_2_3_TEST_DB

That 's it, your disk is ready for use.. You can see it in fdisk.. It is the disk with the name xvdb, as we add specify it in the vm.cfg..

Disk /dev/xvdb: 209.7 GB, 209715200000 bytes
255 heads, 63 sectors/track, 25496 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Dont bother 209 GB, it is not so... Df will show its real size..

[root@ermanoda1~]# df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_ebs-lv_root   20G  3.0G   16G  16% /
tmpfs                        32G  3.9M   32G   1% /dev/shm
/dev/xvda1                  477M  102M  350M  23% /boot
/dev/mapper/vg_ebs-lv_ebs   266G  240G   13G  95% /u01
/dev/xvdb                   193G  188M  183G   1% /cloud


Note tat , we could use xm block-attach in the hypervisor level to hot plug the disk..

xm block-attach [domain ID] /path/to/image /dev/xvdb

Linux -- Migrating a LVM Disk to a new Server

You may need to migrate your applications or database a new server using storage method such as unmapping the disks from old server and mapping them to the new one.. This method saves you time as you dont need copy from server to server, saves you from network and filesystem waits and so..
But if you have configured LVM on your disk partition, then you may get into panic, thinking what will happen to the LVM partitions in the target server..
This blog post is written to show you the way.. So as long as you Linux versions are identical / or lets say compatible, and as long as you follow this action plan, you are good to go!

Lets make an example:

Here we the disks in our source system..

Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root   37G   15G   21G  43% /
tmpfs                         2.0G  228K  2.0G   1% /dev/shm
/dev/sda1                     239M   54M  172M  24% /boot
/dev/mapper/vgu02-lvu02        40G  5.0G   33G  14% /u02

Lets say we unmap and map the disk mounted to /u02 ..

We umount the /u02

[root@localhost ~]# umount /u02/

[root@localhost ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root   37G   15G   21G  43% /
tmpfs                         2.0G  228K  2.0G   1% /dev/shm
/dev/sda1                     239M   54M  172M  24% /boot

As you see above, it is unmount..

Then we inactivate the volume group associated with it..

[root@localhost ~]# vgchange -an vgu02
  0 logical volume(s) in volume group "vgu02" now active

Next, we export the volume group..
[root@localhost ~]# vgexport vgu02
  Volume group "vgu02" successfully exported

When we use the command to see the physical volume info, we see the vgu02 is exported as follows..

root@localhost ~]# pvdisplay
  Physical volume "/dev/sdb" of volume group "vgu02" is exported
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               vgu02 (exported)
  PV Size               30.00 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              7679
  Free PE               0
  Allocated PE          7679
  PV UUID               Pu5lqh-z0ls-8bbL-kUDC-ORE6-115h-ffEl1k

So at this point our disk is ready from unmapping/mapping , or say it is ready for migration..

In this example, we use Oracle Virtual Box ..It is just for the demo, you can unmap/map your devices from your storage , it does not really matter.. It is the same thing..
Okay, we have 2 Oracle VM Server, we just clone the disk from our source vm for the use of our target vm.. We use xboxmage clonehd command..

vboxmanage clonehd "sourcepath\existingdisk.vmdk" "sourcepath\newcloneddisk.vmdk" --format VMDK

We started to target machine and executed pvscan command..

[root@orcl12c ~]# pvscan
  PV /dev/sdb     is in exported VG vgu02 [30.00 GiB / 0    free]
  PV /dev/sdc1    is in exported VG vgu02 [9.99 GiB / 0    free]
  PV /dev/sda2   VG vg_orcl12c   lvm2 [29.51 GiB / 0    free]
  Total: 3 [69.50 GiB] / in use: 3 [69.50 GiB] / in no VG: 0 [0   ]

As you see above, it discovered the PV volumes in this newly migrated disk.. We see the exported Volume Groups as well..

Next , we you vgimport the import te Volume Group..

[root@orcl12c ~]# vgimport vgu02
  Volume group "vgu02" successfully imported

At this point, we are good to go.. Our volume group is ready. Only thing we need to do is activating and mounting it..

[root@orcl12c ~]# vgchange -ay vgu02
  1 logical volume(s) in volume group "vgu02" now active
[root@orcl12c ~]# mkdir /u02
[root@orcl12c ~]# mount /dev/vg
vga_arbiter  vg_orcl12c/  vgu02/

[root@orcl12c ~]# mount /dev/vgu02/lvu02 /u02/
[root@orcl12c ~]# cd /u02/
[root@orcl12c u02]# ll
total 24
drwxrwxrwx. 4 oracle root  4096 Oct 13  2014 erman
drwxrwxrwx. 2 root   root 16384 Oct 13  2014 lost+found
drwxrwxrwx. 3 root   root  4096 Oct 13  2014 test

Wednesday, May 13, 2015

Linux -- ISCSI, mounting ISCSI disks

          ISCSI is a abbrevation for Internet Small Computer System Interface, and it is a transport layer protocol which works on top of TCP.

ISCSI is a protocol that make us able to use SCSI command over an IP network.. In other words, by using ISCSI ,we use IP switches and Ethernet Cards to use the SCSI commands, to reach devices located in the SAN systems and NAS systems . So it is a cheaper solution than Fibre Channel.

Using ISCSI , we use the shared disks without having to use FCP , FC Host Bus Adapters and SAN switches.. ISCSI does not provide you mount point in the basis of Network filesystems. It actually makes the remote disks , block devices available to you as local disks.. It works via network, but it is not like NAS.
As ISCSI devices are like a local hard disk to you, you can use any filesystem on top of them.. (Ext3,ex2,reiserfs etc..)


In detail , In the standart SCSI protocol, the initiator sends a SCSI command information unit to the target device. ISCSI takes it to a higher level, as ISCSI transports the block level data between the Server and the Storage using TCP packets...

In this context, Server side called as ISCSI initiator and the Storage side called as ISCSI target..

ISCI protocol puts the SCSI commands and data into TCP packets which are sent over the network between the initiator and target..

The ISCSI protocol also put out the commands from the TCP packets when they are reached to their destination.. This way OS sees the storage disks a local SCSI devices.

Things to keep in mind about ISCSI:

CIP,iFCP,FCOE and AoE are aother alternatives to ISCSI .. They all transmit storage data over the IP networks

iSCSI adds extra load to the CPU of the server. Although hardware iSCSI HBAs, which takes this load from the CPU exist, I have never seen they are used in our clients..

Okay.. Enough for the introduction for noew.. I want to write about ISCSI Speed comparisons , as well.. But it is another story.. Today I will write how to mount an iscsi Disk in Linux..

HOW TO MOUNT ISCSI DISK  IN LINUX 


1. For the first time usage, you need install iSCSI Software with below command.. I suppose you have yum confiugured..

yum install iscsi-initiator-utils

2. Make sure iscsi.conf node parameter is set to automatic. This makes iscsi targets to be discovered and connected after rebooting the Linux machine .

[root@erman ~]# vi /etc/iscsi/iscsid.conf

node.startup = automatic

3. Create the disks on NAS storage interface as in the image below. Here we are using a Seagate for our tests..




4. After creating the disks and configured them accessible from ISCI, we start our discovery in the Linux machine to see the iscsi targets..

[root@erman~]# iscsiadm -m discovery -t st -p 10.10.10.101 >>>>>> NAS STORAGE IP

10.10.10.101:3260,1 iqn.1992-09.com.seagate:erman1
10.10.10.101:3260,1 iqn.1992-09.com.seagate:g
10.10.10.101:3260,1 iqn.1992-09.com.seagate:serverad
10.10.10.101:3260,1 iqn.1992-09.com.seagate:cabackup
10.10.10.101:3260,1 iqn.1992-09.com.seagate:oracle02
10.10.10.101:3260,1 iqn.1992-09.com.seagate:oracle01

5. Next, using the iscsiadm with the proper arguments, we attach the storage disk in to our Linux machine..

[root@erman ~]# iscsiadm -m node --targetname "iqn.1992-09.com.seagate:serverad" --portal "10.10.10.101:3260" –login

Logging in to [iface: default, target: iqn.1992-09.com.seagate:serverad, portal: 10.10.10.101,3260] (multiple)
Login to [iface: default, target: iqn.1992-09.com.seagate:serverad, portal: 10.10.10.101,3260] successful.

We check the disk after attaching to it... In this example it is /dev/sda

[root@erman~]# fdisk -l

Disk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes
255 heads, 63 sectors/track, 36468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 4356 34885147+ 82 Linux swap / Solaris
/dev/cciss/c0d0p3 4357 36468 257939640 83 Linux

Disk /dev/cciss/c0d1: 899.8 GB, 899898718208 bytes
255 heads, 63 sectors/track, 109406 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 1 54703 439401816 83 Linux
/dev/cciss/c0d1p2 54704 109406 439401847+ 83 Linux
Disk /dev/sda: 1099.5 GB, 1099512676352 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sda doesn't contain a valid partition table

6. Next we mount the disk with a classic mount command. In this example, the disk was already formatted with ext3 filesystem.. If it would not be so, we could format it and then mount it..

Note that: there is an important note here when you use Iscsi on linux:

When you restart the machine or ISCSI service, your physical device names can be changed . This can lead a problem with your mount. So, too make the mount points to be stable you have to add the ISCSI disk names to /etc/fstab with their corresponding UID.

To find the UID of an ISCSI disk, follow the action plan described in step7

[root@erman /]# mount /dev/sda /yedek2

[root@erman /]# df –h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p3 239G 88G 139G 39% /
/dev/cciss/c0d0p1 99M 22M 73M 24% /boot
tmpfs 32G 0 32G 0% /dev/shm
/dev/cciss/c0d1p1 413G 388G 4.6G 99% /data1
/dev/cciss/c0d1p2 413G 156G 236G 40% /data2
/dev/sda 1008G 77G 931G 8% /yedek2

7. To find UID use the command below..

[root@erman /]# blkid

/dev/sr0: LABEL="OL5.8 x86_64 dvd 20120229" TYPE="iso9660"
/dev/cciss/c0d1p2: UUID="11c9bd31-46db-43eb-ae12-2c6ecab9d3f1" TYPE="ext3"
/dev/cciss/c0d1p1: UUID="37cb3250-e7ce-4f2a-b21a-dda2acbe650c" TYPE="ext3"
/dev/cciss/c0d0p3: LABEL="/" UUID="0acaad8c-6542-4bd5-ae16-674b4ddfaebe" TYPE="ext3" SEC_TYPE="ext2"
/dev/cciss/c0d0p2: TYPE="swap" LABEL="SW-cciss/c0d0p2"
/dev/cciss/c0d0p1: LABEL="/boot1" UUID="0b5563e1-8467-4e00-bca3-119449036729" TYPE="ext3" SEC_TYPE="ext2"
/dev/sda: UUID="56659ab7-76f6-4da5-9281-e23fa1bcfe47" TYPE="ext3" SEC_TYPE="ext2"
/dev/sdc: UUID="1e92b1d2-c67f-4f44-b4dd-d67653db9e6a" TYPE="ext3"
/dev/sdd: UUID="76f943d9-ff21-4483-a387-da57126a0fc9" TYPE="ext3" SEC_TYPE="ext2"
Once you find the UUID , add the necessary line to /etc/fstab file as follows;

[root@erman/]# cat /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SW-cciss/c0d0p2 swap swap defaults 0 0
/dev/cciss/c0d1p1 /data1 ext3 defaults 0 0
/dev/cciss/c0d1p2 /data2 ext3 defaults 0 0
UUID="56659ab7-76f6-4da5-9281-e23fa1bcfe47" /yedek2 ext3 _netdev 0 0

That 's all. I hope you will find this post useful..

Friday, May 8, 2015

EBS/SSL-- support/certification for SHA2 and TLS 1.1,1.2 and above

Regarding SHA2

EBS , even the latest version EBS 12.2 does not support SHA2 certificates.
Oracle states this as follows;
Ref : Oracle Support
"At the present, there is no Oracle solution to this problem. An internal   Bug 8839166- support for sha2 at ssl level has been raised.
For Fusion Middleware 11g, the future plans are that these algorithms will be supported when a release of FMW is released that incorporated
11.2.0.3 Required Support Files or higher."

The workaround for using SHA2 certificates with EBS is using a proxy server or load balancer in  front of the EBS Application Server.

Here is the action plan for accomplishing that:

Option a) Proxy server:

1. Download and install vanilla Apache 2.2 and configure mod_ssl and openssl accordingly.
2. Configure Apache 2.2 as a proxy server to Oracle Application Server" See: Note 1275428.1 - Support Status for SHA2 in Oracle Application Server (10.1.2.X.X/10.1.3.X.X) and Fusion Middleware 11g (11.1.1.X)
The document can be followed for proxy based configuration is :380490.1 Oracle E-Business Suite R12 Configuration in a DMZ / 5.4.1: Update Oracle E-Business Suite Applications Context File

Option b) Load Balancer:

376700.1 Enabling SSL in Release 12 / Step 8 - Update the Context File / Changes when using an SSL Accelerator

Regarding TLS version > 1.0 

EBS does not support TLS versions above 1.0.. Only TLS 1.0 has been certified with EBS R12..
EBS cant support TLS versions above 1.0 due to a limitation in Oracle HTTP Server that comes bundled with EBS installations. On the other hand; Oracle development have plans to certify TLS version > 1.0 with EBS 12.1 and 12.2 ... Unfortuneatly, planned release dates of these certifications are not publicly available yet.
To get the latest certifications about TLS versions, following blog can be followed: Please continue to review the Steve Chan 's blog : https://blogs.oracle.com/stevenChan/entry/out_with_the_old_ssl

So, if you disable TLS version 1.0 in the client browsers due to security issues, you can't use an SSL enabled EBS properly..

The workarounds for using TLS versions above 1.0 with EBS can be using a Reverse Proxy or a Load Balancer in front of EBS Application tier.

Full path ssl, or partially ssl should work , altough it is not tested..


Client --- TLS 1.2 -- Proxy -- TLS 1.0 -- EBS Application Server
Client --- TLS 1.2 -- Load Balancer -- TLS 1.0 -- EBS Application Server


For using Proxy, following action plan can be used:

1. Enable SSL/TLS for EBS  "Enabling SSL or TLS in Oracle E-Business Suite Release 12" ( Doc ID 376700.1 ) 

2. Configure Reverse Proxy according to your Proxy  documentation

3. Configure EBS to point to the Reverse Proxy by following note: "Oracle E-Business Suite R12 Configuration in a DMZ" ( Doc ID 380490.1 )

In conclusion,
As EBS does not support SHA2 and TLS > 1.0 , Reverse Proxy and Load Balancer configurations are needed .. These type of configurations modify the general topology, require installation&configuration work and maintanence..
So, the choice is yours.. Using SHA1 can be an alternative for SHA2 and continuing with TLS 1.0 can be an alternative for TLS versions > 1.0.. 
Maybe Oracle will certify both SHA2 and TLS soon.. Maybe it will not be certified, we dont know yet.. 
So, the choice is yours..
The reason of this blog post is to show you the workarounds which can be used if you must use TLS > 1.0 or SHA2 certificates with your EBS environments..

Lastly, I will share the following picture(ref:https://technology.amis.nl) to show you what the topology looks like when using a proxy server to supply one of these workarounds...


So, The EBS web entry point is the Reverse Proxy URL.
The Clients are speaking TLS 1.2 and the proxy ( configured  properly to start a new separate conversation with EBS server) is speaking  TLS 1.0 with EBS Application Servers.. This configuration should work even if TLS 1.0 is disabled in the client browsers..
For SHA2 certificates, the situation is the same..
I mean [Client]--HTTPS (443)-->[Reverse Proxy] --HTTPS (443 or 4443) --> [EBS Application Tier] will work..

I would appreciate your comments on the paper including the acceptance or rejection on the basis of the things described above.

Wednesday, May 6, 2015

Rdbms -- Generating output HTML reports using SQLPLUS

Just a little reminder for the ones who already know, and a useful feature for the ones who has never used it before.
We can generate Html formatted outputs using sqlplus..
This feature eases our job for producing detailed environment reports in a more readible form..
Using html formatted sqlplus outputs we can scriptize our control scripts or even our health check reports , thus expend less energy while dealing with them..

Oracle uses this to gather info for its components in a formatted manner using this feature(Ref: How To Gather & Backup ASM/ACFS Metadata In A Formatted Manner version 10.1, 10.2, 11.1, 11.2 and 12.1? (Doc ID 470211.1) ).. Why not we?

Here is an example;

Connect to the database using sqlplus

sqlplus apps/apps
SQL>
--Copy & Paste the following:
SET MARKUP HTML ON SPOOL ON PREFORMAT OFF ENTMAP ON  -
HEAD "<TITLE>Database Parameters Report</TITLE> -
<STYLE type='text/css'> -
<!-- BODY {background: #FFFFF6} --> -
</STYLE>" -
BODY "TEXT='#000AFF'" -
TABLE "WIDTH='95%' BORDER='4'"
set pagesize 1000
set linesize 1000
set feedback off
set serveroutput off
spool erman.html
select * from v$parameter;
spool off

After that, execute exit; command.


Here is the HTML formatted output file.. It displays the contents in v$parameter view in this example..

Tuesday, May 5, 2015

Exadata/ACFS -- ASM Resilvering vs ASM Rebalance

The term resilvering is actually a new thing for me.. I saw resilvering processes running in ODA X4-2 systems and wanted a shed some light on them.. How resilvering differs from rebalancing?
Resilvering simply means copying of data from one side of the mirror to another. It is like rebuilding.
Rebalancing, on the other hand; means distributing the data accross the available disks on a disk group..

When we look to these terms from a failure perspective, we can say that;

Resilvering can be seen in Exadata.. For example: ASM resilvering takes place when a write-back mode enabled flash disk fails.. Actually while the mirror copy of it is being written to the disk..
Following picture describes the resilvering process used in Exadata:


Picture Ref: http://www.lunar2013.com

ASM resilvering can also be seen in ACFS based virtualized ODA X4-2 environments. The shared repository volumes may be resilvered after a failure situation.

Here is a quote from an example log file taken from an ODA system:

Asm_startRecovery: recovery needed. 
Asm_relGateway: odlm_unlock returned 0. 
ADVMK-00010: Mirror recovery for volume VMREPO1 in diskgroup DATA started. 
asmResilver2[63209] Asm_resilverVolume: vol VMREPO1, start 

Here is what I can found about in Oracle Community:

As ACFS and ADVMK use ASM in the backend, the ADVM driver will ensure ASM Dynamic volume mirror consistency and for the recovery of only the dirty regions in cases of node and ASM instance failures. This is accomplished using DRL. DRL is an
industry-common optimization for mirror consistency and the recovery of mirrored extents. DRL requires write ahead logging for its mirrored writes.

Resilvering also appears in front of us in the ZFS worlds.
Almost the same thing but a different system..
Reference: Oracle
The process of replacing a device can take an extended period of time, depending on the size of the device and the amount of data in the pool. The process of moving data from one device to another device is known as resilvering and can be monitored by using the zpool statuscommand.
Traditional file systems resilver data at the block level. Because ZFS eliminates the artificial layering of the volume manager, it can perform resilvering in a much more powerful and controlled manner. The two main advantages of this feature are as follows:
ZFS only resilvers the minimum amount of necessary data. In the case of a short outage (as opposed to a complete device replacement), the entire disk can be resilvered in a matter of minutes or seconds. When an entire disk is replaced, the resilvering process takes time proportional to the amount of data used on disk. Replacing a 500-GB disk can take seconds if a pool has only a few gigabytes of used disk space.
Resilvering is interruptible and safe. If the system loses power or is rebooted, the resilvering process resumes exactly where it left off, without any need for manual intervention.

Rebalancing is more familiar to us.. It takes place when a disk fails or when we add new disks in an ASM environment ..
That is ; If a disk fails(as I said we are looking from a failure perspective), ASM distributes the data stored in the failed disk to the rest of available disks. The rebalancing can be described with something like the following picture:


                                                                  Ref: manchev.org
That's all about this topic for now..
any comments will be appreciated..

Monday, May 4, 2015

Exadata X4-2 -- EBS Scan Listener Registration Problem

When using EBS with Exadata, you may encounter a problem with the remote database listener registration.. That is scan listener registration..
Even if your have remote_listener parameter is set and scan listener is configured and running properly, you may see that your database is not registered with the scan listener..
If that 's the case ; set your "ORA_NLS10" environment using "srvctl setenv" and restart your database..
After this action, you will see your database registered with scan listeners.. 
Set your TNS_ADMIN environment variable, as well (just in case)

Example
srvctl setenv database -d LIVE -T TNS_ADMIN=/u01/app/
srvctl setenv database -d LIVE -t "TNS_ADMIN=$ORACLE_HOME/network/admin,ORA_NLS10=$ORACLE_HOME/nls/data/9idata"
srvctl stop database -d LIVE
stvctl start database -d LIVE

Info about ORA_NLS10:

ORA_NLSxx is used to indicate where Oracle RDBMS/client software can locate the definitions of Charactersets (used in NLS_LANG or as NLS_CHARACTERSET/NLS_NCHAR_CHARACTERSET), NLS_SORT, NLS_LANGUAGE (or derived/related parameters) or NLS_TERRITORY (or derived/related parameters).
Those definitions are stored in .nlb files who can be found in the ORA_NLSxx directory.


When database is already created and Oracle Net connections need to be established, ORA_NLSxx is used by the listener and client software to acces the nlb file to define date/time format masks and perform characterset conversions.

For 9i and above , or do not set this so the default is used, or when explicit defining the ORA_NLSxx make sure it's set to the correct location.


RDBMS -- PMON default listener registration , dynamic listener registration

If there is a listener running on port 1521, by default Pmon registers the database service with that listener..

This is the default behaviour of pmon..

In 11g , we can disable this behaviour though. There are 4 ways to disable this behaviour in 11g.

1) For versions >=11g: disable the automatic listener registration on the listener side use DYNAMIC_REGISTRATION_listener_name=off , in listener.ora -- DYNAMIC_REGISTRATION_LISTENER = off

2) disable the automatic registration from database side -- set EVENT= "10258 trace name context forever, level 4" on the database side

3) Do not use the default port (1521) for any listener

4) use the init.ora "local_listener" parameter to define the second database listener, LISTENER2, as the "default listener".


the 3rd and 4th options are familiar but what about the 1st and 2nd.. Okay.. Lets take a look at the automatic listener registration ...

Automatic listener registration is on by default. In 11g it is controlled by dynamic_registration_listener parameter..

With dynamic registration enabled(it is on/enabled by default), our database can register itself to the listener, without specifying SID_LIST_listener_name parameter in listener.ora ..  As mentioned above, pmon by default register the database with the listener on default port(1521) dynamically/automatically..
In adddition to that; if we use local_listener parameter in our database parameter file , pmon can register the database with the desired listener listening on non-default port dynamically, too..

When "DYNAMIC_REGISTRATION_listener_name" is set to "off", the listener refuses dynamic registration, so we need to specify the services which we want to register our listener with..
Like the following;

#listener.ora
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = ERMAN)
(ORACLE_HOME = /erman/dbhome_1)
(SID_NAME = ERMAN)
)
(SID_DESC =
(GLOBAL_DBNAME = ERBANT)
(ORACLE_HOME = /erman/dbhome_1)
(SID_NAME = hyd)
)
)
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ermanhost)(PORT = 1521))
)
Lastly;
EVENT= "10258" controls this behaviour from the database size, when it is set, the database does not register itself with the listener dynamically..

Friday, May 1, 2015

Exadata X4-2 -- Third party Backup Agent connectivity problem

In an Exadata X4-2 migration project, a problem was escalated to me..
It was related with the Data Protector agent..
The agent, which was installed in one of the Exadata db nodes, could not be reached by its Server.
When I used telnet to check the agent's port, I saw the message "Escape character is '^]'." and after that a line which was saying "connection closed by foreign host"..
So , the server agent could not be reached from the outside, but the message "Escape character.." gave me the clue.. Actually the agent could be reached physically, because telnet was saying the "Escape character.. ", but then something else was closing the connection..
Note that : Firewalls and iptables were not running.
So having these in mind; I thought that the problem must be in an additional layer of security after the firewall and the service daemon.

Thus, immediately checked the TCP wrappers/hosts.allow/hosts.deny files and saw the problem..
Please read if you are interested with the idea of TCP wrappers: 
"TCP WRAPPER Network monitoring, access control, and booby traps" by Wietse Venema.. 
I find that doc excellent.. Here is the pdf file : ftp://ftp.porcupine.org/pub/security/tcp_wrapper.pdf

Anyways, inthe hosts.allow file the configuration was as follows;

cat /etc/hosts.allow
sshd : ALL
snmpd : ALL
ALL : localhost

To fix the problem, I added the data protector server 's IP address next to the localhost as follows;

cat /etc/hosts.allow
sshd : ALL
snmpd : ALL
ALL : localhost , 10.255.1.77

Note that: hosts.deny was file as follows;

cat /etc/hosts.deny
ALL:ALL