Saturday, April 26, 2014

Exadata -- For Exadata Database Machine Admins -- Filtered Information

This post will be based on the information concerning Exadata Administrators.. You will find information gathered on the way for being certified on Exadata..
The information provided will be item by item, and I think
this post will be useful for Exadata Admins even though they already have field experience..
This post will be unstructured,.I mean the information provided in this document is not grouped into subtitles, but it will present the whole picture about Exadata Administration, when you will read the all of it. The information provided in this document will be like snips from the Exadata knowledge..
In order to understand this documents, you should already know the basic concepts of Exadata.( like Storage indexes, smart scan and etc.. )

Let start;

You can make indexes invisible to increase the chance for using storage indexes.
You can load your data stored on the filtered columnd to maximize the benefit of storage indexes.
Bind variables can be used with storage indexes.
Oracle Database QOS management can offer recommendations for the CPU bottlenecks. QOS management can not provide recommendations for the Global Cache resource bottlenecks. OQS management cannot resolve IO resource bottlenecks, too.
For media based backups, it s recommended to allocate equivalent number of channels and instances per tape drive.. For this type of backups, the network cables between Exadata and media servers should be connected through Exadata Database nodes. 
Bonding on Exadata can be used both for reliability and load balancing, but when I look to a Production Exadata Machine , which is X3, I see that the bonding is configured for reliability.

cat /proc/net/bonding/bondeth0
Ethernet Channel Bonding Driver: v3.2.3 (December 6, 2007)
Bonding Mode: fault-tolerance (active-backup)
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"

active-backup or 1 -> Sets an active-backup policy for fault tolerance.
Transmissions are received and sent out via the first available bonded slave interface.
Another bonded slave interface is only used if the active bonded slave interface fails.

To guarentee proper cooling for an Exadata Machine, perforated floor tiles should be placed at the front, because the air flow is from front to back.

Creating multiple grid disks on a single disk in Exadata provides us multiple storage pools with different performance characteristics and multiple pools that can be assigned to different databases.

Here is a general information about Disk layout in Exadata
Physical disk -> Lun -> Cell Disk( its like a filsystem on Lun..) -> GridDisk ( it s like a partition)
Lastly Grid disks are served to the ASM diskgroups..
Note that , we can create flash based ASM diskgroups,too. They occupy may share space with the Flash Cache on the flash disks.

Things to consider for Exadata migrations,
For Transportable Database method-> Source database should be and little endian
For Data Pump method ->, is good, but it is time consuming..
For Data guard physical method ->, Source platform must be Linux, Solaris x86 or Windows and source must be on 11.2.
For ASM rebalance method -> 11.2 Linux x86 64 database that uses  ASM w/ 4MB AU.
Transportable tablespaces method -> Big endian source >= 10.1 or Little endian source>=10.1, <11.2  is needed
Logical standby method -> Source does not need to be on 11g. Logical Standby is not supported from HPUX to Linux migrations.

Also here is an useful information about the methods:

Standby Physical Standby:
Source platform must be Linux, Solaris x86 or Windows (see Document 413484.1) and source on 11.2
Source database in ARCHIVELOG mode
NOLOGGING operations to the database are not permitted, i.e. FORCE LOGGING is turned on at the database level

Transportable Database

Source system must be 10gR2 ( and little endian
Stricter service level requirements that obviate the required downtime with Oracle Data Pump

Transportable Tablespaces
Any source platform
EBS Release 12 with source database RDBMS release 10gR2 ( or higher
EBS 11i with source database RDBMS release 11.2
Service levels cannot tolerate a significant outage.  Using transportable tablespaces instead of Oracle Data Pump export/import can significantly reduce the outage time for large (> 300 GB) EBS databases.  Thorough testing will determine the precise outage time.
For a point of reference, tests on an Oracle Exadata Database Machine quarter rack with the Rapid Install Vision database (about 300 GB) took about 12 hours.  This time should remain about the same regardless of the amount of data in the database.  This is because the metadata creation takes the longest time in the migration process and accounts for the bulk of time.

Oracle Data Pump
Any source platform
Source database is RDBMS release 10.2 or higher
To implement Oracle Exadata Database Machine best practices on the target
Service levels can tolerate a significant outage.
For a point of reference, tests on an Oracle Exadata Database Machine quarter rack with the Rapid Install Vision database (about 300 GB) took about 24 hours (export - 7:42; import - 16:42) using Network Storage and no dump file copy (i.e. the export dump storage was mounted on the source and the target).
Timings will vary depending on your system configuration and increase as the amount of data increases.

In Exadata, Oracle Enterprise Agents must be deployed to the compute nodes. .Oracle Exadata Plug-in deployed with the Agent. Plugins allow you to monitor the following key components of Exadata machine. There are several plugins for Grid Control and Cloud Control (such as Avocent MergePoint Unity Switch, Cisco switch, Oracle ILOM, Infiniband switch, PDU) .. Note that : A trap forwarder is required to catch cisco switch and kvm traps due to a port mismatch..
Agent communicates with Storage Server and Infiniband Switch targets directly. Oracle Exadata Plug-in also monitors the other DBM components. Oracle Enterprise Manager 12c agent collects data and communicates with the remote Enterprise Manager Repository.

In Exadata X3, we have 512 MB flashlogs on the storage servers.
In compute nodes, we have raid 5 arrays;

Virtual Drives
Virtual drive : Target Id 0 ,VD name DBSYS
Size : 556.929 GB 
State : Optimal
RAID Level : 5

Exadata -> Action plan to replace a Flash disk when Griddisks are created them,
Ref: Replacing FlashCards or FDOM's when Griddisks are created on FlashDisk's (Doc ID 1545103.1)

If the flash card needs to be replaced, drop the disks used in +ASM for the FlashDisk
Drop FlashCache / Flashlog and delete celldisks of type Flashdisk
create flashdisks
Now create your griddisk's back on DiskType: FlashDisk
add back the disks to the diskgroup. (it is optional ->If you used 'force' when dropping the disks from ASM then Exadata auto management should automatically add these disks back into ASM.)

Note that, we have different failure types in Exadata. For example a disk which is in predictive failure state must be replaced immediately. ASM will drop these kind of disks automatically from the associated disk group , and start a rebalance operation.
For flash a flash disk also; if the disk is in predictive failure then ->
If the flash disk is used for flash cache, then flash cache will be disabled on this disk thus reducing the effective flash cache size. If the flash disk is used for flash log, then flash log will be disabled on this disk thus reducing the effective flash log size. If the flash disk is used for grid disks, then Oracle ASM rebalance will automatically restore the data redundancy..

If you want to change a memory dimm or want to make a similar activity, you just need to shutdown the affected cell.. The database, and the other cell servers will not be affected from this operation.
What you have to do is;
Ensure asmdeactivityoutcome of the Griddisks first..
Then inactive all the griddisks on that cell and ,shutdown the cell using shutdown -h now ..

quartery battery learn cycle is a normal maintanence activity.. It is used for charging the batteries. When this kind of activity is happening, you can end up with a performance degregation in terms of I/O, as this event may require the related device to be put in the write through mode.. Keep that in mind and, If you can or need, set the device back to write back mode...

Configure and use smart flash logs if you need a better LGWR performance, as LGWR writes redo data both flash and disk in parallel.. It considers whichever of these writes completes first, as done..
Smart Flash Logs are a new feature comes with cell software. They are not for reading, they are used like a circular buffer for Redo writes.. Smart Flash logs can enhance the performance of an OLTP database. By default they occupy 512 MB per cell. (32 MB size per flash disk *16 flash disk) space in the Flash Cards.. So Smart Flash logs reduce the size of Flash Cache in a manner.
Note that , Flash Smart logs can be enabled or disabled using IORM according to the databases if needed..
Also LGWR writes and Controlfile IO will be automatically on high priority in IORM..  On the other hand, DBWR IO is managed automatically at normal priority.
Flash logs if needed somehow, can be dropped using drop flashlog command through the cellcli utility residing on the storage servers..

Consider using CTAS for bulk data loading from external tables to Exadata.. CTAS automatically uses direct path load.. Insert /*+append*/ can also be used for this kind of data loading, as by using the append hint, oracle will use direct path loading in insert operations, too..

cellip.ora is the configuration file , if you want to separete cells which are connected by the asm instances. This file is located on where ASM resides(compute node), and it basically tells ASM which cells are available. cellip.ora is located in every compute node, and its contents are like following;

cat /etc/oracle/cell/network-config/cellip.ora

This file should also be used if you want to add an expansion rack to the storage grid.

In default configuration , DATA and RECO diskgroups are build on top of non-interleaving disks. For detailed information about Interleaving , please see my following post
Also, in default configuration, we dont have any free space in Flash Disks;

CellCLI> list celldisk where name='FD_15_cel01' detail
name: FD_15_cel01
creationTime: 2012-01-10T10:13:06+00:00
deviceName: /dev/sdy
devicePartition: /dev/sdy
diskType: FlashDisk
errorCount: 0
freeSpace: 0
id: 8ddbd2c8-8446-4735-8948-d8aea5744b35
interleaving: none
lun: 5_3
size: 22.875G
status: normal

Note that , you can use infiniband to connect an Exadata to a Exalogic. You can also use infiniband to connect and Exadata to a Oracle ZFS Storage ZS3, as ZS3 has a native infiniband connectivity.
Alternatively, Sun Zfs Storage 7420 Appliance can be connected to Exadata directly from infiniband.. 
In addition, you can connect any media servers which have infiniband cards, to Exadata via infiniband. Then you can connect those media servers to tape libraries to maximize the tape backup thorughput..
In terms of tape backups, Oracle docs suggest to have Disk-to-disk-to-tape, in other words D2D2T strategy, which allows keeping old backups on tape while retaining new/fresh backups on disks for achieving fast recovery times.  You can consider the following Oracle Slide as a good Exadata-tape backup scenario;

Uncommited transactions and migrated rows can cause cell single block physical reads even if you are doing a Full table scan. Note that: Single block reads and Multi block reads may benefit from the Flash Cache,
Also note that, smart scan can not be done against Index Organized Tables and clustered tables.

When you need to apply a bundle patch  to Oracle Homes in Exadata, you will need to use oplan utility.
oplan utility generates instructions for applying patches, as well as instructions for rollback. It generates instructions for all the nodes in the cluster. Note that, oplan does not support DataGuard .. Oplan is supported since release . It basically eases the patching process, because without it you need to read Readme files and extract your instructions yourself..
It is used as follows;
as Oracle software owner,(Grid or RDBMS) execute oplan;
$ORACLE_HOME/oplan/oplan generateApplySteps <bundle patch location>
it will create you patch instructions in html and txt formats;
Then, choose the apply strategy according to your needs and follow the patching instructions to apply the patch to the target.
That 's it.. 
If you want to rollback the patch;
execute the following;(replacing bundle patch location)
$ORACLE_HOME/oplan/oplan generateRollbackSteps <bundle patch location>
Again,   choose the rollback strategy according to your needs and follow the patching instructions to rollback the patch from target.

It is mandatory to know which components/tools are running on which servers on Exadata;
Here is the list;

DCLI -> storage cell and compute nodes, execute cellcli commands on multiple storageservers
ASM -> compute nodes -- it is ASM instance basically
RDBMS -> compute nodes -- it is Databas software
MS-> storage cells, provides a Java interface to the CellCLI command line interface, as well as providing an interface for Enterprise Manager plugins.
RS -> storage cells,RS, is a set of processes responsible for managing and restarting other processes.
Cellcli -> storage cell, to run storage commands
Cellsrv ->storage cell. It receives and unpacks iDB messages transmitted over the InfiniBand interconnect and examines metadata contained in the messages
Diskmon -> compute node, In Exadata, the diskmon is responsible for:,Handling of storage cell failures and I/O fencing
,Monitoring of Exadata Server state on all storage cells in the cluster (heartbeat),Broadcasting intra database IORM (I/O Resource Manager) plans from databases to storage cells,Monitoring or the control messages from database and ASM instances to storage cells ,Communicating with other diskmons in the cluster.

The following strategy should be used for applying patches in Exadata:

Review the patch README file (know what you are doing)
Run Exachk utility before the patch application (check the system , know current the situation of the system)
Automate the patch application process (automate it for being fast and to minimize problems)
Apply the patch
Run Exachk utility again -- after the patch application
Verify the patch ( Does it fix the problem or does it supply the related stuff)
Check the performance of the system(are there any abnormal performance decrease)
Test the failback procedure (as you may need to fail back in Production, who knows)

Multiple Grid disks can be created on a single Cell Disks, as you know. While creating multiple Grid disks on a Single Disk, you can end up having multiple disks which have different performance characteristics . In order to have more balanced disk layout, you can use interleaving options or Intelligent Data Placement technology which is based on ASM..

The internal Infiband network used in Exadata transmits IDB messages between compute nodes and storage servers as well as Rac interconnect traffic packets between the compute nodes in the cluster.

Dbfs or a Nfs filesystem can be used as a stage for loading data from the external tables. If choosing Nfs to load data, the Nfs share should be mounted to the preferred compute node..
Dbfs can be created on DBFS_DG , as well as on a standart filesystem..
It can enhance performance if you need to bulk load data in to your Production System , which resides on Exadata.. By using DBFS created on ASM, you will have a parallelization in storage, which will enhance perfomance of IO. script can be used to gather the diagnostic information with software logs, trace files and OS information..

Exadata storage server has alerts defined on it by default.. This alerts are based on some defined metrics.. We can define new metrics also, this metrics will persist accross cell node reboots.

If you have a database(little endian) and want to migrate it to Exadata with minimum downtime, you can upgrade it to 11.2.0 and use Data pump physical to minimize downtime or alternatively, you can use Golden Gate for this. Ofcourse you can use datapump, as well, but your with datapump the downtime will be significant..

According to the Non-interleaving disk configuration in Exadata(which is default), we can say that the first Grid diskdisks created using Create Griddisk... command will have the best performance.  So the Diskgroup created based on the first created griddisk will have better performance than the other Diskgroup on the Same Exadata Machine. ...

If you want to do  some administrating stuff on Exadata Storage servers, like dropping the cell disk and etc;
you can use celladmin OS account in storage servers to do this kind of an operation. You can use cellcli(on every cell) or dcli(from one cell to administer all of the cells) utility to execute your commands..
DCLI is a pyhton script. It is used to execute command on the cells remotely without login in to them.. (In first execution it create ssh keys with the following -> dcli -k -g mycells.)

Note that, we have firewall (iptables) configured with Oracle Supplied rules on cell servers.

In OLTP systems,
Exadata write-back flash cache can enhance performance , as it also caches database block writes.
Also Flash log can be useful for enhanching perfomance of OLTP systems, as fast response time for Log Writer is crucial in OLTP..
For Big OLTP systems, Flash cache and Flash log can enhance performance and High Capacity Disks in Exadata can meet the storage size needs..

Note that , IDP, IDB and IORM manager can only seen on an Exadata Environment.

IORM is used for managing the Storage IO resources.

Here is a diagram for the description of the architecture of IORM. (reference: Centroid)

So we can manage our IO resources based on the Categories, Databases and consumer groups. There is a hierarchy as you see in the picture.. The hierarchy used to distribute I/O.
IORM must be used on Exadata if you have a lot of databases running on Exadata Machine.. IORM is a friend of consolidation projects, in my opinion..

If you configured ASR manager in your environment, note that : database uses SNMP to transfer the notifications from database to ASR Manager and these notifications are forwarded using SNMP from ASR to Enterprise Managern In addition Faults are tranferred to the Oracle securely using https.

The fault telemetry sent from ASR manager is in the below format;

Telemetry Data:

System-id: System serial number.
¦ Host-id: hostname of the system sending the message.
¦ Message-time: time the message was generated.
¦ Product-name: Model name of the server

As known, Exachk is the utility to validate an Exadata Environments. We check the system using this utility time to time(before patches, after patches and etc..) Also, it can be scheduled to run regularly.. To schedule exachk, we can create a cron job or we can create a job in Enterprise Manager..

Compression on Exadata can only be done on the Compute nodes.. Decompression, on the other hand, can be done on Compute Nodes or Storage Cells.. Decompression can be done on Storage Cells if the associated operation is based on the Smart Scan.. Same rule applies for Encryption and Decryption, too..

Here is a general information about compression types:
BASIC compression, introduced in Oracle 8 already and only recommended for Data Warehouse
OLTP compression, introduced in Oracle 11 and recommended for OLTP Databases as well
QUERY LOW compression (Exadata only), recommended for Data Warehouse with Load Time as a critical factor  --> HCC
QUERY HIGH compression (Exadata only), recommended for Data Warehouse with focus on Space Saving --> HCC
ARCHIVE LOW compression (Exadata only), recommended for Archival Data with Load Time as a critical factor --> HCC
ARCHIVE HIGH compression (Exadata only), recommended for Archival Data with maximum Space Saving --> HCC

If your index creation takes long time in Exadata, consider the following information:
Cell single block physical read can impact the peformance of an index creation activity. Cell single block physical reads are like db file sequential reads in a tradional system.
Migrated and chained rows can cause cell single block read events on an Exadata Machine.  Also uncommited rows during a query is handled based on the consistency.. For supplying the consistency, Database nodes may require additional blocks. These blocks are sent by the Cell servers to the database nodes.. This activity can cause cell single block physical reads , as well. If you have a lot of blocks in one of these conditions, then your index creation time can take long time..

When migrating to Exadata, you should consider Database Type(oltp or warehouse), size of the source database, the version of the source database and the Endian format of the Source Operating Systems.. By analyzing these inputs you can choose an optimal migration method and strategy.

In Exadata Enterprise Manager monitoring, the communication flow from the ILOM of the Storage Servers to Enterprise Manager is through the Storage Server's MSprocesses. ILOM sends data using snmp to MSprocess, and MSprocess send the data to the Enterprise Manager using snmp. Data is triggered and transfered through the snmp traps.
Based on the preset thresholds defined in ILOM, we can monitor motherboard,memory,power, and network cards of Database Nodes using Enterprise Manager. We can see the faults and alerts  produced for these hardware components from Enterprise Manager.

Oracle Auto Service Request (ASR) is a secure, scalable, customer-installable software feature of warranty and Oracle Support Services that provides auto-case generation when common hardware component faults occur. ASR manager can be installed on an external Oracle Linux or Oracle Solaris server. Also you can use one of Exadata db nodes for installing ASR manager (not preferred).
ASR manager communicates with Oracle using https..

In Exadata, some database work can be offloaded to Storage Servers as you know.. Besides operations like full table scan, single row functions and simple comparison operators, some joins can be offloaded to storage. Column filtering, Predicate filtering  and Virtual Column filtering are the things that can be offloaded to the Exadata Storage Servers, as well.
If you want to see all the functions that can benefit from smart scan; you can use the following sql:
select name from v$sqlfn_metadata
The output will be like;


Operations like full table scan and fast full index scan executed in parallel always generate Smart scans.. For full table scan, you can see cell smart table scan, and for fast full index scan, you can see cell smart index scan events.. Fast full index scan operations can be executed through the smart scan because in fast full index scan, Oracle just reads the index block as they exist on the storage.. I mean a bulk read will be performed, and that makes Oracle to use smart scan even if it is an index operation.  Smart scan is performed during Direct path read operations.. Parallel queries use direct path read, that 's why they make use of smart scans.. 
So if we need to gather all together; in order to have smart scans, we need to execute queries in parallel, we need to use direct path reads towards the process memory and we need to have cell.smart_scan_capable parameter set to TRUE for our ASM diskgroups.

We have Sun servers in Exadata.. Compute nodes and storage servers are acutally Sun Servers. They have ILOM cards on them. These ILOM cardscan be used to administer these servers remotely.. For Example ILOM can be used to power-on database servers or open a remote console for a Storage Server.

Note that, if you want to check the status of all ports located in the Infiniband Switch, you can use Enterprise Manager or script located in the Infiniband switch.

To properly shutdown the Exadata, we need to first stop the database and grid services. We may use crsctl stop cluster -all command for this. Then we need to shutdown Exadata storage servers first. Then we need to shutdown Database servers. Lastly we need to power off the network switches and cut the power using power switches on PDUs. Why firstly shutdown storage servers? ( this is still a mystery for me)

After Exadata migrations, analysis are made to drop the indexes . This activity is done to increase the chance of using smart scans on our query, but care must be taken while dropping those indexes. For example, in an OLTP system we need fast response times for single block reads, so dropping an index may result a negative performance impact for some of our OLTP queries.. So it s better to drop an index after analyzing the queries of the corresponding application. You can use invisible indexes to see the difference .. You can check Execution plans to see if Oracle wants to use smart scan instead of an index access for a query.. According to your analysis, make the decision to drop the unnecessary indexes..

As you know Exadata contains Compute Nodes and Storage Nodes.. For Storage Nodes, you dont have the choice to use a different OS than Linux. Exadata Storage Servers are Linux, and will continue to be Linux. Oracle Linux servers are coming with Uek kernels.
 On the other hand, you can choose to have Solaris 11 rather than Linux for compute nodes. Operating systems are selectable at install time.

Note that, In Exadata we have different networks, like Management network and public network, infiniband network.

Following is a picture representing those networks; (it is for X4 actually, but it is useful)

Ssh, for example , works from the Management network, as it is an utility to manage the corresponding servers.  If you want to change the network that ssh listens from, you can use ssh config file to do that.

Following inputs can be used for configuring the Exadata Machine at install time..

Customer name
Compute OS
Database Machine_Prefix
Admin Network -> Start Ip address for pool,pool size,Ending Ip address for pool,Subnet Mask,Gateway
Admin Network -> Database Server admin name, Storage Server Admin Name, ILON Name,
Client Network ->Start Ip address for pool,pool size,Ending Ip address for pool,Subnet Mask,Gateway, Adapter speed (1gbe/10gbe Base-t or 10gbE sFP+optical)
Infiniband NEtwork -> Start Ip address for pool, pool size, ending ip address for pool, subnet mask, compute node private name
Backup / Data Guard Ethernet NEtwork
Os configuration -> Domain , DNS, NTP , Grid ASM home os user ,asm dba group, asm home oper group, asm home admin group, rdbms home os user, rdbms dba group, oinstall group, rdbms home oper group, base loc for grid and rdbms --> you can set userid,groupid of all users and groups
Home and Database -> ınv location, grid home, db home loc, software install, db name, data reco group names and redundancy (cant change their sizes), block size , type DW or OLTP
Cell Alerting -> Enable Email Address
ASR conf
OCM conf
Grid Control agent

In order to actually have the Exadata Storage Servers send notifications via email (or alternately SNMP) each of the servers has to be configured with the appropriate settings. This is done using the ALTER CELL command in CELLCLI.

ALTER CELL smtpServer='mailserver', -
smtpFromAddr=', -
smtpPwd='email_password', -
smtpToAddr='', -
notificationPolicy='critical,warning,clear', -

The alerts may be stateful and stateless, as well..
If the alert is based on a threshold, it gets cleared automatically when it no longer violates the threshold;
For example; a filesystem alerts , as follows;

CellCLI> list alerthistory detail
name: 4_1
alertMessage: "File system "/" is 82% full, which is above the 80% threshold.
Accelerated space reclamation has started.
This alert will be cleared when file system "/" becomes less than 75% full."

Also alerts can be fired for Critical, Warning and Information purposes..

We have Storage indexes located in the physical memory of  the  Storage Servers.. Storage indexes are maintained automatically  the cellsrv and they are not persistent across reboots..(as they are in memory).. Cellsrv build these indexes based on the filter columns of the offloaded queries.
In Storage indexes, Oracle builds range regions based on the column values. By the use of storage indexes Oracle can easily find where to look in the storage for a given column value..
Maximum 8 columns for a table are indexed per storage region, but different storage regions can have different columns indexed for the same table.

Storage servers are very sensitive environments in Exadata, so Oracle doesnt support a lot of activities on them.. For example, we can change root password of these servers or we can set up a ssh equivalence for cellmonitor users(
 But for example, if we want to upgrade the storage server software, we need to use patchmgr utility..
the patchmgr utility is a tool Exadata Database Administrators use to apply (or rollback) an update to the Oracle Exadata Storage Cell.
Here is another restrictions is ;
Oracle Exadata Storage Server Software and the operating systems cannot be modified, and customers cannot install any additional software or agents on the Exadata Storage Servers.

Lastly, I will mention about placing multiple Exadata Machines in a System Room/Data Center.
If you have multiple Exadata Database machines, you need to place them side by side while ensuring the exhaust air of one rack does not enter the airinlet of another. If you have multiple clusters running on several Exadata Machines, you can place the racks that are part of a common cluster together/side by side..

That' all for now. I hope you will find it useful.


  1. Hi Erman,

    I'm referring the above post to clear my Exadata (1Z0-027) Certification. I'm worried now because X4 Administration also got added in the same Certification. Now total there are 90 questions in 1Z0-027 Certification. Tomorrow is my exam date. Can you please tell me what was your score for this exam ?

    Do you have any X4 related Q &A to refer ?


  2. Hi Manish,

    I know, the exam has changed recently. X4 specific questions have been added. So you need to study for Exadata X4 , too. You need to refer DOCUMENTATION FOR EXA 11.2 & 12.1, It is on Oracle Support, it comes with Patch 10386736.

  3. Also note that, this is just an article, I wrote what I did learn while studying for Exa X3 exam..

  4. Thanks Erman for the Oracle Support Link. Will let you know about my result tomorrow.

    1. What happened to your result ???

  5. Exam questions are totally changed and more questions from IORM

  6. Hello Emran,

    Indeed it's very nice article.

  7. anyone has a updated test king? it's impossible to pass :(

  8. Hi Emran,

    This is not correct in Exadata:

    Transportable Database

    Source system must be 10gR2 ( and little endian
    Stricter service level requirements that obviate the required downtime with Oracle Data Pump

  9. Why not correct?
    Reference :Migrating an Oracle E-Business Suite Database to Oracle Exadata Database Machine (Doc ID 1133355.1)

  10. Hi Erman,
    official recommendation for tape-based baclup configurations goes like this:
    Configure one RMAN channel per tape drive and add tape drives to scale backup rates
    though this post is not accurate and may be misleading:
    'For media based backups, it s recommended to allocate equivalent number of channels and instances per tape drive..'
    Following this recommendation I can allocate more than 1 channel per tape which is contrary to BP:
    Do you agree?
    Any references supporting your statement?

    Oracle ACS

  11. Hi Timur,

    Refencence: An Oracle White Paper September 2011 Oracle Exadata Database Machine - Backup & Recovery Sizing: Tape Backups

    Maximum Availability Architecture (MAA) best practices recommend configuring the same number of
    RMAN channels as the number of tape drives allocated for backing up the database.

    "In other words, allocate one RMAN channel to one tape drive. "

  12. So if you have two tape drives, again allocate one rman channel to one tape drive. Then you will have two rman channels mapped to these two tape drives..
    I can not see why you find it not accurate?

  13. I totally agree with all that you've just said.
    It's just a matter of how you understand of what you read.
    I saw the simialr answer for the question:
    'Which two are Oracle recommendations for media based backups performed for a database
    running on a Database Machine?'
    while preparing for the exam.
    There is no word 'one rman channel' in this context, so equivalent number of channels and instances for me means
    that number of channels should match number of instances (per tape drive). It is not said here that
    one rman channel should be equivalent one tape drive.
    Besides if your statement (answer) is correct there will be no option to choose another 100% correct answer for this question.
    I think the problem is that answer is poorly formulated.

  14. I understand what you mean. Okay. But this blogpost is not an exam review. The things written in here, is just some bunch of information which I have gathered while preparing the exam. The purpose of this blog post (also this whole blog) is to enhance te technical knowledge of mine and the ones who follow this blog. Thanks for you interest , really. I m glad for you to spend time reading and making comments about this post. The true and more understandable statement is then "allocate one RMAN channel to one tape drive." , "allocate two RMAN channel to two tape drives, one for each" and so on."

  15. Hi Erman,

    Regarding bonding , I think the bonding improve the reliability and performance. i find the following: Network bonding enables two or more network interfaces to act as one, simultaneously, increasing network bandwidth and/or providing network interface redundancy.(Doc ID 738269.1), and the bonded ports can be configured for load balancing, fault tolerance and BETTER network utilization (Oracle Exadata Database Machine - Backup & Recovery Sizing: Tape Backups).

    Please tell me what do you think about it?

    Thanks - great blog.

  16. Yes, bonding can be be used both for improving performance and reliability.
    In this post, I wrote "it improves reliability in Exadata", because in version X3, bonding was active-backup by default.

    When channel bonding is configured for the client access network during initial configuration, the Linux bonding module is configured for active-backup mode (mode=1). If a different bonding policy is preferred, then you may reconfigure the bonding module after initial configuration.,

    For example: In case of Infiniband; In the default deployment of the Exadata X4-2 Database Machine, the InfiniBand bonding is set up in the active/active mode, on all database servers and all storage cells. Earlier models of the Exadata Database Machine (V2, X2-2, X3-2) have the InfiniBand set up in the active/passive mode.

    (Oracle Exadata Database Machine - Backup & Recovery Sizing: Tape Backups : FIGURE 1 RMAN BACKUP USING INFINIBAND CONNECTIVITY AND ORACLE SECURE BACKUP)
    Looking at the best practices, MAA makes Exadata compute nodes to connect to Media servers using Single Infiniband QDRs.
    Each compute nodes uses 1 QDR to connect to IB switch. From IB switch to the Media servers, there are 2 QDR used per media servers. (active-active mode)
    I didnt understand this one. Why using 2 QDRs between IB switch and media servers? If the answer is reliability then why dont they use multiple QDRs between computenodes and the IB switch.. Strange.
    Anayways;continuing.. In MAA the Media Servers connect to the Tape Devices using 4 Fibers.
    each Media servers connects to the tape drives using 4 fibers (in active active mode..) I think this is for increasing fibers throughput...

    1. The bonding and backup configruration is actually It depends on your needs..
      In the same paper there is another scenario.
      Example Scenario 2: Gigabit Ethernet

      As you may see; there are some calculation there.. Throughput calculation.. According to these calculations:
      For example: Customer wants 4 hrs to backup 6TB
      So it makes @ 2TB/hr = 560MB/sec
      According to this info, the configurations is done.

    2. great information ... thanks a lot Erman!!

  17. Hi Sir,
    Please, I am a bit confused about Oplan. I know Oplan is used to generate instructions specific to your environment. what about if you use RAC. Oplan will generate instruction for patching RAC oracle home and grid GRID infrastructure home or it will generate instructions for patching all the components on the database machine.

    Thank you.

  18. OPlan is a utility that facilitates the patch installation process by providing you with step-by-step patching instructions specific to your environment for both in-place and out-of-place patch installation. Currently, OPlan is supported for Exadata Database Bundle Patches. For the latest information, see "Oracle Software Patching with OPLAN" in My Oracle Support Note 1306814.1

  19. correct sequence to properly shutdown the Exadata

    crsctl stop cluster -all on database server
    Then we need to shutdown Database servers First
    Then shutdown Exadata storage servers
    Lastly we need to power off the network switches and cut the power using power switches on PDUs.

  20. What is your reference?
    In Document : (Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration. (Doc ID 1093890.1)) says ;

    9) Power off all cells:

    # dcli -g cell_group -l root poweroff

    10) Copy the /opt/oracle.SupportTools/onecommand/dbs_group to /opt/oracle.SupportTools/onecommand/dbs_group2:

    # cp /opt/oracle.SupportTools/onecommand/dbs_group /opt/oracle.SupportTools/onecommand/dbs_group2

    11) Remove the “first compute node name” from the /opt/oracle.SupportTools/onecommand/dbs_group2 file (using any editor, e.g. vi).

    12) Power off all the compute nodes (except compute node # 1):

    # dcli -g dbs_group2 -l root poweroff

    13) Power off the compute node #1:

    # poweroff

    14) Now, both Cell Servers & Compute nodes are down for maintenance.

    1. "crsctl stop cluster -all on database server
      Then we need to shutdown Database servers First
      Then shutdown Exadata storage servers
      Lastly we need to power off the network switches and cut the power using power switches on PDUs."


      Doc ID 1093890.1 is also correct and a practical way of doing it, as you prefer to SSH to cells from compute nodes and then carryout your tasks.

      I believe it doesn't matter as long as cluster services are shutdown properly.

      But normally exams questions follow the course ware.

  21. Yes, it seems it does not matter..For example: In doc :Exadata V2/X2-2/X2-8, X3-2/X3-8, X4-2 Customer Actions: Exadata Rack Down (Doc ID 1604373.1),

    It says;

    Prior to field engineer service actions:

    - Shutdown the Exadata Database Severs
    - Shutdown the Exadata Storage Severs
    - Powering Off Oracle Exadata Rack

    So, yes..

  22. Virtual Column filtering by default is not offloaded unless you set the paraemter "_CELL_OFFLOAD_VIRTUAL_COLUMNS". So in esence , smart scan doesn't offload virtual column filtering. Reference Exadata Smart Scan FAQ (Doc ID 1927934.1).

    Also refer to note "Find the statistics related to cell offload (Doc ID 1438173.1)" which states
    "Full scan of a heap table will benefit from smart scan"


  23. Hi..Erman Arslan

    Do you Have referece step by step install SFP module on Oracle Exadata X3-2?? and how to configure on OS.

    Please your answer.


  24. Check this Oracle Support note:

    How to Replace a 10G Ethernet Card on an Exadata Database Node (X3-2/X4-2) (Doc ID 1672412.1)
    It is for replacing the ethernet card, but look what it says:
    Pull out any SFP modules installed in the original 10G Ethernet card and install them into the replacment 10G Ethernet card making sure to orient them properly so that they seat fully into the card.

    So it seems it is pretty straigth forward :)

  25. Thank you Erman

    I will Try it First

    Thank you For your Reply..... :)

  26. Thank you for posting very detailed information oracle exadatafor exadata database machine.