Exadata -- Reimaging Oracle Exadata machines

Nowadays, I 'm mostly working on Exadata deployments.. These deployments I 'm mentioning, are machine deployments, including reimaging, upgrading the Image versions, the first deployments of new Exadata machines and so on.

I find it very enjoyable though, as these kinds of deployments make me recall my System Admin days when I was mounting the servers to the rack cabinets, installing Operating Systems, doing cabling , administrating the SANs and so on :)

Ofcourse, these new deployments, I mean these Exadata deployments are much more compilicated & complex than the server deployments that I was doing between the years 2006-2010.

Actually, the challanges we face during these deployments made this subjects more interesting and enjoyable , so that I can write blog posts about them :)

Today, I m writing about an Exadata X6-2 reimaging that we have done a few days ago.

The machine was an Exadata X6-2 1/8 HP and we needed to reimage it , as it was a secondhand machine used in lots of POCs..

We start the work by running OEDA.
You can read about it by the following link: http://ermanarslan.blogspot.com.tr/2017/07/exadata-initial-deployment-oeda-and.html

So once we get the OEDA ouputs that are required for imaging the Exadata machine, we continue by downloading the required files and lastly we follow the following sequence to reimage the machine..

• First, we connect to the Cisco switch using serial port (switch used for the Admin network) and configure it according to the OEDA output.

• Then connect to the Infiniband switches using serial ports and configure them according to the OEDA output.

• Next, we start up our virtual machine that we configured earlier for these deployments. This virtual machine gives us the ability to boot the Exa nodes using PXE. This Virtual machine has Dhcp, Pxe, TFTP and NFS services running on it.

• So, once started up, we connect our virtual machine to one of the ports that is available on Cisco Switch and we configure it with the IP of one of our PDUs.. (PDU ip address are available during the first deploy, so this move is safe) -- it is important to give the ip address according to the configuration that we done inside our virtual machine.. I mean our virtual machine has services running on top of it.. So if these services are configured on a static ip, then we use that ip for configuring our virtual machine itself.

• Next, we transfer the preconf.csv file , which is created by OEDA to our virtual machine and edit the MAC addresses written in this file according to the MAC adresses of our compute and cellnodes.  (MAC address of the nodes are written in the front panel of the nodes .)

• At this point, we connect to our compute and cell nodes using their ILOMs, and set their first boot devices to PXE. After this setting, we restart the nodes using ILOM  reset commands.

• When the machines are rebooted, they boot from the PXE devices and display the imaging menu, that our virtual machine serves them using PXE. In this menu, we display all the images which can be used for the imaging both the cell and compute nodes.

• Using this approach we image the compute and cell nodes in parallel.. I mean we connect to console of each node using their ILOMS, and select the relevant image from the menu and start the installation.

• Once the installation of nodes are completed, we connect the client network to our Exadata machines. (admin network - to the CISCO switch, client network - directly to the Compute Nodes)

• Lastly, just after imaging is finished, we continue by installing GRID and RDBMS software. In order to do this, we transfer, the GRID and RDBMS installation files + onecommand utility + OEDA outputs to the first compute node and then run the install.sh script which is included in onecommand to install our GRID and RDBMS. (As you may guess, one of the arguments for install.sh script is OEAD xml..)

That 's it.. Our Exadata is ready to use :)