Exadata -- How to Connect Oracle Exadata to 10G Networks Using SFP Modules

In this post, I will share you the way of activating SFP modules (SFP modules on Ethernet cards) in an Exadata X3-2 Quarter Rack env.

As you may know, by default in Exadata X3-2 , we configure our public network using the bondeth0 bond interface. This bondeth0 is actually a virtual bonded device built on top of 2 physical interfaces. (eth1 and eth2) . The bonding mode of this bondeth0 is by default active-backup and as for the speed; it relies on the speed of the underlying eth1 and eth2.

In Exadata X3-2, eth1 and eth2 devices are actually Os interfaces for the underlying 1Gbit cards.
So, this means, by default we are limited to 1Gbit interfaces.

Luckly, Exadata X3-2 also supports 10Gbit SFP interfaces. (Fiber) . So if the Exadata that we are working with, has the necessary SFP modules, then we can configure our public/client network to run on these 10Gbit SFP modules, as well.

In order to activate these SFP modules, what we need to is;

1) The first step is to purchase the proper SFP+ and fiber cables to make the uplink connection.

2) Then we plan a time to reconfigure bondeth0 to use eth4 / eth5 and reboot.

So , it seems simple but it requires attention, since it requires OS admin skills.

Well. Here are the detailed steps;

First, we need to check the SFP modules and see the red light coming from them. (red light means fibre type link is up). Then , we connect the fibre cables to our SFP cards.

After that, we shutdown our databases running on this Exadata and we shutdown the Cluster services as well. We do all these operations using the admin network.. (we connect to Exa nodes using admin network interfaces, using relevant hostnames)

After shutting down the Oracle Stack, we shut down the bondeth0, eth1 and eth2 interfaces.

Then we delete ifcfg-eth1 and ifcfg-eth2 files (after taking backup ofcourse)

After deleting the eth1 and eth2 conf files, we configure the eth4 and eth5 devices.  We make them to be slaves of bondeth0.. (eth4 and eth5 are the OS interfaces for  these 10gbit Sfp cards in Exa X3-2 1/4) Note that: our public/client network ip configuration is stored in bondeth0, so we just modify its slaves, do not touch bondeth0 and the ip setting.

After the modifications, we start eth4, eth5 and bondeth0 (using ifup) and check their link status and their speeds using ethtool.

Once we confirm all the links are up, bonding is okay (cat /proc/net/bonding/bondeth0), we reboot the Exadata nodes and wait our cluster services be automatically up and running again ..

 So that's it :) 

Exadata -- Reimaging Oracle Exadata machines

Nowadays, I 'm mostly working on Exadata deployments.. These deployments I 'm mentioning, are machine deployments, including reimaging, upgrading the Image versions, the first deployments of new Exadata machines and so on.

I find it very enjoyable though, as these kinds of deployments make me recall my System Admin days when I was mounting the servers to the rack cabinets, installing Operating Systems, doing cabling , administrating the SANs and so on :)

Ofcourse, these new deployments, I mean these Exadata deployments are much more compilicated & complex than the server deployments that I was doing between the years 2006-2010.

Actually, the challanges we face during these deployments made this subjects more interesting and enjoyable , so that I can write blog posts about them :)

Today, I m writing about an Exadata X6-2 reimaging that we have done a few days ago.

The machine was an Exadata X6-2 1/8 HP and we needed to reimage it , as it was a secondhand machine used in lots of POCs..

We start the work by running OEDA.
You can read about it by the following link: http://ermanarslan.blogspot.com.tr/2017/07/exadata-initial-deployment-oeda-and.html

So once we get the OEDA ouputs that are required for imaging the Exadata machine, we continue by downloading the required files and lastly we follow the following sequence to reimage the machine..

• First, we connect to the Cisco switch using serial port (switch used for the Admin network) and configure it according to the OEDA output.

• Then connect to the Infiniband switches using serial ports and configure them according to the OEDA output.

• Next, we start up our virtual machine that we configured earlier for these deployments. This virtual machine gives us the ability to boot the Exa nodes using PXE. This Virtual machine has Dhcp, Pxe, TFTP and NFS services running on it.

• So, once started up, we connect our virtual machine to one of the ports that is available on Cisco Switch and we configure it with the IP of one of our PDUs.. (PDU ip address are available during the first deploy, so this move is safe) -- it is important to give the ip address according to the configuration that we done inside our virtual machine.. I mean our virtual machine has services running on top of it.. So if these services are configured on a static ip, then we use that ip for configuring our virtual machine itself.

• Next, we transfer the preconf.csv file , which is created by OEDA to our virtual machine and edit the MAC addresses written in this file according to the MAC adresses of our compute and cellnodes.  (MAC address of the nodes are written in the front panel of the nodes .)

• At this point, we connect to our compute and cell nodes using their ILOMs, and set their first boot devices to PXE. After this setting, we restart the nodes using ILOM  reset commands.

• When the machines are rebooted, they boot from the PXE devices and display the imaging menu, that our virtual machine serves them using PXE. In this menu, we display all the images which can be used for the imaging both the cell and compute nodes.

• Using this approach we image the compute and cell nodes in parallel.. I mean we connect to console of each node using their ILOMS, and select the relevant image from the menu and start the installation.

• Once the installation of nodes are completed, we connect the client network to our Exadata machines. (admin network - to the CISCO switch, client network - directly to the Compute Nodes)

• Lastly, just after imaging is finished, we continue by installing GRID and RDBMS software. In order to do this, we transfer, the GRID and RDBMS installation files + onecommand utility + OEDA outputs to the first compute node and then run the install.sh script which is included in onecommand to install our GRID and RDBMS. (As you may guess, one of the arguments for install.sh script is OEAD xml..)

That 's it.. Our Exadata is ready to use :)

EBS 12.2.7 - Oracle VM Virtual Appliance for Oracle E-Business Suite 12.2.7 is now available!

Oracle VM Virtual Appliance for E-Business Suite Release 12.2.7 is now available from the Oracle Software Delivery Cloud !!

You can use this appliance to create an Oracle E-Business Suite 12.2.7 Vision instance on a single virtual machine containing both the database tier and the application.

RDBMS -- diagnosing & solving "ORA-28750: unknown error" in UTL_HTTP - TLS communication

As you may remember, I wrote a blog post about this ORA-28750 before.. (in 2015).
http://ermanarslan.blogspot.com.tr/2015/09/rdbms-ora-28750-unkown-error-in-web.html

In that blog post, I was addressing this issue with the SHA2 certification lack, and as for the solution , I recommended upgrading the database for the fix .. (this was tested and worked)
I also recommended using a GeoTrustSSLCA-G3 type server side certificate for the workaround. (this was tested and worked)

Later on, last week ; we encountered this error in a 11.2.0.4 database and the server side certificate was GeoTrustSSLCA-G3 certificate.. The code was doing "UTL_HTTP.begin_request" and failing with ORA-28750.
So, the fix and the workaround that I documented earlier, were not applicable in this case.. (DB was up-to-date and the certificate was already GeoTrust..G3)..

As you may guess, this time, there was a more detailed diagnostic needed.

So we followed the note:

"How To Investigate And Troubleshoot SSL/TLS Issues on the DB Client SQLNet Layer (Doc ID 2238096.1)"

We took a tcpdump..  (with the related IP addresses to have a consolidated tcp output..)

Example: tcpdump -i em1 dst 10.10.10.10 -s0 -w /tmp/erman_tcpdump.log

In order to see the character strings properly, we opened the tcpdump's output using Wireshark. *

When we opened the output with Wireshark; we concantrated on the TLS V1.2 protocol type communication and we saw an ALERT just after the first HELLO message;

The problem was obvious.. TLS V1.2 communication was throwing Unsupported Exception error.

This error redirected us to the Document named:  UTL_HTTP : ORA-28750: unknown error | unsupported extension (Doc ID 2174046.1)

This document was basically saying "apply patch 23115139", however; this patch was not written for Oracle Database 11.2.0.4 running on Linux X86-64.. In addition to that, our PSU Version was 11.2.0.4.171017 and the patch was not for it.  

So we needed find another patch which includes the same fix and it was required to be appropriate for our DB & PSU Version..

Now look what we found :) ;

Patch 27194186: MERGE REQUEST ON TOP OF DATABASE PSU 11.2.0.4.171017 FOR BUGS 23115139 26963526

Well.. We applied patch 27194186 and our problem solved.

Now, by the help of this issue and its resolution; I can give 2 important messages; 

1) Use wireshark or a similar tool to analyze the tcpdump outputs.  (analyze the dumps by concantrating on TLS protocol messages)

2) Dont surrender even when the patch that is recommended by Oracle Documents, isn't compatible with your RDBMS and PSU versions.. 

Most of the time, you can find another patch (maybe merged), which is compatible with your RDBMS & PSU versions and that patch may include the same fix + more :)