Exadata X8M-2 & PCA X8-2 -- Part 4 -- Introducing Private Cloud Appliance (actually going a little further than that)

Remember my " Exadata X8M-2 & PCA X8-2 Migration series"? This is Part 4 and here I am, introducing PCA X8-2 to you. Yeah, I know PCA X9-2 just released, but! this machine is still new, as you don't see it in every day in the data centers .. at least in here you don't :)

As you may guess, we are continuing our journey on Exadata and PCA migration, and here it is -> the Private Cloud Appliance! Note that, we already migrated all the databases to Exadata X8M-2 , so from now on (and at least for a while), I will write mostly on the application migration related part of the story :)

This machine provides a high available, too fast, fully engineered and high tech platform for applications. An easy to deploy virtualized apps environment we are talking about here.. It is good for boosting the performance of the apps, getting rid of the single point of failures, and scaling out..This machine is the key for having a state-of-art platform in our data center for hosting our apps reliably.  

Even at the first sight, it looks like a perfect platform for hosting E-Business Application Tier, any kind of FMW product (including Forms and Reports), Oracle Business Ingelligence Application Tier and Clustered Weblogic environments! what do you say?

In this post, I will make a quick overview of the machine and the platform. Of course I will give try to give some insights when appropriate.. Note that; I may also play the role of devil's advocate when necessary :)

Alright.. Let's go!

The installation is simple.. You just fill out a form (for specifying the IP address, hostnames etc..), set a date and a hardware engineer from Oracle installs the system by doing a general health check and completing the easy installation process.

After the installation, you can directly login to the PCA Admin GUI using the standard url (<IP_address>:7002/dashboard).

PCA delivers a ready to use Oracle VM Server environment. 

I feel like I'm hearing those voices "Still Oracle VM Server"? Yes.. It is still okay! :)

But! in PCA X9, we have KVM.

PCA X9 announced -> https://blogs.oracle.com/infrastructure/post/oracle-private-cloud-appliance-x9-simplifies-developing-and-deploying-diverse-workloads

Check the Private Cloud Appliance Web Page as well -> https://www.oracle.com/engineered-systems/private-cloud-appliance/

The platform is fully virtualized and from the an admin point of view; it is not any different than a standard Oracle VM Server environment.. Oracle VM Manager installed and configured during the installation automatically.. So you(admins) just start using the virtualized environment by connecting to Oracle VM Manager. (<IP_address>:7002/ovm/console)

PCA Admin GUI  and Oracle VM Manager run on the active admin node, so that IP Addresses I mentioned above, is the IP address directed to the active admin node.. (Note that we have multiple admin/management nodes in this machine)

Note: OVM Manager works better with Firefox.. In our case, the version built-in OVM manager was 3.4.6.

We have also ZFS storage in this machine!

2 ZFS controllers, 1 shelf, 2 management nodes and 2 compute nodes in the base RACK!

Let's take a look at the virutalized OVM environment;

We have 1 server pool configured. We have 2 servers (compute nodes) in the pool and both of them have 96 cores and 382 GB memory.. (again for the Base Rack)

We have 3 repositories. One of them is named Rack1-Repository.. It is a shared repository, sized 3TB. The other two are local repositories, each of them is 1TB in size.. So these are the defaults.

Rack1-Repository is built on top of ZFS. Local repository are built on top of the local disks of the compute nodes.

Rack1-Repository is presented to both of the nodes at the same time, as it is a shared repository. Each local repository is presented to the related compute node.

We must not touch these repositories, so we leave them as is..

Note that, we may resize the Rack1-Repository if needed.. (check Oracle Support for the document that gives the instuctions for this.)

We don't have any ISOs or templates in the repositories by default.. So if we want to use some ISOs or templates, we need to follow the proper procedure that we generally follow in OVM environments.. (attention to the VM types.. HVM, PVM... In order to have PVM -- paravirtualized virtual machines, the procedure is a little different..)

The server management network is also the live migration network.. (we have live migration option in OVM, as you may already know)

We also have a vm network for cluster heart beat and storage related communicatons.

We have 2 more networks named default external and default internal.

We have bond interfaces in the compute nodes and bond1 is designed to be used for the VM traffic.

In order to manage the ZFS , we need to access the ZFS admin url. (example url : ZFS login url : https://<IP_Address>:215 .. We may also use ZFS command by accessing the ZFS storage using SSH.

The Rack1_Repository that I mentioned above is not sufficient for placing all the VM disks.. So as the first thing, we create a new LUN in the ZFS and present it to our Oracle VM Server environment and make use of it.

Note that, we created 2 repositories on 2 different LUNS on ZFS. One for the TEST environments, and one for the PROD.

[ PCA ] How to Increase Repository Capacity in the Oracle Private Cloud Appliance (Doc ID 1938558.1)

Once we finish our work with the ZFS, we use Oracle VM Manager to define a repository on the new Luns and then upload our ISO images and start/boot our virtual machines.

Note that, the boot disks of the virtual machines should be in the first place in the boot order.. (otherwise, virtual machine can not boot)

We move a bit randomly I know : ) but we will do okay:) 

Okay.. An important recommendation after all I have said so far, is the following;

*I recommend installing EM 13.4 (or higher version) for managing the PCA as a whole. EM can manage both the PCA itself, and the virtualized environment delivered with it. So we integrate PCA and the Oracle VM Server environment to EM. With this move, the administration becomes centralized and the quality of the administration increases.

https://www.oracle.com/a/ocom/docs/engineered-systems/pca-enterprisemanager-3399271.pdf 

* I don't recommend fully utilizing the ZFS storage.. I mean using the 70% of the total space is okay, but we should not exceed it, unless it is really necessary..

Okay.. What else?

ZFS is there for VM images and apps data. ZFS is in HA mode.. Multiple controllers are connected to a HA Shelf. In the standard config, we have 100TB usable space. This space can be extended to 3.3 Petabytes if necessary.(with additional cost of course).. For this extension, we can even choose to use Flash arrays.

There are 2 compute nodes (can be extended to 25 compute nodes) as I mentioned earlier and they are in virtualized mode.. These are the OVM hosts where our VM machines run. So, we have OVM as the OS of these nodes.

In each compute node we have;

2 Intel Cpus (Xeon 8620 24 core) and minimum 384 GB memory.. Note that, these are for Base PCA X8.

Memory in each mode can be extended to 1.5TB..

We have multi tenant option for isolating the virtualization tier, isolating it by having different servers  pool, different storage, network and compute etc.. (tenant groups)

The physical network is redundant. We have 2 networks and we have Cisco spine and leaf switches in each of them. In addition to that, we have another Cisco for the management network.

The internal communication is based on 100gbits ethernet. Things like vm heartbeat, storage communication, live migration can use these channels.

We have also ILOMs and PDUs as you may guess. We have a seperate management & maintanence network for connecting these utilities.

That 100gbit network goes through the leaf and spine switches.

Each compute node connects to 2 leaf switches and each leaf connects to the spine in a redundant manner. Each spine connects to the ports that goes to the storage, magement nodes and customer network.

In spine switches, we have 4 ports for the customer.. (port 1-4) These are reserved to be used for the customer's own network requirements. Port 5 (5/1 and 5/2) is the uplink port.  Port 1-4 can be configured with a 1x 100GBit or1x 40 GBit or 4x25 GBit or 4x10Gbit. Port 5 is 10Gbit.

PCA connects to the outside world using the TOR (Top of Rack) switches.

PCA's internal network (application management, storage access, inter vm communication) uses predefined IP addresses. But, as these ip address are not related with the external networks, they do not interfere.

While each RACK component has 1 IP address; each Storage, management and compute nodes have 1 additional IP addresses for ILOM.

Compute nodes connect to both the internal network and customer's data center network. By default, we don't have assigned IP addresses for connecting to the customer network in compute nodes.. (for the isolation), but we can configure compute nodes and assign them IP addresses from the customer network if we want to.(for instance we want to mount a NFS to the compute nodes.. a NFS hosted by machine in customer network..)

We need to be sure that the subnets given below should not be used in the customer datacenter network in order to prevent conflicts;

192.168.4.0/24 – internal machine administration network: connects ILOMs and physical hosts

192.168.32.0/21 – internal management network: traffic between management and compute nodes

192.168.64.0/21 – underlay network for east/west traffic within the appliance environment

192.168.72.0/21 – underlay network for north/south traffic, enabling external connectivity

192.168.40.0/21 – storage network: traffic between the servers and the ZFS storage appliance

We also need to pay attention to the VLAN settings. It is recommended to avoid the use of the following VLANs outside;

1 – the Cisco default VLAN

3040 – the default service VLAN

3041-3072 – a range of 31 VLANs reserved for customer VM and host networks

3073-3099 – a range reserved for system-level connectivity

VLANs 3090-3093 are already in use for tagged traffic over the /21 subnets listed above.

3968-4095 – a range reserved for Cisco internal device allocation

*Virtual machines can be configured to connect to the the ouside network via the default_external network that comes built-in with the PCA's OVM.. VLANs can also be defined on this network. (actually VLANs can be configured on the internal network as well..) The default_internal network can be used for the private interconnect network between the VMs hosted on PCA's OVM.

Well.. We have different networks in PCA;

- Administration network: Communication via Cisco Nexus 9348GC-FXP Switch. This is for Management. The IP addresses are on 192.168.4.0/24. All the nodes have management IP addresses here. In addition, ILOM IP addresses are also in this range.

PCA Admin GUI and VM manager also serve from this network. Spine switches are used when accessing them from the outside. (5/1 and 5/2 ports)

- Data network: All data flows through Cisco Nexus 9336C-FX2 Switches. They are in leaf-spine design.  Leafs are like the bridges between the RACK components and Spines, on the other hand, act as backbones and do the routing work.

We have bond devices named bond1 on the compute nodes. These  are connected to the Leaf switches. They are configured in link aggregation mode.

What else? Ohh okay -> VLAN for the management network...

If the management network should be on a VLAN, it can also be set in the PCA Admin GUI -> "The management node external network settings are configurable through the Network Settings tab in the Oracle Private Cloud Appliance Dashboard. "If this network is a VLAN, its ID or tag must be configured in the Network Setup tab of the Dashboard."

But! No VLAN for VM traffic or PCA management by default! -> "The default configuration of the PCA places both the default networks for VM traffic and PCA Management traffic on the physical ports 5/1 and 5/2 of the two spine switches, with "no overlying VLANs" to segregate the traffic.."

Here is the list of recommendation in different network configurations for PCA -> 

1.Ports 5/1 and 5/2 are used for unsegregated VM and Management traffic. No VLANs are assigned to either. Not Recommended -> ACTUALLY THIS ONE IS NOT RECOMMENDED.

2. Ports 5/1 and 5/2 are used for segregated VM and management traffic by assigning a single VLAN for management traffic through the dashboard. 

3. Ports 5/1 and 5/2 are used for segregated VM and management traffic by assigning one or more VLANs for VM client traffic on top of the default_external network through OVM Manager

4. Ports 5/1 and 5/2 are used for segregated VM and management traffic by assigning both a single VLAN for management traffic through the dashboard and assigning one or more VLANs for VM client traffic on top of the default_external network through OVM Manager. --> THIS ONE IS THE BEST PRACTICE.

5. Ports 5/1 and 5/2 are used for management traffic only without an VLAN. All customer network traffic is through a new custom network, with or without VLANs on a port group residing on one or more of the physical ports 1-4. Thedefault_external network is unused. .

6. Ports 5/1 and 5/2 are used for management traffic only with a VLAN assigned through the dashboard. All customer network traffic is through a new custom network, with or without VLANs on a port group residing on one or more of the physical ports 1-4. The default_external network is unused. THIS LAST ONE IS ALSO ONE OF THE BEST PRACTICES.

--So as you see in the 6th recommendation, customet networks can also be created and customer network traffic can also be configured to be on that network.. (Note that we are OKAY with the 4th recommendation...)

There are too much to say, but I think that, reading following document is a must ->

https://www.oracle.com/a/tech/docs/networking-the-oracle-pca-and-pcc.pdf

Networking the Oracle Private Cloud Appliance

and Oracle Private Cloud at Customer. A guide to connecting Oracle Private Cloud X8-2 solutions into the datacenter.

The tricky part is the network.. The network admins should configure their switchs according to what we expect..

In the document above, there are some example configuration and commands for the network side.. These examples are given for the Cisco and Juniper switches.. I mean for the customer side switches..

I m sharing the one for the Cisco as follows;

interface port-channel221

 switchport mode trunk

 switchport trunk native vlan 313

 spanning-tree port type normal

 no negotiate auto

interface Ethernet1/19

 description "Rack22 9336 RU22 P5"

 channel-group 221 mode active

interface Ethernet1/20

 description "Rack22 9336 RU22 P5"

 channel-group 221 mode active

interface Ethernet1/21

 description "Rack22 9336 RU23 P5"

 channel-group 221 mode active

interface Ethernet1/22

 description "Rack22 9336 RU23 P5"

 channel-group 221 mode active

Well, this is for Cisco but your network admins  will see this and check their side accordingly.. We have tested the same config with Huawei and Huawei switches can work with PCA as well..

We defined VLANs and made Virtual machines use those VLANs without any problems.. But! the network admin had some throubles configuring their side of switches and this example commands were the cure. 

Network admin checked this, and corrected his side accordingly..

The topologies given in the page 11,Physical Connection to the Data Center Network section of this whitepaper, are also important. There, in those topologies you see the switches, ports and LACP config in a general picture.. So these should also be reviewed..

Note that, we don't need to do anything extra in the PCA switches.. We don't need to do connect them and do some customizations to use VLANs. All the things need to be done in the customer switches. So keep that in mind!

Almost forgot! We have a cloud at customer version of PCA , which we call PCC as well.. 

Customers feel themselves closer to cloud (or have some regulations), can use this one.. 

EXACC / ECC customers already know the advantages of cloud at customer model :)

What else.. Aha ! I have actually gone through all the things that I wanted to share. So, this is a perfect place to end.

Thanks for reading :)