Recently I have taken a look at the Delphix software and its data as a service solution, as it was one of the prerequisites of our new delphix partner enablement process. I must admit that, I find it quite handy.
When we talk about Delphix, we actually talk about the Delphix engine. That is; there is an engine called Delphix engine, which runs on a Virtual machine based on a Solaris Guest OS and it sits between hosts and the storage for enabling fast clones, rewinds, restores, masking, versioning and provisioining of the non-prod enviroments.
Delphix supports a bunch of well known databases such as Oracle and Microsoft Sql Server and as far as I can see it works smoothly. In addition, it supports virtualizing the filesystems(such as Linux ext3 and ext4) and take fast filesystem clones from the sources to targets.
Using Delphix, we can create our clone environments in just 1-2 minutes. Also we can rewind them, restore them, take another clones from them easly and quickly. We can mask the data while creating those clones and we can even offload the rewind and cloning operations to our development teams by giving them the sophisticated Delphix tool called Jetstream.
In this post, I will give you my notes about the Delphix and this will be just a beginning, as we will dive in to this software deeply in the next months.
Before giving you the notes, I want to give you a quick and a -from surfrace- review about how Delphix works in Oracle environments. I want to give the overview about Oracle, as it is our main focus and I want to give it short because as i already mentioned it is just a beginning;
Delphix is installed on the site where our clone environments supposed to be.(recommended)
It is installed on a Virtual machine. This virtual machine is the platform where Delphix engine actually runs.
Delphix engine is responsible for the Delphix operations and it can be invoked using Delphix tools, CLIs and APIS.
Delphix engine has all access to the underlying storage via the Virtual Server platform that it runs on.
For cloning an Oracle Database, Delphix connects to the production machine and triggers and rman level 0 backup there. However, this level 0 is only taken for one time, for the inialization. After the initialization is completed, Delphix regulary takes level 1 rman backups from the production and store them in its storage in a compressed format. These rman backups are called Delphix Snapshots by the way. (note that, there are 2 snapshot terms in Delphix, 1) rman backups 2) the snapshots that Delphix creates its inside, while we are using the jetstream)
Delphix , if required, can also regularly take the redologs + archivelogs from the production. This is called the log sync and it is done in a time basis way for having a more point in time based clones in the target site..
When we initialized our source environment in Delphix, we can take easy clones by just clicking a few buttons located in the delphix engine Web UI. When we clone a database, which is called "creating VDB operation", Delphix restores the database backup that it took in the first place and roll it forward to the point in time that we want to have our clone database to be. Delphix does these operations transparently and when we look at our target machine, we see our new clone database is running on our target environment and its database files are stored in NFS mounts which are coming from Delphix engine.
So, ofcourse there are a lots of things that Delphix can does as well. There are things that Delphix enables too. Imagine a site migration for example, or imagine a cloud migration. Imagine EBS database to be cloned in just a few miniutes and masked, imagine EBS apps tier can be cloned in just a few minutes. Well, lots of things to say, lots of things to remember, lots of procedures to follow and also lots of expertise to have.
I will give you much deeper details in my next blog posts, but for now, I give you the notes;
For what can we use Delphix:
Application developments: Create a QA environment for Developers. Create restore points(bookmarks), switch between those bookmarks with just a couple of clicks.
Refresh/Restore: Replace from Production/Referesh from Production.
Refesh from branches/bookmarks. It can mask the data from PROD when it is restored to DEV.
Creating new version of applications: By creating Branches/bookmarks, we can create a new timeline for the new version without deleting the old version data or applicaiton.
Data Masking: real data is replaced with fake data which is still realistic. Delphix delivers secure data to development environments.(secure virtual copies are created from the production) -- masking done is approx 4 hours..Deplhix masks the data without breaking the referential integrity-> for ex: Mary is always masked as Clara... in every table, every column...
Test: Test, fail, then go back and restest...
Migration: Place delphix to the target site, create clone environments in the target without copying lots of data. (done in Clorox Company, probably they had lots of clones in the source environment)
Delphix Tools:
JetStream is the Delphix UI console for the use of End Users to control Data Copies. (We give this to DEV and TEST developer, so they use it their own) Developers do the rollback and rollforward for their databases.
DEV and TEST developers creates their copies without touching the infrastructure.(no risk there)
Operation Console(Ops console) : is the tool used by Deplhix admins. The things that are done here are; provision,convert, replicate,mask, role based user privileges, policy management and more.
Security console:
Delphix finds the sensitive data by profiling.
Secure sensitive data by masking.
Audit the sensitive data and ensure it is masked. Alert admins if there are vulnerabilities.
Missing Control Management Console: Used for Reporting,monitoring health, enabling auditing for compliance(user based audit,limited data views and so on)
Delphix captures the production data (initialization), then it records the changes continously and lastly, creates the point it time virtual copies on demand.(delphix shares the data blocks for building virtual copies, rather than copying them)
3 test environment runs in parallel on shared data blocks. no copies are needed for building a new/4th environment.
These test environments can be refreshed , rewinded fastly.
References:
Facebook used it. Facebook financials are on EBS. They reduced their Quarterly Financial close from 3 weeks to 2 days. They use delphix for doing parallel environments so that analysts could find the transactional errors before doing the close on production.
Delphix answers "YES" to the following questions:
Create a new fresh copy of 5 TB DB in 5 mins?
Reset Database copy just before descrutive change without losing all of my work?
can you provide 3,5,10 copies of my database?
Can you provide securely masked copies in minutes?
Can I self service my data copies and version control them like source code?
Delphix generally work as -> Take data , compress it in Delphix engine, virtualize and connect the data to any kind of server. (data can be an Oracle DAtabase, or EBS application files, or can be gathered from any data source)
From One physical data, Delphix creates several virtual TEST, DEV etc environments --> %90 decrease in redundant data
Gains both in OPEX (Operation expenditure, management man/hours) and CAPEX(CApital Expenditure, less storage)
In DEV environment, the errors can not be found. Most of the time, the errors are found QA and UAT environments (too late..thus it increases the release time) This is because DEV environments are not fresh, not full, not synch. With delphix, the errors are found in DEV, less work for errors in UAT and QA, which is good.
So the development becomes more efficient.Thus, when the time of production , there are less errors to deal with.
How Delphix enables its capabilities?
Deplhix(Deplhix engine) sits between RDBMS and Storage. Deplhix runs on any hypervisor and on any server.
Deplhix doesnt touch production at all. It just takes data from there.
Delphix take the copy and compress it . Also record the changes from source. So it tracks all the deltas coming from the source. Delphix masks the data if wanted(mask once, use many times, once masked several environments can be created without need to mask once again). Thus, deplhix allows users to provision any point in time.
Gains & Some References & Some Success stories
Deutsche Bank -> Financial stress testing.. There were various reporting jobs which needed to be distributed. With delphix Deutsche Bank, they doubled application development outputs, virtual environments created, offload reporting from prod to these environments. Data management turned into self service.
Coherent -> Required to do the Critical upgrade of EBS 11i to R12 + migration of this system to a new data center. They used Deplhix to accelerate the upgrade and migration. Using delphix they proveed the ability to archive and the ability to recover the database efficiently.(in case needed)
Molina Healthcare-> personel healthcare information(sensitive) should be secured in Dev, QA, Reporting, Staging envs and so on. With delphix, Molina built lots of virtual test environments. (over 3000) They avoided 3Petabytes of Storage capacity. They also used efficient Data masking in deplhix in a way like "mask once", provision many.
Clorox -> increase the migration speed. They parallelized testing, used self service virtual environments for their migration related works.
Cisco -> on demand data access across all region. They increased their testing cycle speed. They also used Deplhix for migration.
US Marines -> They nneded to Oracle EBS 11i to R12, in parallel a migration to a new site was planned. They used Deplhix 2x acceleration in ERP upgrade. They also used it in the retirement of their legacy data centers.
Facebook -> Increased the speed of Financial close from 3 weeks to 2 days.(as a result of increased error resolution and proactivity.)
Wallmart -> Accelerate release cycles, reduced online errors by cutting frequent refresh cycles (from 6 weeks to 2 days). Eliminated operational and capital cost of having multiple environments by %80.
StubHubb -> Accelerated Application releases. Normally, it was taking 3 weeks to create a testing environment. From monthly to daily mobile application releases. They reduced production errors by %20.
Gap -> Used fresh copies for offloading the ETL access and increased the speed the daily and weekly reports. They offloaded all BI and Reporting from Prod to virtual copies.
Informatica -> With Deplhix, accelerated its master data management project 6 months ahead of its schedule.
Wind River -> They used Deplhix several virtual environments for continous and up-to-date access.. Intel could take the data from these virtual environments. (Intel purchased Wind River, so needed to process its data). Generally, production access is decreased from 8 hours to 15 minutes
Carnival -> Their main challange was Data Protection. They used Delphix to create low cost backup and disaster recovery solution. Also they gained 10x Storage and 2X Server utilization improvement.
P&G -> They used Delphix for application development, delivering quick copies, data protection. They also improved their RTO and RPO. They get superior SLAs.
Comcast -> They needed Much faster point in time recovery and they used Delphix for this, as Deplhix provides quick access to the data in any point of time. They increased the speed of their recovery also. Delphix -> Agility & Availability for Comcast
One of the business usecase of Delphix is moving data between site securely, such as "Delivering secure data from on-premise to cloud".
Using Deplhix, the data can be securely transffered between delphix engines. So Delphix can do engine to engine replication. This replication is done with the compression , so it is very quick.
Using Delphix bookmarks and branches several versions application can be coexist and developed in parallel.
We can create a new version , work on it, then create a bookmark to save our work. After that we can switch the old version's branch and work on it, save our work by creating a bookmark etc...
We can create branches from bookmarks, as well. For example: for version 1.1 -> to 1.1.1 : we create a branch from version's 1.1 's bookmark.
So we restore, we go forward-backward in time, but our data in any data & time is not deleted unless we want it to be deleted.
Bookmarks can be deleted according to the retention policies, automatically.
We do all the branch-bookmark things using jetstream tool.
With Delphix migrataions become : virtualize and migrate. Delphix even provides Linux to Unix endian conversions for Oracle Databases.
Accessing Delphix Information/the sources:
Support Portal: support.delphix.com (download delphix software, knowledge base, public documentation, community forum and more)
Public Documentation: docs.delphix.com (most current documentation, all versions are available)
Shart Tank: Delphix user community, a forum community.delphix.com
Virtual Databases are created using Deplhix Engine graphical interface.
-Deplhix has Web Server GUI, CLI through SSH and APIs for administration. Two user roles are there for administration in Delphix: Delphix Admin(the real admin) and Sysadmin(a lightweight admin who can do the day to day tasks).
Delphix can virtualize Oracle, DB2, postgres, Mysql, Sybase ASE databases.
DELPHIX FOR ORACLE:
---------------------------
Delphix engine is installed as a VM guest on VMware VSphere environment(Version 4.x or 5.x) or Amazon Ec2 Cloud . It requires minimum 8vcpus , 64 gb memory minimum.
Delphix supports Oracle Databases 9.2.0.8, 10.2, 11.1, 11.2 and 12.1
Delphix supports Oracle Databases on Solaris(Sparc and X86), RHEL, OEL, Suse, AIX and HPUX.
There is no supported storage list for Delphix , so it can run on any storage.
Delphix supports both standalone and RAC hosts.
In Delphix, target environment means Virtual Databases.
In Delphix, the delphix-side representation of a Source database is called as Dsource (Data Source).
Source environment speaks with Delphix using SSH, JDBC and DSP. DSP(Delphix Session Protocol) runs on port 8415 . Delpix requires port 8415 to be opened between Source and Delphix systems.
Basically, what Delphix does is , it takes the Dsource and creates a VDB from it and then provision it to a target system as a Read Write fully functional Oracle Database.
spfile is also provisioned by Delphix while create VDBs.
Flow of data for Oracle:
Rman takes Rman Level 0 backup using Rman APIs for the initalization. (this is done once while adding the source system in to Delphix).
Delphix takes that level 0 backup in to the Delphix system for the initalization and then takes level 1 and optionally archive/redo backups to synch itself with the source.
After the level 0 is taken, Delphix compress it 2 to 4 times. Level 1 backups and archivelog backups used to synch the Delphix are also compresses in the same way.
The VDBs are presented to the target servers using NFS or DNFS (Direct NFS). --> Direct NFS has direct IO (no OS cache) and aysnc IO capabilities and it is the power of it. (when compared to NFS)
The changes done in the VDBS are stored in the Delphix Storage Layer Server , compressed.
the VDBS are like the snapshots of Dsources, so no extra storage is needed for creating them, and so that they are very fast to create.
The changes in VDBS are stored in the Delphix Storage layer (like snapshot deltas)
Dsources have a timeflow. We can see the snapshots in those timeflow. The snapshot are actually the level1 backups taken in to the Delphix. (Delphix iniates level1 backups by sending commands to the source environment)
This snapshots, in other words; level1 backups are done via the Delphix Snap Sync service.
Alternatively, the snapshots can be done by collecting the archivelogs and redologs from the source system. This is done via the Delphix Log Sync service.
IO is done : from target server -> Delphix -> Storage.
Deplhix can take data from ASM but can not provision to ASM. (In this scope, Exadata can be a source or target, but Delphix can not provision to Exadata Storage, but it can provision the data to Exadata as an NFS mount point)
VDBs can be created from other VDBS as well.
VDBs can be rolled back or rewinded point in time, as they have also their own timeflow. This rewinds takes place purely in Delphix's Storage Layer (No oracle technologies are used for that)
VDBs can be refreshed from production as well. (from the dsource and from the snaphsots taken for that dsource)
The IO performance can be monitored and analyzed using Delphix 's Network Performance and Performance Analytics pages.
DELPHIX FOR MSSQLSERVER:
---------------------------
The requirements for the Delphix engine is the same as Oracle .
Delphix supports SQL Server 2005, 2008, 2008R2, 2012, 2014 and Windows Server 2003 SP2, 2008, 2008R2, 2012 and 2012R2 (64 bit)
Delphix supports Always on(for only source) + Windows Failover clusters(for both source and target)
When we talk about the database term in Delphix means Data files + Transaction logs + transaction log backups for Sql Server.
As for the Sql Server, Delphix recreates the source system in an intermediate system called the validated sync target/environment.
Validated sync environment is a replica, which is recovered using the database file backups and log backups from the production server via the SMB protocol.
Validated sync environment can be a target server or a standalone sqlserver.
The reason for having this environment is to take the backups from source sqlserver contiously.
When a dsource is created, Delphix kick offs a job to have the validated sync server to take the backups from the source server.
Validated Sync should be in the same Sql Server version as the source.
There is a database created in the validated sync environment to be recovered by the source continously. (this databases is called the staging database)
there should be 1 Validated Sync should be for each Delphix engine. (one to one) . This is recommended.
So, Delhix itself does not consume any backup files.
Once the initilaziation is completed, Delphix remains in sync in terms of differential backups and transaction log backups.
Delphix talks with the source instance using jdbc and see if there are new transaction logs or backup have been taken. Then Delhpix instructs the validated sync database to apply them.
As for the sqlserver (like Oracle), Delphix provides network based storage services for the target servers (ISCI protocol)
Again, snap sync service takes backups from the source database.
Log sync service can also be used for retrieving the transaction logs.
The source Sql Server database must be Full recovery mode.
Without log sync, you can't restore to any time you want. So, with snap sync, you can only recover the time that are associated with backups.(you can only recover to snapshots)
Note: Backups taken from the source automatically creates a snapshot in Delphix.
Key factors on Installation of Delphix:
With IOCollector.sql , performance test are done.
We run it in PROD. Then we run it in current CLONE environment and see the difference.
Then, we will take the CLONE outputs as a performance goal, because we are producing clones(so no need to compare Delphix with the Production environment performance)
We expect most of the CLONE performance to be %50 of PROD performance.
If there are lots of clone environments needed, then multiple engine can be implemented. (this decision will be based on the outputs collected in the analysis phase/sizing phase)
For analyisis and sizing , 3 things matter.
1)CPU,RAM for Delphix Engine
2)Network between Source and Delphix + Delphix and Target
3)IO for Storage
Installation and POV should be done by following the document: Delphix Pre Install User Guide
Jumbo Frames can be enabled between the Source and Deplhix Engine and Target. Jumbo Frames will increase the network performance (because we do DB IO , with big block sizes)
However, if the network between Source,Delphix and Target is not isolated, then all the network will be affected from that.
Jumbo frames will be enabled for all the hosts in the network and thay may harm them.. So be careful and analyze the environment of the customer.
Delphix should be installed closer to the target environment (not the source)
Delhpix Storage should be created on top of one to one relationship. 1 lun should be connected for 1 data store and for 1 vmdk....
Raid configuration is a concern for Delphix. Delhpix can only give recommendations... It does not matter for Delphix as long as it performs well.
VMDKs should be "thick" provisioned zeroed. (not thin provisioning) Thin provisioning can be used for this VMDKs as well.
Landshark environment is the demo environment for Delphix. It consists of 1 Delphix Engine, 1 target and 1 source systems.
The network for installing Landshark is very important. The documentation should be followed for configuring the network of the virtual machines, before starting the Landshark environment.
Landshark when configured, comes as a preconfigured inventory. So that, it automatically sees the delphix engine in it sees its source and its target automatically.
With just 1-2 click, we can create our first VDBs in our target. Also, we can create our Vfiles as well.
Vfiles is used when there is need to create a target filesytem from a source filesystem.
Cloning EBS apps tier using Delphix is an example where we use vfiles.
When we talk about Delphix, we actually talk about the Delphix engine. That is; there is an engine called Delphix engine, which runs on a Virtual machine based on a Solaris Guest OS and it sits between hosts and the storage for enabling fast clones, rewinds, restores, masking, versioning and provisioining of the non-prod enviroments.
Delphix supports a bunch of well known databases such as Oracle and Microsoft Sql Server and as far as I can see it works smoothly. In addition, it supports virtualizing the filesystems(such as Linux ext3 and ext4) and take fast filesystem clones from the sources to targets.
Using Delphix, we can create our clone environments in just 1-2 minutes. Also we can rewind them, restore them, take another clones from them easly and quickly. We can mask the data while creating those clones and we can even offload the rewind and cloning operations to our development teams by giving them the sophisticated Delphix tool called Jetstream.
In this post, I will give you my notes about the Delphix and this will be just a beginning, as we will dive in to this software deeply in the next months.
Before giving you the notes, I want to give you a quick and a -from surfrace- review about how Delphix works in Oracle environments. I want to give the overview about Oracle, as it is our main focus and I want to give it short because as i already mentioned it is just a beginning;
Delphix is installed on the site where our clone environments supposed to be.(recommended)
It is installed on a Virtual machine. This virtual machine is the platform where Delphix engine actually runs.
Delphix engine is responsible for the Delphix operations and it can be invoked using Delphix tools, CLIs and APIS.
Delphix engine has all access to the underlying storage via the Virtual Server platform that it runs on.
For cloning an Oracle Database, Delphix connects to the production machine and triggers and rman level 0 backup there. However, this level 0 is only taken for one time, for the inialization. After the initialization is completed, Delphix regulary takes level 1 rman backups from the production and store them in its storage in a compressed format. These rman backups are called Delphix Snapshots by the way. (note that, there are 2 snapshot terms in Delphix, 1) rman backups 2) the snapshots that Delphix creates its inside, while we are using the jetstream)
Delphix , if required, can also regularly take the redologs + archivelogs from the production. This is called the log sync and it is done in a time basis way for having a more point in time based clones in the target site..
When we initialized our source environment in Delphix, we can take easy clones by just clicking a few buttons located in the delphix engine Web UI. When we clone a database, which is called "creating VDB operation", Delphix restores the database backup that it took in the first place and roll it forward to the point in time that we want to have our clone database to be. Delphix does these operations transparently and when we look at our target machine, we see our new clone database is running on our target environment and its database files are stored in NFS mounts which are coming from Delphix engine.
So, ofcourse there are a lots of things that Delphix can does as well. There are things that Delphix enables too. Imagine a site migration for example, or imagine a cloud migration. Imagine EBS database to be cloned in just a few miniutes and masked, imagine EBS apps tier can be cloned in just a few minutes. Well, lots of things to say, lots of things to remember, lots of procedures to follow and also lots of expertise to have.
I will give you much deeper details in my next blog posts, but for now, I give you the notes;
For what can we use Delphix:
Application developments: Create a QA environment for Developers. Create restore points(bookmarks), switch between those bookmarks with just a couple of clicks.
Refresh/Restore: Replace from Production/Referesh from Production.
Refesh from branches/bookmarks. It can mask the data from PROD when it is restored to DEV.
Creating new version of applications: By creating Branches/bookmarks, we can create a new timeline for the new version without deleting the old version data or applicaiton.
Data Masking: real data is replaced with fake data which is still realistic. Delphix delivers secure data to development environments.(secure virtual copies are created from the production) -- masking done is approx 4 hours..Deplhix masks the data without breaking the referential integrity-> for ex: Mary is always masked as Clara... in every table, every column...
Test: Test, fail, then go back and restest...
Migration: Place delphix to the target site, create clone environments in the target without copying lots of data. (done in Clorox Company, probably they had lots of clones in the source environment)
Delphix Tools:
JetStream is the Delphix UI console for the use of End Users to control Data Copies. (We give this to DEV and TEST developer, so they use it their own) Developers do the rollback and rollforward for their databases.
DEV and TEST developers creates their copies without touching the infrastructure.(no risk there)
Operation Console(Ops console) : is the tool used by Deplhix admins. The things that are done here are; provision,convert, replicate,mask, role based user privileges, policy management and more.
Security console:
Delphix finds the sensitive data by profiling.
Secure sensitive data by masking.
Audit the sensitive data and ensure it is masked. Alert admins if there are vulnerabilities.
Missing Control Management Console: Used for Reporting,monitoring health, enabling auditing for compliance(user based audit,limited data views and so on)
Delphix captures the production data (initialization), then it records the changes continously and lastly, creates the point it time virtual copies on demand.(delphix shares the data blocks for building virtual copies, rather than copying them)
3 test environment runs in parallel on shared data blocks. no copies are needed for building a new/4th environment.
These test environments can be refreshed , rewinded fastly.
References:
Facebook used it. Facebook financials are on EBS. They reduced their Quarterly Financial close from 3 weeks to 2 days. They use delphix for doing parallel environments so that analysts could find the transactional errors before doing the close on production.
Delphix answers "YES" to the following questions:
Create a new fresh copy of 5 TB DB in 5 mins?
Reset Database copy just before descrutive change without losing all of my work?
can you provide 3,5,10 copies of my database?
Can you provide securely masked copies in minutes?
Can I self service my data copies and version control them like source code?
Delphix generally work as -> Take data , compress it in Delphix engine, virtualize and connect the data to any kind of server. (data can be an Oracle DAtabase, or EBS application files, or can be gathered from any data source)
From One physical data, Delphix creates several virtual TEST, DEV etc environments --> %90 decrease in redundant data
Gains both in OPEX (Operation expenditure, management man/hours) and CAPEX(CApital Expenditure, less storage)
In DEV environment, the errors can not be found. Most of the time, the errors are found QA and UAT environments (too late..thus it increases the release time) This is because DEV environments are not fresh, not full, not synch. With delphix, the errors are found in DEV, less work for errors in UAT and QA, which is good.
So the development becomes more efficient.Thus, when the time of production , there are less errors to deal with.
How Delphix enables its capabilities?
Deplhix(Deplhix engine) sits between RDBMS and Storage. Deplhix runs on any hypervisor and on any server.
Deplhix doesnt touch production at all. It just takes data from there.
Delphix take the copy and compress it . Also record the changes from source. So it tracks all the deltas coming from the source. Delphix masks the data if wanted(mask once, use many times, once masked several environments can be created without need to mask once again). Thus, deplhix allows users to provision any point in time.
Gains & Some References & Some Success stories
Deutsche Bank -> Financial stress testing.. There were various reporting jobs which needed to be distributed. With delphix Deutsche Bank, they doubled application development outputs, virtual environments created, offload reporting from prod to these environments. Data management turned into self service.
Coherent -> Required to do the Critical upgrade of EBS 11i to R12 + migration of this system to a new data center. They used Deplhix to accelerate the upgrade and migration. Using delphix they proveed the ability to archive and the ability to recover the database efficiently.(in case needed)
Molina Healthcare-> personel healthcare information(sensitive) should be secured in Dev, QA, Reporting, Staging envs and so on. With delphix, Molina built lots of virtual test environments. (over 3000) They avoided 3Petabytes of Storage capacity. They also used efficient Data masking in deplhix in a way like "mask once", provision many.
Clorox -> increase the migration speed. They parallelized testing, used self service virtual environments for their migration related works.
Cisco -> on demand data access across all region. They increased their testing cycle speed. They also used Deplhix for migration.
US Marines -> They nneded to Oracle EBS 11i to R12, in parallel a migration to a new site was planned. They used Deplhix 2x acceleration in ERP upgrade. They also used it in the retirement of their legacy data centers.
Facebook -> Increased the speed of Financial close from 3 weeks to 2 days.(as a result of increased error resolution and proactivity.)
Wallmart -> Accelerate release cycles, reduced online errors by cutting frequent refresh cycles (from 6 weeks to 2 days). Eliminated operational and capital cost of having multiple environments by %80.
StubHubb -> Accelerated Application releases. Normally, it was taking 3 weeks to create a testing environment. From monthly to daily mobile application releases. They reduced production errors by %20.
Gap -> Used fresh copies for offloading the ETL access and increased the speed the daily and weekly reports. They offloaded all BI and Reporting from Prod to virtual copies.
Informatica -> With Deplhix, accelerated its master data management project 6 months ahead of its schedule.
Wind River -> They used Deplhix several virtual environments for continous and up-to-date access.. Intel could take the data from these virtual environments. (Intel purchased Wind River, so needed to process its data). Generally, production access is decreased from 8 hours to 15 minutes
Carnival -> Their main challange was Data Protection. They used Delphix to create low cost backup and disaster recovery solution. Also they gained 10x Storage and 2X Server utilization improvement.
P&G -> They used Delphix for application development, delivering quick copies, data protection. They also improved their RTO and RPO. They get superior SLAs.
Comcast -> They needed Much faster point in time recovery and they used Delphix for this, as Deplhix provides quick access to the data in any point of time. They increased the speed of their recovery also. Delphix -> Agility & Availability for Comcast
One of the business usecase of Delphix is moving data between site securely, such as "Delivering secure data from on-premise to cloud".
Using Deplhix, the data can be securely transffered between delphix engines. So Delphix can do engine to engine replication. This replication is done with the compression , so it is very quick.
Using Delphix bookmarks and branches several versions application can be coexist and developed in parallel.
We can create a new version , work on it, then create a bookmark to save our work. After that we can switch the old version's branch and work on it, save our work by creating a bookmark etc...
We can create branches from bookmarks, as well. For example: for version 1.1 -> to 1.1.1 : we create a branch from version's 1.1 's bookmark.
So we restore, we go forward-backward in time, but our data in any data & time is not deleted unless we want it to be deleted.
Bookmarks can be deleted according to the retention policies, automatically.
We do all the branch-bookmark things using jetstream tool.
With Delphix migrataions become : virtualize and migrate. Delphix even provides Linux to Unix endian conversions for Oracle Databases.
Accessing Delphix Information/the sources:
Support Portal: support.delphix.com (download delphix software, knowledge base, public documentation, community forum and more)
Public Documentation: docs.delphix.com (most current documentation, all versions are available)
Shart Tank: Delphix user community, a forum community.delphix.com
Virtual Databases are created using Deplhix Engine graphical interface.
-Deplhix has Web Server GUI, CLI through SSH and APIs for administration. Two user roles are there for administration in Delphix: Delphix Admin(the real admin) and Sysadmin(a lightweight admin who can do the day to day tasks).
Delphix can virtualize Oracle, DB2, postgres, Mysql, Sybase ASE databases.
DELPHIX FOR ORACLE:
---------------------------
Delphix engine is installed as a VM guest on VMware VSphere environment(Version 4.x or 5.x) or Amazon Ec2 Cloud . It requires minimum 8vcpus , 64 gb memory minimum.
Delphix supports Oracle Databases 9.2.0.8, 10.2, 11.1, 11.2 and 12.1
Delphix supports Oracle Databases on Solaris(Sparc and X86), RHEL, OEL, Suse, AIX and HPUX.
There is no supported storage list for Delphix , so it can run on any storage.
Delphix supports both standalone and RAC hosts.
In Delphix, target environment means Virtual Databases.
In Delphix, the delphix-side representation of a Source database is called as Dsource (Data Source).
Source environment speaks with Delphix using SSH, JDBC and DSP. DSP(Delphix Session Protocol) runs on port 8415 . Delpix requires port 8415 to be opened between Source and Delphix systems.
Basically, what Delphix does is , it takes the Dsource and creates a VDB from it and then provision it to a target system as a Read Write fully functional Oracle Database.
spfile is also provisioned by Delphix while create VDBs.
Flow of data for Oracle:
Rman takes Rman Level 0 backup using Rman APIs for the initalization. (this is done once while adding the source system in to Delphix).
Delphix takes that level 0 backup in to the Delphix system for the initalization and then takes level 1 and optionally archive/redo backups to synch itself with the source.
After the level 0 is taken, Delphix compress it 2 to 4 times. Level 1 backups and archivelog backups used to synch the Delphix are also compresses in the same way.
The VDBs are presented to the target servers using NFS or DNFS (Direct NFS). --> Direct NFS has direct IO (no OS cache) and aysnc IO capabilities and it is the power of it. (when compared to NFS)
The changes done in the VDBS are stored in the Delphix Storage Layer Server , compressed.
the VDBS are like the snapshots of Dsources, so no extra storage is needed for creating them, and so that they are very fast to create.
The changes in VDBS are stored in the Delphix Storage layer (like snapshot deltas)
Dsources have a timeflow. We can see the snapshots in those timeflow. The snapshot are actually the level1 backups taken in to the Delphix. (Delphix iniates level1 backups by sending commands to the source environment)
This snapshots, in other words; level1 backups are done via the Delphix Snap Sync service.
Alternatively, the snapshots can be done by collecting the archivelogs and redologs from the source system. This is done via the Delphix Log Sync service.
IO is done : from target server -> Delphix -> Storage.
Deplhix can take data from ASM but can not provision to ASM. (In this scope, Exadata can be a source or target, but Delphix can not provision to Exadata Storage, but it can provision the data to Exadata as an NFS mount point)
VDBs can be created from other VDBS as well.
VDBs can be rolled back or rewinded point in time, as they have also their own timeflow. This rewinds takes place purely in Delphix's Storage Layer (No oracle technologies are used for that)
VDBs can be refreshed from production as well. (from the dsource and from the snaphsots taken for that dsource)
The IO performance can be monitored and analyzed using Delphix 's Network Performance and Performance Analytics pages.
DELPHIX FOR MSSQLSERVER:
---------------------------
The requirements for the Delphix engine is the same as Oracle .
Delphix supports SQL Server 2005, 2008, 2008R2, 2012, 2014 and Windows Server 2003 SP2, 2008, 2008R2, 2012 and 2012R2 (64 bit)
Delphix supports Always on(for only source) + Windows Failover clusters(for both source and target)
When we talk about the database term in Delphix means Data files + Transaction logs + transaction log backups for Sql Server.
As for the Sql Server, Delphix recreates the source system in an intermediate system called the validated sync target/environment.
Validated sync environment is a replica, which is recovered using the database file backups and log backups from the production server via the SMB protocol.
Validated sync environment can be a target server or a standalone sqlserver.
The reason for having this environment is to take the backups from source sqlserver contiously.
When a dsource is created, Delphix kick offs a job to have the validated sync server to take the backups from the source server.
Validated Sync should be in the same Sql Server version as the source.
There is a database created in the validated sync environment to be recovered by the source continously. (this databases is called the staging database)
there should be 1 Validated Sync should be for each Delphix engine. (one to one) . This is recommended.
So, Delhix itself does not consume any backup files.
Once the initilaziation is completed, Delphix remains in sync in terms of differential backups and transaction log backups.
Delphix talks with the source instance using jdbc and see if there are new transaction logs or backup have been taken. Then Delhpix instructs the validated sync database to apply them.
As for the sqlserver (like Oracle), Delphix provides network based storage services for the target servers (ISCI protocol)
Again, snap sync service takes backups from the source database.
Log sync service can also be used for retrieving the transaction logs.
The source Sql Server database must be Full recovery mode.
Without log sync, you can't restore to any time you want. So, with snap sync, you can only recover the time that are associated with backups.(you can only recover to snapshots)
Note: Backups taken from the source automatically creates a snapshot in Delphix.
Key factors on Installation of Delphix:
With IOCollector.sql , performance test are done.
We run it in PROD. Then we run it in current CLONE environment and see the difference.
Then, we will take the CLONE outputs as a performance goal, because we are producing clones(so no need to compare Delphix with the Production environment performance)
We expect most of the CLONE performance to be %50 of PROD performance.
If there are lots of clone environments needed, then multiple engine can be implemented. (this decision will be based on the outputs collected in the analysis phase/sizing phase)
For analyisis and sizing , 3 things matter.
1)CPU,RAM for Delphix Engine
2)Network between Source and Delphix + Delphix and Target
3)IO for Storage
Installation and POV should be done by following the document: Delphix Pre Install User Guide
Jumbo Frames can be enabled between the Source and Deplhix Engine and Target. Jumbo Frames will increase the network performance (because we do DB IO , with big block sizes)
However, if the network between Source,Delphix and Target is not isolated, then all the network will be affected from that.
Jumbo frames will be enabled for all the hosts in the network and thay may harm them.. So be careful and analyze the environment of the customer.
Delphix should be installed closer to the target environment (not the source)
Delhpix Storage should be created on top of one to one relationship. 1 lun should be connected for 1 data store and for 1 vmdk....
Raid configuration is a concern for Delphix. Delhpix can only give recommendations... It does not matter for Delphix as long as it performs well.
VMDKs should be "thick" provisioned zeroed. (not thin provisioning) Thin provisioning can be used for this VMDKs as well.
Landshark environment is the demo environment for Delphix. It consists of 1 Delphix Engine, 1 target and 1 source systems.
The network for installing Landshark is very important. The documentation should be followed for configuring the network of the virtual machines, before starting the Landshark environment.
Landshark when configured, comes as a preconfigured inventory. So that, it automatically sees the delphix engine in it sees its source and its target automatically.
With just 1-2 click, we can create our first VDBs in our target. Also, we can create our Vfiles as well.
Vfiles is used when there is need to create a target filesytem from a source filesystem.
Cloning EBS apps tier using Delphix is an example where we use vfiles.
No comments :
Post a Comment
If you will ask a question, please don't comment here..
For your questions, please create an issue into my forum.
Forum Link: http://ermanarslan.blogspot.com.tr/p/forum.html
Register and create an issue in the related category.
I will support you from there.