Software -- Self Developed

I developed several applications in my career. In this page you will find some applications I developed. Actually these are my favorites. These applications are used in Production Systems.

As a dba and system engineer, sometimes application development was a part of my job. But most of the time I have been developed the applications (system applications) for extending my job and gaining a developer perspective...

Note that , I have a great interest in Operating System kernel, but so far I couldnt have much time to get into it...

DB SNIPER

I developed this Application , when I was working in Acıbadem HealthCare Group to ease the deployment process of the new codes to the Production systems, as well as to ease the administration and control processes.

This application was developed in Java.

Features:

• GUI is dynamically generated. No need to add or modify code to add/modify/delete a database/schema connection. All the configuration are done through the repository control file.
• Configuration of the GUI can be changed in the run time, as well.
• With Oracle jdbc 10g , quick connection establishment, and detailed message and error parsing.
• Easy installation.
• Dynamic invalid check, compile and several control operations.
• Detailed Event logger.
• Real time&detailed sql operation tracking through the Gui Panel.
• Execution log ( Ready to send to the Application Owner -- OS user, Networking information, Deployed SQls, errors, warnings and etc..)
• Coherent and detailed Log file that formated by the date and time.
• Two seperate tabs for DML/DDL and Query operations.
• Always active GUI-- GUI runs without get blocked by the dml/ddl and query operations.
• Parallel process capability. Parallel process are managed and controlled individually. A parallel process can be stopped, aborted by the administrator without effecting other parallel processes.
• Ignore errors feature for resuming operations on ignorable errors.
• Intelligent GUI that creates queries in a event based manner.
• Sql Control Generator feature , that can generate sql by the inputs supplied from administrator. Ability to generate and run queries without writing any sql codes. 
• Optimized data grid for controlling operations. High cost queries can be transformed into samples -- in the background on the fly.
• Stable GUI, controlling threads and window management through the Sniper Thread Manager
• Ability to make batch processing. (with abort, and ignore options.)

DB Sniper Technical Architecture

DB Sniper Running Parallel Threads- Gui ScreenShot

Threads of Db Sniper running dml, ddl and query operations concurrently in several Databases.
Db Sniper run the threads and controls them in background, GUI does not become blocked during this Operations.

Thead Tracking and Managament

Yellow : Thread that aborted by the admin..

Red : Thread that encountered error

Green: Thread that completed its work successfully..

All the thread logs are  reflected to the Execution Log Panel with detail.

Query Generator

Sql Query generator generates control queries for the deployment. It generates sql for controlling the deployment effect to the database system. It simplifies the control work of the database administrator.

Export the Log Feature

Db Sniper Deployment Log file Example

Log File named with date suffix for tracking the versions of deployments.

Log file contains OS username, machine name and IP adress that belongs to the administrator makes this deployment. Sniper takes this information from Client OS (not from the database).

----------------------------------------------------------

LIMITER

I developed this application in order to prevent batch processing and reporting applications consume high resources. Limiter was a tool that had a critical role in System optimization project of a big heathcare company.It has beed used in EBS Production environments for preventing reporting applications consume high resources and block Erp processes.

Limiter was developed in C language. 

The main job of Limiter is to control Cpu cycles (run queue) of the Linux Operating system with Operating System Signals.
Signal distrubution is under control of Operating System kernel. These signals can not be catched, because these signals are not reflected to the processes. Os handles them. The outputs of the processes are not affected by these signals.
Limiter is not designed to decrease the Elapsed time of a running process. It is designed to decrease the CPU time/per min.

Features:

Ability to limit the resources of numerous running processes through the limiter processes/forks and execs running on background.

Easy installation by its setup program.

Automatic prerequisite checks on setup.

Ability to control the resources of running process, ability to list, increase and decrease the limit percentages through the limiter repository.

Self managemed repository. Ability to take the rule back in case of process termination and completion.

Process based resource limitation.

Group based resource limitation.

Ability to deploy the current rules to the new processes.

Usage information is based on the Linux Man page.

Ability to catch interrupts like CTRL+c.

Limiting sensivity (%1 to %99)

Limiter Code:

Used C libraries:

•<sys/types.h>,<unistd.h>,<errno.h>,<sys/wait.h>,<dirent.h>,<stdlib.h>,<sys/dir.h>,<sys/stat.h>,<stdio.h>

Functions:

–Check_process : Control process existence

–Catcher: Catches the interrupts

–Search:Process search

–Write_to_rep: To write or modify the repository

–Write_to_log : Writes to the logfile

–Release_pid : Revokes the rules applied to a process

–List_rep:  Lists repository contents

–Exit_from_limiter : Graceful exit.

–End_all : Takes out all the rules in once.

–Delete_from_rep : Deletes a record from repository

–Clear_rep : For cleaning the repository.

Installation:

Unistallation:

Limiter Linux Man Page (deployed with the installation):

                             man limiter> 

Limiter - Limiting the process

In this example, Limiter limits an oracle job process by a percentage of %50.

This means this process can use the half of the resources in unit time. (even all the resources are available.) Also, Repository content can be listed by list rep command , as in the example below.

Processes that are limited by Limiter, are placed into stopped mode according to the quantum defined. There is no risk in terms of process failure or data loss in this operation. Everyting is under control of OS.

Process List:

Ts status means stopped.
When you limit the process resources with limiter, stop and continue signals are begin sent to the processes. Stopping a process actually is the key factor that limits the resource consumption of a process.
These signals are uncatchable.
When the process stops processing, its context is saved by the OS. When a process start working again/continues its context is loaded back and process works without losing anything.

Limiter -  Rolling back the process limit to their orginal values.

----------------------------------------------------------

STEADY BACKUP

This application has been defined a sophisticated backup program to achieve the following: 

– Improved backup performance
–Effortlessness but certain backup management and control
–Minimized storage needs.

Features:

•Reliable 6 Different Backup techniques.
•High flexibility by the help of parameter file.
•Changing backup style and parameter file via command line user input.
•Easy setup via setup script.
•Ability to backup without downtime, using hot backup option.
•Gzip option for Cold,export and document backups to minimize storage.
•Schema by schema export opportunity.
•Advanced backup and avaliable size comparison.
•Capability of running user made custom backup scripts.
•Archive log deletion option considering backup achievement.
•Able to erase/store/overwrite old backups.
•Critical and non critical backup event tracking.
•Backup reliability test.
•Capable of take consistent backup, by administrating oracle services.
•Mail notification with detailed information.

Code:

•1258 lines of code written in Perl, operating plsql/sql and shell code generally for database interaction. 

•Usage of following system calls and system programs: 

–Directly from Perl via following functions: 

•Fork,open,close,wait,floc 

–From Shell via following functions: 

•ls,rm,mt, ncftp,Sendmail,Du,df,gzip,gunzip,tar 

•Almost no internal variable is defined in the code, program takes all the input variable from parameter file. 

Parameters:

•GZIP=1 # Set it to 1 for gzipping backup file on the fly while copy phase is running.
•GZIP_FACTOR=5 # Set it to a positive integer. It is used for gunzipped backup size estimation
•CUSTOM_COPY_SCRIPT=0 #Set it to 1 if a custom script will be used in backup copying phase, for custom script edit the copy_script_custom file script_path
•PARALLEL_SERVERS=2 #Set it to a positive integer ( default 1) to specify how many parallel workers will be used for copy.
•GENERATED_SCRIPT_PATH=/asg_backup/scripts #Scripts generated in runtime will be stored here.
•COLD_DIR=/yedek/OraYedek/cold #Cold backup location, to store the backup files.Directory permissions are important.
•SQLDIR=/asg_backup/sql #Sql files generated in runtime will be stored here.
•LOGFILE=/tmp/backup.log #Logfile
•ORACLE_CONNECT="/ as sysdba" #Oracle connection type.
•SQLPLUS="/home/oraasg/orahome/product/10g/bin/sqlplus" #Sqlplus path just in case if env is not set properly.
•LIBRARY_PATH=/home/oraasg/orahome/product/10g/lib #Oracle Libraries path just in case if env is not set properly.
•DB_HOME="/home/oraasg/orahome/product/10g" #Oracle Home path just in case if env is not set properly.
•ORACLE_USER=oraasg #Operating system Oracle User
•SID=ADNDB #Oracle SID of database that will backed up.
•SCRIPT_PATH=/asg_backup #Script path
•DEL_BCK_BEFOREHAND=1 #Set it to 1 to delete backup file in COLD_DIR directory before the new backup starts.
•LISTENER_NAME=LISTENER #Listener name to start..stop..
•LISTENER_PATH=/home/oraasg/orahome/product/10g/bin/lsnrctl #Listener_path just in case if env is not set properly.
•SHUTDOWN_ORACLE=1 #Set it to 1 for stopping database before the copy phase. Listener will also be stopped. STARTUP_ORACLE=1 #Set it to 1 for starting database after the copy phase. Listener will be started, too.
• REWIND_TAPE_AFTER_BCK=1 # Set it to 1 if you want to rewind the tape after backup
• TAPE_BACKUP_DIRS=/yedek/OraYedek # Write directories with one space
•EXP_PATH=/home/oraasg/orahome/product/10g/bin/exp #Set it for specifying the export binary just in case if env is not set properly.
•EXP_DIR=/yedek/OraYedek # Exp backup location to store exp backups.
•EXP_SCHEMA=full # Exp backup style full or schema_name..for multiple schemas use comma seperator
•EXP_USER=system # Exp database user
•EXP_USER_PASSWD=oracle8 # Exp databaase user password
•MOVE_OLD_EXPORT=1 # Set it to 1 if old backup should be moved with old suffix at the same directoy as exp_dir
•EXP_LOG_FILE=/tmp # Set it to the directory for locating exp log file.
•EXP_GZIP=0 # Set it to 1 gzip export dump files on the fly.
•DOC_BACKUP_DIR=/yedek/OraYedek/doc # Doc backup location
•DOC_FILES_DIR=/data/filemerkez/Documents/HIS # Doc files to be backed up
•DOC_BACKUP_TO_KEEP=3 # Set it to a positive integer to specify how many doc backup to keep
•FTP_PATH=/asg_backup/ncftp/ncftp-3.2.3/bin/ncftpput # ftp client binary
•FTP_SERVER="10.90.0.112" # ftp server
•FTP_USER=ftpuser # ftp user for login
•FTP_PASSWORD=ftpsifre # ftp user passwd for login
•SOURCE_DIR=/yedek/OraYedek # source directory containing backup file to be send via ftp
•TARGET_DIR=/Public/DRBackup # target directory, in which backup files will to be stored.

Cold Backup Style:

I.Oracle services control . 

II.Gathering information from database. 

III.Needed storage calculation. (even if with gunzip option) 

IV.Oracle services shutdown .(optional) 

V.Putting Oracle into hot backup mode (optional) 

VI.Deleting old backups beforehand (optional) 

VII.Parallel copy and/or zip backup files on the fly. (zip and paralel options are optional) 

VIII.Taking out Oracle from hot backup mode (optional) 

IX.Backup reliability check. ( except custom script if specified) 

X.Oracle services startup (optional) 

XI.Archivelog remove ( optional, and if backup is successful ) 

XII.Mailing the log file

•All the steps are controlled by the event monitor routines. 

•Whenever and wherever an error is produced, it appears in the log file which will be mailed to the specified addresses.

Export Backup Style:

•Move old export backup (optional) 

•Export specified schemas ( gunzip optional) 

•Export full database ( gunzip optional) 

•Mail notification

•All the steps are controlled by the event monitor routines. 
•Whenever and wherever an error is produced, it appears in the log file which will be mailed to the specified addresses.

Doc/Binary Backup Style

I.Rotate old doc backups ( parameter doc_to_keep) 

II.Tar and zip Document files 

III.Mail notification

•All the steps are controlled by the event monitor routines. 
•Whenever and wherever an error is produced, it appears in the log file which will be mailed to the specified addresses.

Ftp Backup Style

I.Ftp connection 

II.Ftp file upload 

III.Notification mail

•All the steps are controlled by the event monitor routines. 
•Whenever and wherever an error is produced, it appears in the log file which will be mailed to the specified addresses.

Tape Backup Style

I.Choose tape device ( st0 or nst0) 

II.Rewind tape ( optional) 

III.Archive files to tape using tar

•All the steps are controlled by the event monitor routines. 
•Whenever and wherever an error is produced, it appears in the log file which will be mailed to the specified addresses.

Steady Backup Notification Mail / Log File

Backup log file is written in a human readable form, which actually shows every steps from backup phase to database operations. Log file even lists the detailed information about the work that the parallel workers do.

Steady backup even checks the backup (even if it is zipped) and decide wheter it is reliable.

Every routine of the code has an error handler. So everyting is beging catched and logged.

----------------------------------------------------------

APPS TECH INTERFACE

These interface is written by bash. It is designed to ease the system administration. Through this interface, application stop/start, db stop/start, session kill operations, patching operations and all related admin operations can be done by clicking the menu buttons.

Users of this interface should not be System administrators. Everyone can use this application and make administration operations.

User will connect to the system using ssh, and this apps tech interface menu will be displayed in its terminal( putty or ssh secure shell or zoc does not matter)..

Whenever the user quits from the application , the connection will be closed too. So linux will act like a kiosk machine.

to be continued...