Moving to Continuous integration and Continuous Delivery CICD

All companies (Big/small/startup) want to adopt to CICD because it helps to accelerated time to market (step ahead of the competition), improved productivity and efficiency and improved customer satisfaction and so on. New organizations have the benefit of implementing CICD from scratch. Existing companies have to drive steadily from manual/semi-automated to highly automated development. 

Continuous Delivery is a modern software development/release practices and principles aimed at building, testing and releasing software faster and more frequently. Building a successful CD pipeline means creating a DevOps culture of collaboration among the various teams involved in software delivery (developers, operations, quality assurance teams, management, etc.), as well as reducing the cost, time, and risk of delivering software changes by allowing for more incremental updates to applications in production.

Challenges:

Identifying and mitigating any challenges early in the CICD implementation will help you to head start the project as well as help you to address future challenges.  Most common challenges of enterprise:

Process Change: Most of the company has their own change control process handoff, Signoff, change control approval process and it takes its own time to approve.  Identify the delayed process and come up with acceptable solutions and fast forward solutions.

Technical Challenges:  For CICD, we will be using more third party tools and APIs. Avoid vendor lock-in for third party software, harden the security of open source software, and develop widely accepting standards and procedures for API and plugins.  Automation process needs its own testing and error handling properly to avoid any quality degradation in the pipeline. Frequent script changes are error-prone and need to be avoided. 

Organization change:  Organization need to make cultural change of Agile CICD model across the company.  Teams would have worked independently and worked on own timeline but with new Agile CICD model collaborative approach needs to be engaged in defined timeline.  The leadership team need to restructure the organization to break down barriers among teams and promote a collaborative culture.

Automation of CICD pipeline: (Need not to be same and depends on the organization requirement).

Ø  Maintain a Single Source Repository.     

Ø  Automate the Build                                       

Ø  Make Your Build Self-Testing

Ø  Everyone Commits To the Mainline Every Day

Ø  Every Commit Should Build the Mainline on an Integration Machine

Ø  Deployment to integration environment

Ø  Integration tests

Ø  Scenario/regression tests

Ø  Deployments to UAT and Performance test environment

Ø  More scenario/regression tests

Ø  Performance tests

Ø  Alerts, reports and Release Notes sent out

Ø  Deployment to release repository

Ø  Deploy to production

Source Control Management: Source control management tools provide the ability to store separate versions of things, including the ability to manage variation through branching and merging.

Tools: Git, Subversion, Mercurial, Veracity, TFS, ISPW

Continuous Integration: integration is a software development practice where developers regularly merge their code changes into a central repository, after which automated builds and tests are run.

Tools: Jenkins, codeship, Teamcity,continuum, Travis CI, CircleCI, Bamboo, Visual Studio, shippable

Build automation: Build automation tools are software applications that automate the compilation and integration of executable software components from source code.

Tools: Maven, Gradle, Ant, Gulp, Cmake, MSbuild

Artifact Repository: Artifact repository tools are used to store and manage binary objects that are used to build software.

Tools: Dockerhub, npm, Nexus, Artifactory

Continuous delivery: Continuous Delivery simply builds on this foundation and extends it more into the domain of deployment and project delivery.

Tools: Jmeter, Cucumber,Junit,Mocha,Selenium,Qunit,TestING, Jasmine, Karma,Qunit.LoadRunner, WebLoad, Testing Anywhere

Configuration & provisioning:  Configuration management tool automates deployment of just about any infrastructure and application stack used to create modern cloud and enterprise IT.

Tools: Chef, Ansible, Salt,CFengine, puppet, Terraform, vagrant

Application Security: Application security solutions need to be naturally integrated into the CICD workflow. 

Tools:  HPE Fortify, Tripwire, Snort

Application logging:  Application logging system helps to troubleshoot application problems and investigate security incidents in minutes instead of hours or days, avoid service degradation or outages, deliver compliance at lower cost and gain new business insights.

Tools: Logstash, Splunk, SumoLogic, Logentries

Application monitoring: Application monitoring needed for the entire system to ensure systems, applications, services, and business processes are functioning properly. In the event of a failure, system can alert technical staff of the problem, allowing them to begin remediation processes before outages affect business processes, end-users, or customers.

Tools: Nagios, Zabbix, NewRelic, Appdynamics, Datadog, Dynatrace

Closing statement:

Identifying the common challenges to put CICD into practice, evaluate the situation should be a first preparatory step in your implementation. Gain an accurate picture of the current baseline process, structure of your implementation in measurable phases, and then work on dealing with these challenges to clear the way for your first delivery pipelines.   Integration tools tie together from multiple vendors into a single integrated tool chain, ensuring data consistency and integrity is important. Don’t need to automate the entire system at once, implement automation in phased approach. Good luck with your CICD Move!

Moving to Cloud computing (Legacy application migration to cloud)

Huge momentum has built to migrate applications to cloud computing nowadays. There are several motivation factors to adapt to cloud:  cost savings IT infrastructure and maintenance, time to market, rapid deployment and provisioning, auto-scaling, flexible capacity, and business continuity.

Moving existing applications is not simply a matter of lifting and shifting to a cloud platform. Migration of existing applications to cloud computing must be done in a strategic and systematic manner. Identify individual business problems with existing applications that cloud computing can potentially address and provide specific business justification that the cloud is the right strategic alternative.

Enterprise applications must be thoroughly assessed to determine which application can benefit most from early migration to the cloud. Key considerations including costs of migration, application redesign, application performance and availability, security and privacy requirements, and regulatory requirements must be taken into account.  Create a pilot for one or two of these types of applications, test thoroughly, and gather customer feedback so that trust and confident can be built before going live.

The important point is that decisions about moving to the cloud need to start from an application standpoint, with a thorough consideration of followings:

Stateless: The thump of rule is keep your application from being affected by dynamic scaling: Build your application to be as generic and stateless as possible.  Avoid singleton.

Hardware dependency: Services that application calls should not be pointing particular host names or IP addresses. Abstracting environment-specific dependencies into a set of property files is still an inadequate. A better approach is to use an external service registry to resolve service endpoints like load balancer with virtual name.

Avoid local file system:  Avoid local file system as a store for temporary information, put temporary information in a remote store such as an S3, Amazon EFS, RDS or NoSQL database.

State-fullness: HTTPSession state is often stored in-memory, which presents a problem if your entire application server can be added or removed or moved at any time in the cloud. The best practice is to push it out to a highly available store that is external to your application server; that is, put it in a distributed caching store, such as  Memcached, or Redis.

Application Log Fie:  Application services are in auto scaling added/moved/removed often and storing log file in the local file system will crash your debugging option or the file you are looking for may not be available for debug. Use an open source aggregator, such as Scribe or Apache Flume, ELK or a commercial product, such as Splunk.

Asynchronous Calls: Consider that asynchronous protocols (Queue & RESTAPI) are still alive and well and can be extremely effective for cloud application programming.

Portability: Avoid OS dependency and application should be portable to any time of middleware.  Use Cygwin and Mono to port between Linux/.Net.

Closing statement:

Management should realize that every significant transition in technology has resulted in a significant rewrite or re-implementation of the technologies that businesses depend on but deep analysis and systematic approach of changing to service-oriented architecture (SOA) can make your transition smooth and success.

MYSQL master / Slave setup and rebuilding steps

MYSQL  Replication

MySQL Replication, a solution suggested by our resident database guru, is the process of setting up a master/slave relationship between two database servers, where the slave continually and automatically gets fed all MySQL queries from the master machine, resulting in a replication of data across the two machines.

Once replication has been configured and activated, the slave machine will act as a live MySQL database, holding an up-to-the-minute snapshot of all data from the master machine.

What we love about this solution is that it leaves no "holes". No potential for data loss and no need to ever take the database down.

Once the slave machine is ready, with the flip of a switch, we could redirect all queries to the new machine, terminate the master/slave relationship and we'll be up and running with the new machine.

Here's how the actual implementation went:

Step 1: Set up an account on the master server that the slave server can use to connect. This account must be given the REPLICATION SLAVE privilege.

mysql> use mysql;
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%.mydomain.com' IDENTIFIED BY 'slavepass';

Step 2: Turn on binary logging of MySQL transactions. The server cannot act as a replication master unless binary logging is enabled. Locate the my.cnf file on the master database server machine and add these two lines to the file, under the [mysqld] section:

[mysqld]
log-bin=mysql-bin
server-id=1

If those options were not present (we had them), you will have to restart the server.

Step 3: Flush all the tables and block write statements by executing a FLUSH TABLES WITH READ LOCK statement - this is important so that we can safely copy the database files, with no fear of the files being changed during the copy operation. Make sure you keep the shell window where you issue the command below open throughout the process, until the tar finishes. Once this window closes, the lock will be automatically released.

mysql> SET GLOBAL WAIT_TIMEOUT=600000; SET WAIT_TIMEOUT = 600000; FLUSH TABLES WITH READ LOCK;

Step 4: Take a snapshot of all data on the master database server. The easiest way to create a snapshot is to use tar. (Make sure you have sufficient storage space on your master server to hold the tar. If unsure, calculate the current size of your database and plan for an additional 70% of available space)

cd /var/db/mysql
tar -cvf /tmp/mysql-snapshot.tar ./ --exclude mysql &

On our system this operation took a little over four hours to complete. Notice the "&" operand - this causes tar to run as a background process. Useful if your shell gets disconnected. You'll know it's done by periodically viewing the process-list using "ps -ax"

Step 5: While the read lock placed by FLUSH TABLES WITH READ LOCK is in effect, read the value of the current binary log name and offset on the master:

mysql > SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+---------------+----------+--------------+------------------+
| mysql-bin.003 | 73 | test | manual,mysql |
+---------------+----------+--------------+------------------+

The File column shows the name of the log and Position shows the offset within the file. In this example, the binary log file is mysql-bin.003 and the offset is 73. Record these values. You need them later when you are setting up the slave. They represent the replication coordinates at which the slave should begin processing new updates from the master.

Step 6: Now that you have taken the snapshot and recorded the log name and offset, you can re-enable write activity on the master:

mysql> UNLOCK TABLES;

Step 7: Hop on the new database server machine (slave). Shutdown the database, then locate the my.cnf file and add these lines to the file:

[mysqld]
server-id=2

Step 8: Copy and extract master database snapshot to the slave server under /var/db

cd /var/db

ftp
open masterdb.com
cd /tmp
get mysql-snapshot.tar
exit

tar --extract --file=mysql-snapshot.tar

Step 9: Start the slave database and issue these commands, replacing the option values with the actual values relevant to your system:

mysql> CHANGE MASTER TO
MASTER_HOST='master_host_name',
MASTER_USER='replication_user_name',
MASTER_PASSWORD='replication_password',
MASTER_LOG_FILE='recorded_log_file_name',
MASTER_LOG_POS=recorded_log_position;

Step 10: Start the slave

mysql> START SLAVE;

If everything went well, your slave database will now connect to the master database, downloading all transactions following the time you took the snapshot. From this moment on, the slave database will continually feed itself with data from the master database, always staying in sync.

Confirm the slave server is up-to-date by running a:

mysql> SHOW SLAVE STATUS;

and comparing the position markers with what the master database reports under:

mysql> SHOW MASTER STATUS;

Step 11: Update database-connect functions

Once the slave database is up-to-date, it is now time to update your application code, routing all calls to the new server.

Assuming you have all database access functions encapsulated in a database.php / database.c layer, locate the function that establishes a connection to the database and update the IP address or domain name to point to the slave database.

Voila! There you have it -- migration of MySQL from machine A to machine B without ever taking the database down.

Step 12: Test

Run a tail on the master database MySQL log file, or run SHOW PROCESSLIST and make sure no one is accessing the master database any more. Once confirmed, shutdown the master database and test for any strange side effects.

I recommend you keep the old database server up for a few days until you're absolutely certain the transfer went through smoothly.

Running a mysqlcheck --all-databases on the new database is highly recommended.

MYSQL  Replication rebuild

A few assumptions before getting started:
*You are repairing a previously existing master/slave replication process. If not, you'll have to add a user to Master who the Slave can use to get its info. Will also have to set up some other variables etc. Can find that info somewhere.
*If your server/database is live and are worried about performance, use:
"nice" before linux commands
"--skip-lock-tables" on mysqldumps to prevent locking insert/reads
"nohup ... &" to execute the linux/unix command inthe background, unattended

Here is the process I did:

1) Drop Slave database.
mysql>drop database x;

2) Create dump of Master's create table info:
nohup mysqldump --no-data --skip-lock-tables --single-transaction -uuser -ppassword db > file.sql &

3) Dump any tables you know won't change or that you can stop from changing without problems. Use same command as in 2)

4) **This is the part you need to lock tables
ON MASTER:
mysql>reset master; **This took me ~9 minutes
mysql>flush tables with read lock; **~17 seconds
mysql>show master status;

(VERY IMPORTANT)
Copy down the File and Position information

(From another terminal, still on master's server)
linux>nohup mysqldump --skip-lock-tables --single-transaction --no-create-info --routines --ignore-table=db.table_already_dumped -uuser -ppassword db > file2.sql &

mysql>UNLOCK TABLES;

(If databases are on different servers)
5)Copy dumps to other server
nohup nice scp file1.sql user@IPAddress:/path/ &
nohup nice scp file2.sql user@IPAddress:/path/ &
(Include other dump(s) from step 3) )

(ON SLAVE)
6)Import dumps
mysql>\. file.sql
*Might want to use nohup mysqlimport .... if you want to leave UNATTENDED!

(Still on Slave)
*The x,y on the change master... command are the data you got from "show master status" in step 4)
7)Start replication
mysql>stop slave;
mysql>reset slave; (This will delete the relay logs, etc)
mysql>change master to MASTER_LOG_FILE='x', MYSQL_LOG_POS='y';
mysql>start slave;

mysql>show slave status \G

How to do successful oracle 11g upgrade?

I recommend users who are upgrading to oracle 11g to check the sys/system

duplicate objects before upgrade. Duplicate check step is not 

part of the 11g upgrade doc or manual installation doc.

I recommend following all steps which are provided in your 

installation doc and use the following procedure to find any 

duplicate before you proceed with your upgrade. 

Oracle should include this steps part of upgrade doc to make 

upgrade successful. 

Problem Description: 

==================== 

If the same data dictionary object has been created under both user SYS and 

SYSTEM schema then errors will often occur when trying to use the database 

features associated with these objects. 

Problem Explanation: 

==================== 

During the installation of Oracle many scripts are run which create the 

underlying data dictionary objects. Most of these scripts are run at database 

creation time, but others are only run if specific database features (such as 

replication or shared pool management) are needed. These scripts are usually 

run manually after the database has been created. 

Solution Description: 

====================== 

In order to clean up the duplicate objects you need to issue a SQL script to 

find out the names of the duplicate objects. You can then manually drop the 

objects or use a 'SQL generating SQL' script to generate a list of drop 

commands. 

Below is a SQL*Plus script that will list all objects that have been created 

in both the SYS and SYSTEM schema: 

column object_name format a30 

select object_name, object_type 

from dba_objects 

where object_name||object_type in 

   (select object_name||object_type  

    from dba_objects 

    where owner = 'SYS') 

and owner = 'SYSTEM'; 

The output from this script will either be 'zero rows selected' or will look 

something like the following: 

OBJECT_NAME                OBJECT_TYPE 

------------------------------ ------------- 

ALL_DAYS                       VIEW 

CHAINED_ROWS                   TABLE 

COLLECTION                     TABLE 

COLLECTION_ID                  SEQUENCE 

DBA_LOCKS                      SYNONYM 

DBMS_DDL                       PACKAGE 

DBMS_SESSION                   PACKAGE 

DBMS_SPACE                     PACKAGE 

DBMS_SYSTEM                    PACKAGE 

DBMS_TRANSACTION               PACKAGE 

DBMS_UTILITY                   PACKAGE 

If the select statement returns any rows then this is an indication that at 

least 1 script has been run as both SYS and SYSTEM. 

Since most data dictionary objects should be owned by SYS (see exceptions 

below) you will want to drop the objects that are owned by SYSTEM in order to 

clear up this situation. 

EXCEPTION TO THE RULE 

===================== 

THE REPLICATION SCRIPTS (XXX) CORRECTLY CREATES OBJECTS WITH THE SAME NAME IN 

THE SYS AND SYSTEM ACCOUNTS. LISTED BELOW ARE THE OBJECTS USED BY REPLICATION 

THAT SHOULD BE CREATED IN BOTH ACCOUNTS. DO NOT DROP THESE OBJECTS FROM THE 

SYSTEM ACCOUNT IF YOU ARE USING REPLICATION. DOING SO WILL CAUSE REPLICATION 

TO FAIL! 

The following objects are duplicates that will show up (and should not be removed)

when running this script in 8.1.x and higher.

Without replication installed:

INDEX           AQ$_SCHEDULES_PRIMARY

TABLE           AQ$_SCHEDULES

If replication is installed by running catrep.sql:

INDEX           AQ$_SCHEDULES_PRIMARY

PACKAGE         DBMS_REPCAT_AUTH

PACKAGE BODY    DBMS_REPCAT_AUTH

TABLE           AQ$_SCHEDULES

When database is upgraded to 11g using DBUA, following duplicate 

objects are also created

OBJECT_NAME                OBJECT_TYPE 

------------------------------ ------------- 

Help                           TABLE 

Help_Topic_Seq                  Index

The objects created by sqlplus/admin/help/hlpbld.sql must be owned by SYSTEM 

because when sqlplus retrieves the help information, it refers to the SYSTEM 

schema only. DBCA runs this script as SYSTEM user when it creates the database 

but DBUA runs this script as SYS user when upgrading the database (

reported as an unpublished BUG 10022360).  You can drop the ones in SYS schema.

Now that you have a list of duplicate objects you will simply issue the 

appropriate DROP command to get rid of the object that is owned by the SYSTEM user. 

If the list of objects is large then you may want to use the following 

SQL*Plus script to automatically generate an SQL script that contains 

the appropriate DROP commands: 

set pause off 

set heading off 

set pagesize 0 

set feedback off 

set verify off 

spool dropsys.sql 

select 'DROP ' || object_type || ' SYSTEM.' || object_name || ';' 

from dba_objects 

where object_name||object_type in 

   (select object_name||object_type  

    from dba_objects 

    where owner = 'SYS') 

and owner = 'SYSTEM'; 

spool off 

exit 

You will now have a file in the current directory named dropsys.sql that 

contains all of the DROP commands. You will need to run this script as a 

normal SQL script as follows: 

$ sqlplus 

SQL*Plus: Release 3.3.2.0.0 - Production on Thu May  1 14:54:20 1997 

Copyright (c) Oracle Corporation 1979, 1994.  All rights reserved. 

Enter user-name: system 

Enter password: manager 

SQL> @dropsys 

Note: You may receive one or more of the following errors: 

      ORA-2266 (unique/primary keys in table 

      referenced by enabled foreign keys):  

      If you encounter this error then some of the tables 

      you are dropping have constrints that prevent the table from being 

      dropped. To fix this problem you will have to manually drop the 

      objects in a different order than the script does. 

      ORA-2429 (cannot drop index used for enforcement of 

      unique/primary key): 

      This is similar to the ORA-2266 error except that it points to 

      an index. You will have to manually disable the constraint 

      associated with the index and then drop the index. 

      ORA-1418 (specified index does not exist): 

      This occurs because the table that the index was created on 

      has already been dropped which also drops the index. When the 

      script tries to drop the index it is no longer there and thus 

      the ORA-1418 error. You can safely ignore this error.

Oracle Enterprise Manager 11g for Oracle Database 11g Release 2

What's new in OEM 11g R2

Compression Advisor: A Compression Advisor has been added in Oracle Database 11g Release 2 to facilitate choosing the correct compression level for your data. The Compression Advisor is available through Oracle Enterprise Manager 11g Grid Control console. As part of the existing Advisor framework in Oracle Database 11g, the compression advisor analyzes objects in the database, discovers the possible compression ratios that could be achieved, and recommends optimal compression settings.

Instance Caging: Oracle Enterprise Manager 11g introduces more flexible alternatives to server partitioning to support resource management. Instance caging allows you to limit CPU count per instance and enables resource management that has more consistent support across operating systems compared to resource managers available in many operating systems. It also has lower administration overhead than traditional virtualization techniques.

Global RAC Diagnostics: Oracle Enterprise Manager 11g extends Automatic Database Diagnostics Monitor (ADDM) for Oracle Database 11gR2 by greatly enhancing cluster-wide performance analysis for Real Application Clusters (RAC) databases. For RAC environments ADDM analyses the RAC cluster and reports on issues that are affecting the entire database as well as its individual instances. DBAs can now use ADDM to perform database-wide analysis of global resources, such as high-load SQL, global cache interconnect traffic, network latency issues, skew in instance response times, and I/O capacity.

Tuning Standby Databases: Active Session History (ASH) is now available for performance diagnostics for standby databases. Using Grid Control, DBAs can quickly drill-down into problem areas that are affecting performance and resolve them with minimal impact.

High Availability Console: Accessed through Grid Control, the new High Availability Console allows you a configuration-wide summary view, with charts, of RAC and HA setups instead of individual targets. It provides HA configuration best practices and standards, along with recommendations from the new Maximum Availability Architecture Advisor, to help you implement a HA system. The HA Console also uses EM's powerful workflows and provisioning automation to minimize downtime for things like single instance to MAA conversions.

What's new in PostgreSQL 9.0.2

PostgreSQL 9.x has two good features:

Hot Standby and Streaming Replication are the two new features that mark Version 9.0 as a landmark in PostgreSQL's development . 

Other benefits:

• Avoid failure when trying to report an impending transaction wraparound condition from outside a transaction. This oversight prevented recovery after transaction wraparound got too close, because database startup processing would fail, backpatched to all versions.
• Fix recovery from base backup when the starting checkpoint WAL record is not in the same WAL segment as its redo point, backpatched to all supported versions.
• Force default wal_sync_method to be fdatasync for Linux, backpatched to all supported versions.
• Fix memory leak with ANALYZE and complex index expressions, backpatched to all supported versions.
• Fix memory leak in contrib/xml2 functions, backpatched to all supported versions.
• Fix for segfault in pgcrypto when attempting to access an undefined public key, backpatched to all supported versions.
• Prevent inlining an SQL function with multiple OUT parameters, backpatched to all supported versions.
• Fix bugs in WAL replay logic for GIN indexes, backpatched to 8.2.
• Fix segfault when accept() or a call made right after accepting a connection fails when server is compiled with GSSAPI support, backpatched to 8.3.
• Fix error handling in tempfile deletion with log_temp_files active, backpatched to 8.3.
• Prevent slowing of of autovacuum workers because of leakage in cost parameter over life of worker, backpatched to 8.3.
• Check for too many children before returning "okay to proceed", backpatched to 8.4.
• Improve performance of internal tuplestore structures for window functions, backpatched to 8.4.

The 9.0.2 update also contains several fixes for issues with features introduced or changed in version 9.0:

• Fix low-risk potential denial of service against RADIUS login.
• Make OFF keyword unreserved.
• Fix bug in tracking of latest removed WAL segment in streaming replication.
• Fix for buffer overrun in pg_upgrade.
• Fix memory leak in autovacuum launcher.
• Fix for race condition in transaction ID assignment and lock acquisition in hot standby.
• Improve performance for parallel restore with many blobs.

Cool SCAN feature 11gR2

SCAN Concepts

• Single client access name (SCAN) is the virtual hostname to provide for all clients connecting to the cluster (as opposed to the vip hostnames in 10g and 11gR1).  
• SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS).
• By default, the name used as the SCAN is also the name of the cluster and must be globally unique throughout your enterprise. The default value for the SCAN is based on the local node name. SCAN name must be at least one character long and no more than 15 characters in length, must be alphanumeric - cannot begin with a numeral and may contain hyphens (-). If you require a SCAN that is longer than 15 characters, then select an Advanced installation.
• For installation to succeed, the SCAN must resolve to at least one address.
• SCAN VIP addresses must be on the same subnet as virtual IP addresses and public IP addresses.
• Oracle strongly recommends that you do not configure SCAN VIP addresses in the hosts file. But if you use the hosts file to resolve SCAN name, you can have only one SCAN IP address.
• If hosts file is used to resolve SCAN hostname, you will receive Cluster Verification Utility failure at end of installation (see Note: 887471.1 for more details)
• For high availability and scalability, Oracle recommends that you configure the SCAN to use DNS Round Robin resolution to three addresses.
• Because the SCAN is associated with the cluster as a whole, rather than to a particular node, the SCAN makes it possible to add or remove nodes from the cluster without needing to reconfigure clients. It also adds location independence for the databases, so that client configuration does not have to depend on which nodes are running a particular database.
• Clients can continue to access the cluster in the same way as with previous releases, but Oracle recommends that clients accessing the cluster use the SCAN. Clients using the SCAN can also access the cluster using EZCONNECT.
• Grid Infrastructure will start local listener LISTENER on all nodes to listen on local VIP, and SCAN listener LISTENER_SCAN1 (up to three cluster wide) to listen on SCAN VIP(s); 11gR2 database by default will set local_listener to local LISTENER, and remote_listener to SCAN listener.

Oracle 11gR2 Basics

The Basics of 11gR2:

• The clusterware is now known as Grid Infrastructure.
• CRS and ASM binaries reside under a single Oracle Home known as GRID Home.
• GRID Home and the Oracle Home must be installed in different locations.
• Oracle Grid Infrastructure cannot be placed under Oracle Base since permissions will be changed to root.
• OCR and Voting File can no longer be placed on RAW or block devices. They need to be placed either in ASM or on Shared NFS Storage.
• Voting file does not need to be backed up using dd command and is not supported anymore.The contents of Voting file are backed into the OCR.

Oracle 11g Known bugs

Oracle 11g bug.
BUGs:

• Hugepages not being used. First check if /etc/security/limits.conf is set correctly for 'memlock'. You can also check 'ulimit -l'. However there is a bug with 11.2.0.1 where if database is started by srvctl and not sqlplus then hugepages are not used. The workaround is to edit either $GRID_HOME/bin/ohasd or /etc/init.d/ohasd and put 'ulimit -l unlimited' explicitly. This bug is supposedly fixed in 11.2.0.2.
• If loopback database link is used, the SQL would hang forever waiting on 'DFS Lock Wait'. Till date the issue persists and there doesn't seem to be fix from Oracle. Unofficially though I know one of the underscore parameter fixes the issue but would essentially disable majority of cluster features.

Oracle update performance speed best practices

Oracle update performance speed best practices

The SQL standard for DML UPDATE statements can be complex and convoluted and there are best practices that can help you to write efficient UPDATE statements.

    * Run updates in batch mode
    * Use CTAS in lieu of large updates
    * Include the SET condition in the WHERE clause
    * Simplify the WHERE predicates
    * Have a small, separate data cache for high DML tables

Running updates in batch mode

If you can take your application offline during the monthly update, you use many update performance features to speed up the job performance:

    * Drop indexes/constraints and rebuild after mass update:  Dropping indexes before a mass update and rebuilding them afterwards can improve update performance significantly. Oracle removes index entries without re-balancing the index tree (a "logical delete"), but this is still time-consuming, especially if you have lots of indexes on the target table.  Also note that you can rebuild the dropped indexes nologging mode.

    * Parallelize the updates:  If you have an SMP server you can run Oracle parallel DML.    You can also manually parallelize the update by breaking them into multiple jobs and submit them simultaneously using dbms_scheduler, cron  or nohup.

Use CTAS in lieu of large updates

When you are updating the majority of rows in a table, using Create Table As Select (CTAS) is often more efficient performance than a standard update.  For example, assume that the following update changed 75% of the table rows:

    update
       mytab
    set
       status = 'new'
    where
       status = 'old;

In this case, a parallelized CTAS may perform far faster (Note: Make sure that you have an SMP server before using the parallel degree option):

    create table new_mytab NOLOGGING as
    select  /*+ full parallel(mytab,35)*/
       decode (status,'new','old',status,
       col2, col3, col4
    from mytab;

    -- rebuild indexes, triggers and constraints to new_mytab

    rename mytab to bkup_mytab;
    rename new_mytab to mytab;
    

Include the SET condition in the WHERE clause

This note shows a case where the developer forgot to include the SET condition in the UPDATE WHERE clause, causing high redo waits (log file parallel write waits, log file sync waits).  Simply including the existing state of the SET clause can result in a huger performance improvement for UPDATE statements:

-- zillion row update
UPDATE HISTORY SET FLAG=0 WHERE CLASS='X'

-- hundred row update
UPDATE HISTORY SET FLAG=0 WHERE CLASS='X' AND FLAG!=0
The select for update is not a good locking strategy because there are many things that can go wrong.  Instead of select for update, savvy Oracle developers will adopt alternatives mechanisms like a re-read upon update commit where the transaction re-read the rows and ensure that it has not changed since the original read.
Have a small, separate data cache for high DML tables

As more people adopt 64-bit servers with giant data buffers, we see a delay caused by the database writer process having to scan through giant data buffers seeking dirty blocks.  Many shops are replacing their platter-style disks with solid-state disks, and creating a very small data buffer, just for the updates.  The book Oracle Tuning: The Definitive Reference notes that if you are still using traditional disks, many shops segregate high-updates objects (tables & indexes) into a separate blocksize so that they can have a separate, small data buffer.

    "By segregating high activity tables into a separate, smaller data buffer, Oracle has far less RAM frames to scan for dirty block, improving the throughput and also reducing CPU consumption. This is especially important for high update tables with more than 100 row changes per second."

MOSC Note:223299.1 also embraces the importance of multiple blocksizes, listing the multiple buffer regions as among the most important tuning parameters in Oracle9i.
Simplify the WHERE clause predicates

The most common issue with updates is the requirement to have a complex SELECT statement is the where clause to identify the rows to be updated.  The best techniques for simplifying UPDATE where clauses include:

    * Predicate pushing
    * Rewriting subqueries as outer joins (if possible)
    * Oracle SQL subquery unnesting
    * Partitioning Pruning
    * Avoid IN and NOT IN - It is a good idea to discourage the use of the NOT IN clause (which invokes a sub-query) and to prefer NOT EXISTS (which invokes a correlated sub-query).

11g R2 New Features

Application Development

Application Express:

• Application Date Format
• Custom Themes
• BLOB Support
• JavaScript Libraries
• Report Printing (Enhanced)
• Forms Conversion
• Security (Enhanced)
• Runtime-Only Install

General Features:

•  Pro*C
•  “WITH HOLD” Option for Cursors
•  8-byte Numeric host Variable
•  Pro*COBOL
•  8-byte Numeric host Variable

JDBC

• TZ Patching
• SecureFile Zero-Copy

OCI

• 8-byte Numeric host Variable

Availability

• Backup and Recovery:
• Automatic Block Repair
• OSB Cloud Module (Amazon Service)
• DUPLICATE
• Without Target Connection
• NOREDO and “UNDO TABLSPACE”
• Tablespace Checks
• TSPITR (Enhanced)
• “SET NEWNAME” format and clauses  

Online Maintenance and Upgrade:

• Edition Based Redefinition
• “CREATE OR REPLACE TYPE” to Allow FORCE
• Fine-Grained Dependencies for Triggers
• IGNORE_ROW_ON_DUPKEY_INDEX Hint (Insert Statement)

 Data Guard:

• Compressed Table (Logical Standby and LogMiner)
• Real-Time Query Lagging
• FAN Support
• 30 Standby Databases

BI and Data Warehousing

Analytics:

• Analytic Functions 2.0
• Recursive WITH

Data Loading:

• EXECUTE Privilege for Directory Objects
• ORACLE_LOADER Access Driver Extended

Partitioning:

• Virtual Columns in PK or FK
• System-Managed Indexes for List Partitions

Performance and Scalability:

• In-Memory Parallel Execution
• Auto POD
• DBMS_PARALLEL_EXECUTE Package
• On-Commit Fast Refresh (Significant Performance Improvement)

Warehouse Builder:

• Mapping Editor
• Advanced Find
• Copy and Paste of Operators and Attributes
• Grouping and Spotlighting of Objects
• Business Intelligence Tool Integration (BI Standard and Enterprise)
• Current Configuration Dropdown in Design Center
• Flat File Imports (Enhanced Support)
• Table Function (Enhanced Support)
• Experts Available in Editor Menu
• Expression Editing in Operator Edit
• Management/UI of Locations Registered (Enhanced)
• Key Lookup Operator (Enhanced)
• Mapping Debugger (Enhanced)
• JDeveloper-Style UI
• Operator References Included in Generated PL/SQL Code
• Quick Mapper
• Repository Browser (Enhanced)
• Simplified Repository Upgrades
• LONG Data Type Data Extraction (Supported)
• Sub-Queries in Join Operator (Supported)

Clustering :RAC in-use

• Cluster Time Service
• Oracle Cluster Registry (Enhancements)
• Voting Disk (Enhancements)
• Grid Plug and Play
• Oracle Restart
• Policy-Based Cluster/Capacity Management
• Clusterware Resource Modeling (Enhanced)
• Role-Separated Management
• Agent Development Framework
• Zero Downtime Patching (Clusterware and RAC)
• Out-of-Place Clusterware Upgrade
• Server Control (SRVCTL)
• Oracle Restart (Support)
• Grid Plug and Play (Support)
• Single Instance Database in a Cluster (Support)
• Universal Connection Pool (UCP)
• Data Guard Integration
• RAC Integration
• JDBC (Support)
• Enterprise Manager
• Clusterware Resource Management
• Provisioning (Clusterware and RAC)
• Grid Plug and Play (Support)
• Oracle Restart (Support)
• Universal Installer
• Removing RAC (Support)
• Installation Rollback
• Oracle Restart Integration
• Out-of-Place Upgrade (Support)
• Fixup Scripts and Grid Infrastructure Checks
• Typical Installation (Option)
• Single Client Access Name (SCAN)
• Downgrade Database Configured with DBControl
• Java API for RAC FAN Events
• Intelligent Platform Management Interface (IPMI) Integration
• ASM Integration with Clusterware
• Configuration Assistants
• New Features (Support)
• Removing RAC (Support)
• Cluster Verification Utility
• New Features (Enhanced)
• Integration into Universal Installer

Database General:

• Flashback Data Archive DDL (Support)
• Instance Caging
• Scheduler:
• E-mail Notification
• File Watcher
• Multiple Destination Jobs
• Remote Database Jobs
• Utilities:
• Data Pump Legacy Mode
• IPv6:
• JDBC Thin Clients (Support)
• JVM and Java Debuggers (Support)
• Smart Flash Cache
• Stored Outline Migration to SQL Plan Management
• Client Result Cache Using Table Annotations (Support)
• 4 KB Sector Disk Drive (Support)

Security

• Audit Data Management:
• Audit Trail Cleanup
• Encryption Key Management
• Tablespace Master Key Rekey

Server Management

• Automatic Storage Management (ASM) for all Data
• ASM Dynamic Volume Manager (DVM)
• ASM Cluster File System (ACFS)
• ASM FS Snapshot
• Oracle Cluster Registry (OCR) and Voting Disk
• ASM Intelligent Data Placement
• ASM Configuration Assistant
• ASM File Access Control
• ASMCMD (Extension)
• Enterprise Manager Integration
• ACFS (Support)
• ASM File Access Control
• EMCA for New RAC Configurations (Support)
• Patch Application with DBConsole (Single-Instance)
• Automatic Patching of Time Stamp w/ Time Zone Data
• Prevent Data Loss with Local Time Zone Data Type
• Segment Creation on Demand
• Zero-Sized Unusable Indexes and Index Partitions
• Metadata SXML Comparison Tool
• Compare Period Report
• Compare SQL Tuning Sets
• Sampling for Active Data Guard
• Exadata Simulation
• Global RAC ASH Report and ADDM Backward Compatability
• Replay capture on MTS and Streams (Support)
• SQL Tuning Advisor (Support)
• Parallel Query (Auto DOP)
• Action Plan Recording
• Export SQL Tuning Set (STS)
• Replay Synchronization Controls

Multimedia and DICOM:

• Extraction of Requested Attributes Only
• Client –Side DICOM Attribute Extraction
• DICOM (Enhancements)
• Watermarking and Image Processing (Enhancements)

Spatial:

• 3D Visualization Framework
• Network Data Model (Enhancements)
• GeoRaster JAVA API
• Raster Reprojections and Ground Control Point-Based Georeferencing
• Routing and Geocoding (Enhancements)

SecureFiles:

• Database File System (DBFS)
• LZO Compression (Support)

XML DB:

• Binary XML (Enhancements)
• XML DB Repository Performance (Enhancements)
• XMLIndex (Enhancements)
• XMLType Partitioning

Oracle JDBC encryption

How to configure encrypted connection to encrypt data over the wire, and check summing, which prevents certain classes of network attacks. .

Preliminary - Gather Information

DBA must have installed the Oracle server software with the Advanced Security option, and the DBA must have configured the database listener to at least allow encrypted sessions and check summing.

Install Oracle Client

The standard Oracle client software is sufficient on the client side. There are no special client editions of the software to consider. Install the client software and test it as per usual.


Configuring the Client for Encryption

Edit your $ORACLE_HOME/network/admin/sqlnet.ora file on the client machine. For this example, we'll assume DBA has setup your instance to use RC4_256 for encryption and MD5 for checksums. We'll also assume you want your connection to be encrypted, or else let the connection attempt fail.

To configure this, add the following lines to your sqlnet.ora :

SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT = (MD5)
SQLNET.ENCRYPTION_TYPES_CLIENT = (RC4_256)
SQLNET.ENCRYPTION_CLIENT = required
SQLNET.CRYPTO_CHECKSUM_CLIENT = required
SQLNET.CRYPTO_SEED = 'insert a random string from 10-70 characters here'

The full set of options for both ENCRYPTION_CLIENT and CRYPTO_CHECKSUM_CLIENT are: accepted, rejected, requested, required

The full set of options for ENCRYPTION_TYPES_CLIENT are: RC4_256, RC4_128, RC4_56, RC4_40, AES256, AES192, AES128, 3DES168, 3DES112, DES, DES40 (Note: Oracle recommends RC4_256 for performance reasons.)

The full set of options for CRYPTO_CHECKSUM_TYPES_CLIENT are: MD5, SHA1

The CRYPTO_SEED must be quoted in single or double quotes.


It's possible to specify more than one encryption or checksum algorithm. To do so, separate them by commas like so:

SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256,RC4_256,DES)

They will be tried in order until a match is found. If no match is found, then the connection is refused (if either client or server require encryption) or allowed but is unencrypted (all other cases). This may be useful if two different instances have a disjoint set of accepted encryption algorithms.

Note: There is no plausible reason we can think of to configure the client or server to always reject encrypted connections. However, if one side is set to reject, and the other to require, the result is what you'd expect: a connection can never be established.

Verifying encryption is working

To verify the encryption is working, it's both tedious and error prone to use a packet sniffer to verify the connect to the Oracle instance is encrypted. It is not trivial to distinguish compressed data streams from encrypted ones.

A more straightforward and reasonably solid approach is to enable client-side tracing and let Oracle's diagnostic information verify if encryption is working. To do this, edit your sqlnet.ora file, and temporarily add the following lines:

TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=(a fully qualified directory to which you have write permissions)

NOTE: On Windows, TRACE_DIRECTORY_CLIENT is reported to be ignored. Instead, look under:

"C:\Documents and Settings\\Oracle\oradiag_\diag\clients\user_\host_<...>\trace\"

Use sqlplus to login to your instance. We recommend not using more elaborate or GUI tools for this test, as many such tools tend to automatically issue a number of queries on login, and will make the trace logs extremely verbose. After you have logged in successfully, logout.

In the TRACE_DIRECTORY_CLIENT, you should see a file named cli_NNNNN.trc, where NNNNN is an integer. (Note for Windows: the trace files are reported to begin with ora_ rather than cli_.) Grep or visually inspect this file, looking for the string 'encryption'. For example, a session that has successfully set up an encrypted connection will contain a line such as:

[24-OCT-2006 10:00:32:087] na_tns: encryption is active, using RC4_256

A cleartext connection would instead look like:

[24-OCT-2006 10:07:34:707] na_tns: encryption is not active

If you are familiar with Oracle client traces, please note encrypted session negotiation will not turn up at trace levels below 10. You must use an ADMIN or SUPPORT trace level.

Once testing is complete, you will probably want to remove TRACE_LEVEL_CLIENT and TRACE_DIRECTORY_CLIENT from your sqlnet.ora file.

Reference:
http://download.oracle.com/docs/cd/B28359_01/java.111/b31224/clntsec.htm#insertedID5

Six reasons to choose NOSQL

NOSQL

Beat performance bottlenecks and quick response time.
Run on clusters of cheap PC servers.
Open Source and no license fee (reduce cost).
Quickly Horizontal scalable
Distributed database
Easy replication support

Next Generation Database Winners

Application users expects fast response time as specially I am seeing new generation doesn't tolerate 3 secs response time which is widely accepted by industry and companies having challenge in meeting customer expectations with large data sets. NOSQL is the perfect fit for this requirement. Distributed, scalable databases are desperately needed these days. There are lots of players in the NOSQL world but Cassandra/HBase is the garnered competitors.

HBase is the more robust database for a majority of use-cases. Cassandra relies mostly on Key-Value pairs for storage, with a table-like structure added to make more robust data structures possible. It’s a fact that far more people are using HBase than Cassandra at this moment, despite both being similarly recent.

HBase values strong consistency and High Availability. Cassandra values Availability and partitioning tolerance. Cassandra may be useful for storage, but not any data processing. HBase is much handier for that data processing and data ware housing.

HBase also has a nice web-based UI that you can use to view cluster status, determine which nodes store various data, and do some other basic operations. Cassandra lacks this web UI as well as a shell, making it harder to operate.

Installation:
Cassandra is only a Ruby gem install away. That’s pretty impressive. HBase we have to do quite a bit of manual configuration and installation and configuration heavy.

Availability:
Cassandra claims that “writes never fail”, whereas in HBase, if a region server is down, writes will be blocked for affected data until the data is redistributed. This rarely happens in practice, of course, but will happen in a large enough cluster. In addition, HBase has a single point-of-failure (the Hadoop NameNode), but that will be less of an issue as Hadoop evolves. HBase does have row locking, however, which Cassandra does not.

Consistency
Apps usually rely on data being accurate and unchanged from the time of access, so the idea of eventual consistency can be a problem. Cassandra, however, has an internal method of resolving up-to-dateness issues with vector clocks — a complex but workable solution where basically the latest timestamp wins. The HBase/BigTable puts the impetus of resolving any consistency conflicts on the application, as everything is stored versioned by timestamp.

Cassandra only supports one table per install. That means you can’t denormalize and duplicate your data to make it more usable in analytical scenarios. Cassandra is really more of a Key Value store than a Data Warehouse. Furthermore, schema changes require a cluster restart.

Replication:
Cassandra uses a P2P sharing model, whereas HBase (the upcoming version) employs more of a data+logs backup method, aka ‘log shipping’.

Conclusion:
If you need highly available writes with only eventual consistency, then Cassandra is a viable candidate for now. However, many apps are not happy with eventual consistency, and it is still lacking many features. Furthermore, even if writes do not fail, there is still cluster downtime associated with even minor schema changes. HBase is more focused on reads, but can handle very high read and write throughput. It’s much more Data Warehouse ready, in addition to serving millions of requests per second. The HBase integration with MapReduce makes it valuable, and versatile.