Monday, March 28, 2011

Duplicating Oracle 11g Database using Active database option


Source DB
===========
Host Name=oradbsrvr1
DB Name=racdb
Oracle Version=11.2.0.1.0
Oracle Home=/u01/oracle/product/11.2.0/db
dbfiles_loc=/u01/oradata/racdb

Target DB
===========
Host Name= oradbsrvr2
DB Name=testdb
Oracle Version=11.2.0.1.0
Oracle Home=/u01/oracle/product/11.2.0/db1
dbfiles_loc=/u02/oradata/testdb

Step1: Creating Init.ora File on target system
=================================================
*.audit_file_dest='/u01/oracle/oraclebase/admin/testdb/adump'
*.compatible='11.2.0'
*.control_files='/u02/oradata/testdb/control01.ctl'
*.db_block_size=8192
*.db_create_file_dest='/u02/oradata'
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='TESTDB'#Modified by RMAN duplicate
*.db_recovery_file_dest_size=2147483648
*.db_recovery_file_dest='/u02/oradata'
*.db_unique_name='TESTDB'#Modified by RMAN duplicate
*.diagnostic_dest='/u01/oracle/oraclebase'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=testdbXDB)'
*.java_pool_size=134217728
*.job_queue_processes=10
*.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=oradbsrvr2)(PORT=1521))'
*.log_buffer=11485660
*.open_cursors=300
*.pga_aggregate_target=298844160
*.processes=600
*.remote_login_passwordfile='EXCLUSIVE'
*.session_cached_cursors=300
*.sga_target=897581056
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.db_file_name_convert='/u01/oradata/racdb/','/u02/oradata/testdb/'
*.log_file_name_convert='/u01/oradata/racdb/','/u02/oradata/testdb/'

Step2: Creating password files
==================================
orapwd file=orapwtestdb password=oracle ignorecase=y entries=5

Step3: Configuring Listener
==============================
First I tried to connect to database without adding any entries

oracle@oradbsrvr2:/appl/oracle>sqlplus sys/oracle@testdb as sysdba 

SQL*Plus: Release 11.2.0.1.0 Production on Mon Mar 28 16:51:46 2011

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

ERROR:
ORA-12528: TNS:listener: all appropriate instances are blocking new connections

After this error I checked the listener status

oracle@oradbsrvr2:/appl/oracle>lsnrctl status    

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 28-MAR-2011 16:52:15

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                21-MAR-2011 19:16:26
Uptime                    6 days 21 hr. 35 min. 49 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      ON
Listener Parameter File   /u01/oracle/product/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/oracle/oraclebase/diag/tnslsnr/oradbsrvr2/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.100)(PORT=1521)))
Services Summary...
Service "testdb" has 1 instance(s).
  Instance "testdb", status BLOCKED, has 1 handler(s) for this service...
The command completed successfully

testdb instance status was blocked so I added sid_list_listener values

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/oracle/product/11.2.0/grid)
      (PROGRAM = extproc)
    )
    (SID_DESC =
      (GLOBAL_DBNAME = testdb)
      (SID_NAME = testdb)
      (ORACLE_HOME = /u01/oracle/product/11.2.0/grid)
    )
  )

Then restarted the listener

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 28-MAR-2011 16:56:41
Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                28-MAR-2011 16:56:37
Uptime                    0 days 0 hr. 0 min. 4 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      ON
Listener Parameter File   /u01/oracle/product/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/oracle/oraclebase/diag/tnslsnr/oradbsrvr2/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.100)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "testdb" has 1 instance(s).
  Instance "testdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Step4: Duplicating Database
===============================

oracle@oradbsrvr2:/home/oracle>rman target sys/oracle@racdb auxiliary sys/oracle@testdb

Recovery Manager: Release 11.2.0.1.0 - Production on Mon Mar 28 18:53:59 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RACDB (DBID=3431901857)
connected to auxiliary database: TESTDB (not mounted)

RMAN> run
2>  {
duplicate target database to testdb from active database;
3> 4>  }

Starting Duplicate Db at 28-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=20 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  db_name =
''RACDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
''TESTDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u02/oradata/testdb/controlfile/control01.ctl';
   alter clone database mount;
}
executing Memory Script
sql statement: alter system set  db_name =  ''RACDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
sql statement: alter system set  db_unique_name =  ''TESTDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
Oracle instance shut down
Oracle instance started

Total System Global Area     893562880 bytes
Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes

Starting backup at 28-MAR-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=93 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/28/2011 18:54:32
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 18:54:32
ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-12154: TNS:could not resolve the connect identifier specified
ORA-17629: Cannot connect to the remote database server

RMAN> exit

This error was occurred because the TNS entries of our auxiliary database were missing on the source database. This is required as we are duplicating the active database and it need connection to auxiliary database. After adding TNS entries, the rman process was started again

oracle@oradbsrvr2:/home/oracle>rman target sys/oracle@racdb auxiliary sys/oracle@testdb

Recovery Manager: Release 11.2.0.1.0 - Production on Mon Mar 28 19:13:57 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RACDB (DBID=3431901857)
connected to auxiliary database: TESTDB (not mounted)

RMAN> run
2>  {
3>  duplicate target database to testdb from active database;
4>  }

Starting Duplicate Db at 28-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=20 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  db_name =
''RACDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
''TESTDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u02/oradata/testdb/controlfile/control01.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''RACDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
sql statement: alter system set  db_unique_name =  ''TESTDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
Oracle instance shut down
Oracle instance started
Total System Global Area     893562880 bytes
Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes

Starting backup at 28-MAR-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=93 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/28/2011 19:14:17
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:14:17
ORA-17628: Oracle error 19505 returned by remote Oracle server

RMAN> exit

This error was occurred because the controlfile location was specified wrongly. After correcting the control file path, the script executed once again

oracle@oradbsrvr2:/home/oracle>rman target sys/oracle@racdb auxiliary sys/oracle@testdb

Recovery Manager: Release 11.2.0.1.0 - Production on Mon Mar 28 19:17:12 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RACDB (DBID=3431901857)
connected to auxiliary database: TESTDB (not mounted)

RMAN> run
2>  {
3>  duplicate target database to testdb from active database;
4>  }

Starting Duplicate Db at 28-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=20 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  db_name =
''RACDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
''TESTDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u02/oradata/testdb/control01.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''RACDB'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''TESTDB'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area     893562880 bytes

Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes

Starting backup at 28-MAR-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=107 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=/u01/oracle/product/11.2.0/db/dbs/snapcf_racdb.f tag=TAG20110328T191731 RECID=4 STAMP=746997451
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 28-MAR-11

database mounted

contents of Memory Script:
{
   set newname for datafile  1 to
"/u02/oradata/racdb/system01.dbf";
   set newname for datafile  2 to
"/u02/oradata/racdb/sysaux01.dbf";
   set newname for datafile  3 to
"/u02/oradata/racdb/undotbs01.dbf";
   set newname for datafile  4 to
"/u02/oradata/racdb/users01.dbf";
   set newname for datafile  5 to
"/u02/oradata/racdb/example01.dbf";
   set newname for datafile  6 to
"/u02/oradata/racdb/user_data01.dbf";
   backup as copy reuse
   datafile  1 auxiliary format
"/u02/oradata/racdb/system01.dbf"   datafile
2 auxiliary format
"/u02/oradata/racdb/sysaux01.dbf"   datafile
3 auxiliary format
"/u02/oradata/racdb/undotbs01.dbf"   datafile
4 auxiliary format
"/u02/oradata/racdb/users01.dbf"   datafile
5 auxiliary format
"/u02/oradata/racdb/example01.dbf"   datafile
6 auxiliary format
"/u02/oradata/racdb/user_data01.dbf"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 28-MAR-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=/u01/oradata/racdb/system01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:42
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=/u01/oradata/racdb/sysaux01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:43
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=/u01/oradata/racdb/example01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:45
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=/u01/oradata/racdb/undotbs01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:46
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=/u01/oradata/racdb/user_data01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:47
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=/u01/oradata/racdb/users01.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/28/2011 19:17:48
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/28/2011 19:17:48
ORA-17628: Oracle error 19505 returned by remote Oracle server

RMAN> exit

This time the duplicate process was failed because the two parameters were not specified in init.ora parameter file:

*.db_file_name_convert='/u01/oradata/racdb/','/u02/oradata/testdb/'
*.log_file_name_convert='/u01/oradata/racdb/','/u02/oradata/testdb/'

After putting these two parameters, the script was started once again

oracle@oradbsrvr2:/home/oracle>rman target sys/oracle@racdb auxiliary sys/oracle@testdb

Recovery Manager: Release 11.2.0.1.0 - Production on Mon Mar 28 19:24:43 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RACDB (DBID=3431901857)
connected to auxiliary database: TESTDB (not mounted)

RMAN> run
2>  {
3>  duplicate target database to testdb from active database;
4>  }

Starting Duplicate Db at 28-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=20 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  db_name =
''RACDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
''TESTDB'' comment=
''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   backup as copy current controlfile auxiliary format  '/u02/oradata/testdb/control01.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''RACDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
sql statement: alter system set  db_unique_name =  ''TESTDB'' comment= ''Modified by RMAN duplicate'' scope=spfile
Oracle instance shut down
Oracle instance started
Total System Global Area     893562880 bytes
Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes
Starting backup at 28-MAR-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=96 device type=DISK
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=/u01/oracle/product/11.2.0/db/dbs/snapcf_racdb.f tag=TAG20110328T192504 RECID=5 STAMP=746997905
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 28-MAR-11

database mounted

contents of Memory Script:
{
   set newname for datafile  1 to
"/u02/oradata/testdb/system01.dbf";
   set newname for datafile  2 to
"/u02/oradata/testdb/sysaux01.dbf";
   set newname for datafile  3 to
"/u02/oradata/testdb/undotbs01.dbf";
   set newname for datafile  4 to
"/u02/oradata/testdb/users01.dbf";
   set newname for datafile  5 to
"/u02/oradata/testdb/example01.dbf";
   set newname for datafile  6 to
"/u02/oradata/testdb/user_data01.dbf";
   backup as copy reuse
   datafile  1 auxiliary format
"/u02/oradata/testdb/system01.dbf"   datafile
2 auxiliary format
"/u02/oradata/testdb/sysaux01.dbf"   datafile
3 auxiliary format
"/u02/oradata/testdb/undotbs01.dbf"   datafile
4 auxiliary format
"/u02/oradata/testdb/users01.dbf"   datafile
5 auxiliary format
"/u02/oradata/testdb/example01.dbf"   datafile
6 auxiliary format
"/u02/oradata/testdb/user_data01.dbf"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 28-MAR-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=/u01/oradata/racdb/system01.dbf
output file name=/u02/oradata/testdb/system01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:06
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=/u01/oradata/racdb/sysaux01.dbf
output file name=/u02/oradata/testdb/sysaux01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:05
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=/u01/oradata/racdb/example01.dbf
output file name=/u02/oradata/testdb/example01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=/u01/oradata/racdb/undotbs01.dbf
output file name=/u02/oradata/testdb/undotbs01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=/u01/oradata/racdb/user_data01.dbf
output file name=/u02/oradata/testdb/user_data01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=/u01/oradata/racdb/users01.dbf
output file name=/u02/oradata/testdb/users01.dbf tag=TAG20110328T192512
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 28-MAR-11

sql statement: alter system archive log current

contents of Memory Script:
{
   backup as copy reuse
   archivelog like  "/u01/flash_recovery/RACDB/archivelog/2011_03_28/o1_mf_1_143_6s26j8mh_.arc" auxiliary format
"/u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_%u_.arc"   ;
   catalog clone recovery area;
   switch clone datafile all;
}
executing Memory Script

Starting backup at 28-MAR-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log copy
input archived log thread=1 sequence=143 RECID=272 STAMP=746998218
output file name=/u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_1am8ciee_.arc RECID=0 STAMP=0
channel ORA_DISK_1: archived log copy complete, elapsed time: 00:00:07
Finished backup at 28-MAR-11

searching for all files in the recovery area

List of Files Unknown to the Database
=====================================
File Name: /u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_1am8ciee_.arc
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_1am8ciee_.arc

datafile 1 switched to datafile copy
input datafile copy RECID=5 STAMP=746998230 file name=/u02/oradata/testdb/system01.dbf
datafile 2 switched to datafile copy
input datafile copy RECID=6 STAMP=746998230 file name=/u02/oradata/testdb/sysaux01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=7 STAMP=746998231 file name=/u02/oradata/testdb/undotbs01.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=8 STAMP=746998231 file name=/u02/oradata/testdb/users01.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=9 STAMP=746998231 file name=/u02/oradata/testdb/example01.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=10 STAMP=746998231 file name=/u02/oradata/testdb/user_data01.dbf

contents of Memory Script:
{
   set until scn  4176301;
   recover
   clone database
    delete archivelog
   ;
}
executing Memory Script

executing command: SET until clause

Starting recover at 28-MAR-11
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=18 device type=DISK

starting media recovery

archived log for thread 1 with sequence 143 is already on disk as file /u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_1am8ciee_.arc
archived log file name=/u02/oradata/TESTDB/archivelog/2011_03_28/o1_mf_1_143_1am8ciee_.arc thread=1 sequence=143
media recovery complete, elapsed time: 00:00:01
Finished recover at 28-MAR-11

contents of Memory Script:
{
   shutdown clone immediate;
   startup clone nomount;
   sql clone "alter system set  db_name =
''TESTDB'' comment=
''Reset to original value by RMAN'' scope=spfile";
   sql clone "alter system reset  db_unique_name scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

database dismounted
Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     893562880 bytes

Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes

sql statement: alter system set  db_name =  ''TESTDB'' comment= ''Reset to original value by RMAN'' scope=spfile

sql statement: alter system reset  db_unique_name scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     893562880 bytes

Fixed Size                     2218512 bytes
Variable Size                343934448 bytes
Database Buffers             532676608 bytes
Redo Buffers                  14733312 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "TESTDB" RESETLOGS ARCHIVELOG
  MAXLOGFILES     16
  MAXLOGMEMBERS      3
  MAXDATAFILES      100
  MAXINSTANCES     8
  MAXLOGHISTORY      292
LOGFILE
  GROUP  1 ( '/u02/oradata/testdb/redo01.log' ) SIZE 50 M  REUSE,
  GROUP  2 ( '/u02/oradata/testdb/redo02.log' ) SIZE 50 M  REUSE,
  GROUP  3 ( '/u02/oradata/testdb/redo03.log' ) SIZE 50 M  REUSE
DATAFILE
  '/u02/oradata/testdb/system01.dbf'
CHARACTER SET WE8MSWIN1252

contents of Memory Script:
{
   set newname for tempfile  1 to
"/u02/oradata/testdb/temp01.dbf";
   switch clone tempfile all;
   catalog clone datafilecopy  "/u02/oradata/testdb/sysaux01.dbf",
"/u02/oradata/testdb/undotbs01.dbf",
"/u02/oradata/testdb/users01.dbf",
"/u02/oradata/testdb/example01.dbf",
"/u02/oradata/testdb/user_data01.dbf";
   switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to /u02/oradata/testdb/temp01.dbf in control file

cataloged datafile copy
datafile copy file name=/u02/oradata/testdb/sysaux01.dbf RECID=1 STAMP=746998255
cataloged datafile copy
datafile copy file name=/u02/oradata/testdb/undotbs01.dbf RECID=2 STAMP=746998255
cataloged datafile copy
datafile copy file name=/u02/oradata/testdb/users01.dbf RECID=3 STAMP=746998255
cataloged datafile copy
datafile copy file name=/u02/oradata/testdb/example01.dbf RECID=4 STAMP=746998255
cataloged datafile copy
datafile copy file name=/u02/oradata/testdb/user_data01.dbf RECID=5 STAMP=746998255

datafile 2 switched to datafile copy
input datafile copy RECID=1 STAMP=746998255 file name=/u02/oradata/testdb/sysaux01.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=2 STAMP=746998255 file name=/u02/oradata/testdb/undotbs01.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=3 STAMP=746998255 file name=/u02/oradata/testdb/users01.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=4 STAMP=746998255 file name=/u02/oradata/testdb/example01.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=5 STAMP=746998255 file name=/u02/oradata/testdb/user_data01.dbf

contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 28-MAR-11

RMAN>exit

This time we were succeeded in duplicating the database with active database option.

Hope this help

Sunday, February 20, 2011

NFS Configuration for Oracle RAC

Today I was going to Install/configure Oracle 11gR2 RAC without using shared storage on Red Hat Enterprise Linux. I had read about that before but never tested so I tried and failed first time but after some re-configuration, I was able to configure clusterware and was able to store OCR Voting and database files on NFS mount points.

Step 1
=========
On First node e.g. RAC node1, create the following directory. You can use any directory as your NFS source mount point.

mkdir /u01/shared_config
chown -R oracle:oinstall /u01/shared_config

Step 2
=========
Add the following lines to the /etc/exports file.

vi /etc/exports
------------------------------------------------
/u01/shared_config  *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

Step 3
========
Run the following command to export the NFS shares.
Check that the NFS server will get started during boot of this server.

chkconfig --level 345 nfs on

Now, start the NFS server process on the NFS server.
service nfs start

Check, if the directory is exported correctly by issuing the exportfs –v command:
/usr/sbin/exportfs -v

Step 4
========
On both node1 and node2 create the directories in which the Oracle configuration files will be created.

mkdir /shared_config
chown -R oracle:oinstall /shared_config
chmod -R 775 /shared_config

Step 5
=======
Add the following lines to the "/etc/fstab" file.
---------------------------------------------------------------
racnode1:/u01/shared_config /shared_config  nfs  rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0  0 0

Note: I used following Oracle recommended settings for nfs share and at the end my ocr and voting disks configuration process was failed:
racnode1:/u01/shared_config /shared_config  nfs  rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 0 0

Step 6
=======
Mount the NFS shares on all servers.

mount /shared_config

Make sure the permissions on the shared directories are correct i.e oracle.
Check, if the NFS export is correctly mounted with the mount command.
# mount
It will give you output like this:
esidbapoc01:/sharedconfig on /shared_config type nfs (rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0,addr=192.168.1.3)


Hope this will help…………

Thursday, January 27, 2011

Remote Telnet on Unix Servers


I have created simple script which will telnet to any unix server and will perform the tasks required.

#Remote_Telnet.sh
------------------------------------
#!/bin/ksh
. ~/.profile

HOST='192.168.1.100'
DELAY=3
USER='oracle'
PASSWORD='oracle'
COMMAND1='cd /home/oracle'

(
sleep 3
print "${USER}"
sleep 3
print "${PASSWORD}"
sleep 3
print "Starting Patch Deployment on ${HOST}"
sleep 1
print "cd /app/dbsrvr"
print ". ./dbenv.sh"
print "cd bin"
print "./stop_server.sh"
print "cd /app/dbserver/patches"
print "opatch apply"
print "cd /app/dbsrvr/bin"
print "./start_server.sh"
sleep 1
print "exit"
sleep 3
) | telnet "${HOST}"$

------------------------------------

Currently I have hard-coded IP and user details but you can change this to enter the values at run time and change the script to accept those values and connect to that specific server.
for example:
#Remote_Telnet.sh `192.168.1.100` oracle password
------------------------------------
#!/bin/ksh
. ~/.profile

HOST='$1'
DELAY=3
USER='$2'
PASSWORD='$3'
------------------------------------

Monday, March 29, 2010

Oracle Applications (EBS) Useful Commands

Hi All, I am compiling some useful commands for Oracle Application 11i and R12. I will try to update the list as I found new commands. Please share the scripts to me so that I may also include those to have online shared library of these useful commands reference.

How to find oracle apps release version

select APPLICATIONS_SYSTEM_NAME,RELEASE_NAME from apps.fnd_product_groups;

How to find Workflow Version embedded in Apps 11i ?

select TEXT from apps.WF_RESOURCES where NAME='WF_VERSION';

How to find version of JDK Installed on Apps ?

There might be multiple JDK installed on Operating System.
Like JDK 1.3.1, 1.4.2 or 1.5 but in order to find which Version of JDK your Apps is using Context File.
Open your Context File $SID_$HOSTNAME.xml under $APPL_TOP/admin and look for variable
JDK_TOP oa_var="s_jdktop" what so ever value assigned against that parameter go to that directory & cd bin & execute command
./java -version so lets assume entry above is /usr/jdk then cd /usr/jdk/bin & ./java -version

Finding Host Details

select     substr(node_name, 1, 20) node_name,
    server_address, substr(host, 1, 15) host,
    substr(domain, 1, 20) domain,
    substr(support_cp, 1, 3) cp,
    substr(support_web, 1, 3)
    web,
    substr(SUPPORT_DB, 1, 3) db,
    substr(VIRTUAL_IP, 1, 30) virtual_ip
from apps.fnd_nodes;

Checking Profile options: (should be enabled)

select DECODE(b.profile_option_value, 'Y', 'Enabled', 'Disabled') DETAILS
from apps.fnd_profile_options a, apps.fnd_profile_option_values b
where a.APPLICATION_ID = b.APPLICATION_ID
and a.PROFILE_OPTION_ID = b.PROFILE_OPTION_ID
and a.PROFILE_OPTION_NAME = 'CONC_GSM_ENABLED';

Checking listener status

lsnrctl status APPS_ABPRD
where ABPRD is my db sid name

Checking Concurrent Processes

select CONCURRENT_QUEUE_NAME,max_processes,running_processes from
apps.FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

How to check if a patch is applied or not ?

--Patch information is stored in AD_BUGS & AD_APPLIED_PATCHES table.
select bug_number from apps.AD_BUGS where bug_number ='&patch_no';

How to confirm form versions

First of all find the location of form and then enter the following command
strings -a $AP_TOP/forms/US/APXINWKB.fmx |grep '$Header'

More commands will be added soon…………………………………….

Here is the list of few important tables.

Concurrent Manager

FND_CONCURRENT_QUEUES
FND_CONCURRENT_PROGRAMS
FND_CONCURRENT_REQUESTS
FND_CONCURRENT_PROCESSES
FND_CONCURRENT_QUEUE_SIZE

FND

FND_APPL_TOPS
FND_LOGINS
FND_USER
FND_DM_NODES
FND_TNS_ALIASES
FND_NODES
FND_RESPONSIBILITY
FND_DATABASES
FND_UNSUCCESSFUL_LOGINS
FND_LANGUAGES
FND_APPLICATION
FND_PROFILE_OPTION_VALUES

AD / Patches

AD_APPLIED_PATCHES
AD_PATCH_DRIVERS
AD_BUGS
AD_INSTALL_PROCESSES
AD_SESSIONS
AD_APPL_TOPS 

 

More commands will be added soon…………………………………….

Tuesday, March 16, 2010

Convert Oracle RAC database to Single Instance

You will find number of articles on My Oracle Support (metalink) and Google for converting single instance database to RAC but its hard to find the documents to convert RAC to single instance.
I will present two methods to convert the RAC to single instance.

i. Converting RAC to single instance on same machine.
ii. Copying the RAC database to another machine and converting that to Single Instance

Converting RAC to single instance on same machine

This is based on Oracle 10G Release 2 and assumes:
1. Oracle RAC running with cluster file system
2. You have basic knowledge about Oracle RAC

Test Server:
OS : Red Hat Enterprise Linux Server release 5.4
Database Version : 10.2.0.4
File system: OCFS2

1. Stop database and CRS on both node
$ srvctl stop database -d RACDB
# crsctl stop crs

2. Turn Off RAC

SQL> startup
ORA-29702 error occurred in Cluster Group Service operation

Relink with the RAC OFF.
$ cd $ORACLE_HOME/rdbms/lib
$ /usr/ccs/bin/make -f ins_rdbms.mk rac_off

Relinking oracle
$ make -f ins_rdbms.mk ioracle
## OR , both working fine
$ cd $ORACLE_HOME/bin
$ relink oracle

If ASM Instance Exist, run below command as root
# /u01/oracle/product/10.2.0/db/bin/localconfig delete
# /u01/oracle/product/10.2.0/db/bin/localconfig add

3.     Parameter(Pfile/spfile) & database changes
SQL> startup
SQL> alter database disable thread 2;
SQL> alter system set remote_listener='';

3a. Remove unwanted logfile
SQL> select thread#, group# from v$log;
SQL> alter database drop logfile group 3;
SQL> alter database drop logfile group 4;

3b. Remove unwanted tablespace
SQL> drop tablespace UNDOTBS2 including contents and datafiles;

3c.    Rename instance name.
SQL> alter system set instance_name=<new_name> scope=spfile;
SQL> shutdown immediate
SQL> startup
- Change your ORACLE_SID environment

4. Run $ORA_CRS_HOME/install/rootdelete.sh on both node
- This will stop and remove all CRS startup related file

5. Remove $ORA_CRS_HOME binary using Clusterware OUI installer
- Ignore any error if 2nd node already down
- rm -rf $ORA_CRS_HOME

6. Modify listener file
$ vi $ORACLE_HOME/network/admin/listener.ora

6a. Modify tnsname file
$ vi $ORACLE_HOME/network/admin/tnsnames.ora

That’s it. You have successfully converted your RAC database to Single Instance on same machine.
Note: You can convert your single instance DB to RAC again by following metalink note:747457.1

Copying the RAC database to another machine and converting that to Single Instance

You often need to copy your production database to test environment to reproduce any defect or to have same replica of your production environment. This way you will required both RAC and single instance setup available in parallel.
I assume that you have following things available on your system.

OS : Red Hat Enterprise Linux Server release 5.4
Database Version : 10.2.0.4
File system: OCFS2 (I will update the document for ASM based RAC database migration in few days)

This will be updated shortly InshaAllah……………….

Monday, March 15, 2010

How to recover dropped table in Oracle 10g using flashback feature

 

How to recover dropped table in Oracle 10g using flashback feature……………

Enter user-name: fkhalid@malik
Enter password:

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP and Data Mining options

STEP 1 : WE ARE GOING TO DROP TABLE test2.

================================
SQL> desc test2;
Name Null? Type
------------ -------- ----------------------------
ID NUMBER(3)

SQL> select * from test2;

ID
----------
1
2
3
4
5
6

6 rows selected.

STEP 2 : NOW DROP THE TABLE.
=====================

SQL> drop table test2;

Table dropped.

SQL> show user
USER is "FKHALID"

SQL> select * from test2;
select * from test2
*
ERROR at line 1:
ORA-00942: table or view does not exist

Now We are confirmed that table has been dropped.

SQL> show user
USER is "FKHALID"

STEP 3 : CHECK WHEATHER THE TABLE IS AVAILABLE IN RECYCLEBIN.
===========================================

SQL> select object_name,original_name from user_recyclebin;

OBJECT_NAME                    ORIGINAL_NAME
------------------------------ --------------------------------
BIN$gGu93nOJ0VrgQ6wUECjRWg==$0 TEST2

STEP 4 : USE THIS FLASHBACK COMMAND TO RECOVER THE DROPPED TABLE.
===============================================

SQL> flashback table test2 to before drop;

Flashback complete.

SQL> select * from test2;

ID
----------
1
2
3
4
5

6 rows selected.

SQL> disc
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL> exit

NOTE: IF THE DROPPED TABLE IS NOT IN RECYCLEBIN THEN YOU CANT USE FLASHBACK FEATURE TO RECOVER THE TABLE.
You need to specify the retention period for flashback logs to retain the tables for longer duration.

Otherwise you'll need a Backup or a dump to restore your table like duplicating your database elsewhere and get back the Table by DataPump (expdp/impdp).

GC Buffer Busy Waits in RAC: Finding Hot Blocks:

 

I was reading following article from a this blog by Jeremy Schneider"
link to the original blog (http://www.ardentperf.com/2007/09/12/gc-buffer-busy-waits-in-rac-finding-hot-blocks/)

Using the ASH

I’ll show two different ways here to arrive at the same conclusion. First, we’ll look a the ASH to see what the sampled sessions today were waiting on. Second, we’ll look at the segment statistics captured by the AWR.

First of all we will get the snap_id’s from dba_hist_snapshot table.

col begin_interval_time format a30
col end_interval_time format a30

select snap_id,begin_interval_time, end_interval_time
from dba_hist_snapshot
where to_char(begin_interval_time,'DD-MON-YYYY')='15-MAR-2010'
order by snap_id;

   SNAP_ID BEGIN_INTERVAL_TIME            END_INTERVAL_TIME
---------- ------------------------------ ------------------------------
      4486 15-MAR-10 12.00.57.293 AM      15-MAR-10 01.00.06.447 AM
      4487 15-MAR-10 01.00.06.447 AM      15-MAR-10 02.00.17.979 AM
      4488 15-MAR-10 02.00.17.979 AM      15-MAR-10 03.00.20.419 AM
      4489 15-MAR-10 03.00.20.419 AM      15-MAR-10 04.00.22.574 AM
      4490 15-MAR-10 04.00.22.574 AM      15-MAR-10 05.00.24.538 AM
      4491 15-MAR-10 05.00.24.538 AM      15-MAR-10 06.00.29.219 AM
      4492 15-MAR-10 06.00.29.219 AM      15-MAR-10 07.00.40.423 AM
      4493 15-MAR-10 07.00.40.423 AM      15-MAR-10 08.00.58.573 AM
      4494 15-MAR-10 08.00.58.573 AM      15-MAR-10 09.00.29.703 AM
      4495 15-MAR-10 09.00.29.703 AM      15-MAR-10 10.00.38.566 AM
      4496 15-MAR-10 10.00.38.566 AM      15-MAR-10 11.00.45.651 AM
      4497 15-MAR-10 11.00.45.651 AM      15-MAR-10 12.00.52.478 PM
      4498 15-MAR-10 12.00.52.478 PM      15-MAR-10 01.00.56.997 PM

This is the window I’m going to use; 12:00am to 12:00pm today.

select wait_class_id, wait_class, count(*) cnt
from dba_hist_active_sess_history
where snap_id between 4486 and 4498
group by wait_class_id, wait_class
order by 3;

WAIT_CLASS_ID WAIT_CLASS CNT
------------- ------------------------------ ----------
3290255840 Configuration 169
2000153315 Network 934
4108307767 System I/O 7199
3386400367 Commit 7809
4217450380 Application 12248
3875070507 Concurrency 14754
1893977003 Other 35499
97762
3871361733 Cluster 104810
1740759767 User I/O 121999


You can see that there were a very large number of cluster events recorded in the ASH. Let’s look a little closer.



select event_id, event, count(*) cnt from dba_hist_active_sess_history
where snap_id between 4486 and 4498 and wait_class_id=3871361733
group by event_id, event
order by 3;

EVENT_ID EVENT CNT
---------- ---------------------------------------- ----------
3905407295 gc current request 4
3785617759 gc current block congested 10
2705335821 gc cr block congested 15
512320954 gc cr request 16
3794703642 gc cr grant congested 17
3897775868 gc current multi block request 17
1742950045 gc current retry 18
1445598276 gc cr disk read 148
1457266432 gc current split 229
2685450749 gc current grant 2-way 290
957917679 gc current block lost 579
737661873 gc cr block 2-way 699
2277737081 gc current grant busy 991
3570184881 gc current block 3-way 1190
3151901526 gc cr block lost 1951
111015833 gc current block 2-way 2078
3046984244 gc cr block 3-way 2107
661121159 gc cr multi block request 4092
3201690383 gc cr grant 2-way 4129
1520064534 gc cr block busy 4576
2701629120 gc current block busy 14379
1478861578 gc buffer busy 67275


Notice the huge gap between the number of buffer busy waits and everything else. Other statistics I checked also confirmed that this wait event was the most significant on the cluster. So now we’ve got an event and we know that 67,275 sessions were waiting on it during ASH snapshots between 12:00am and 12:00pm today. Let’s see what SQL these sessions were executing when they got snapped. In fact lets even include the “gc current block busy” events since there was a bit of a gap for them too.



select sql_id, count(*) cnt from dba_hist_active_sess_history
where snap_id between 4486 and 4498
and event_id in (2701629120, 1478861578)
group by sql_id
having count(*)>1000
order by 2;

SQL_ID CNT
------------- ----------
6kk6ydpp3u8xw 1011
2hvs3mpab5j0w 1022
292jxfuggtsqh 1168
3mcxaqffnzgfw 1226
a36pf34c87x7s 1328
4vs8wgvpfm87w 1390
22ggtj4z9ak3a 1574
gsqhbt5a6d4uv 1744
cyt90uk11a22c 2240
39dtqqpr7ygcw 4251
8v3b2m405atgy 42292


Wow – another big leap – 4,000 to 42,000! Clearly there’s one SQL statement which is the primary culprit. What’s the statement?



select sql_text from dba_hist_sqltext where sql_id='8v3b2m405atgy';

SQL_TEXT
---------------------------------------------------------------------------
insert into bigtable(id, version, client, cl_business_id, cl_order_id, desc


Its an insert statement. Any guesses yet about what the problem might be? Well an insert statement could access a whole host of objects (partitions and indexes)… and even more in this case since there are a good number of triggers on this table. Conveniently, the ASH in 10g records what object is being waited on so we can drill down even to that level.



select count(distinct(current_obj#)) from dba_hist_active_sess_history
where snap_id between 4486 and 4498
and event_id=1478861578 and sql_id='8v3b2m405atgy';

COUNT(DISTINCT(CURRENT_OBJ#))
-----------------------------
14

select current_obj#, count(*) cnt from dba_hist_active_sess_history
where snap_id between 4486 and 4498
and event_id=1478861578 and sql_id='8v3b2m405atgy'
group by current_obj#
order by 2;

CURRENT_OBJ# CNT
------------ ----------
3122841 1
3122868 3
3173166 4
3324924 5
3325122 8
3064307 8
-1 10
3064369 331
0 511
3122795 617
3064433 880
3208619 3913
3208620 5411
3208618 22215


Well a trend is emerging. Another very clear outlier – less than a thousand sessions waiting on most objects but the last one is over twenty-two thousand. Let’s have a look at all three of the biggest ones.



select object_id, owner, object_name, subobject_name, object_type from dba_objects
where object_id in (3208618, 3208619, 3208620);

OBJECT_ID OWNER OBJECT_NAME SUBOBJECT_NAME OBJECT_TYPE
---------- ---------- ------------------------------ ------------------------------ -------------------
3208618 JSCHDER BIGTABLE_LOG P_2007_09 TABLE PARTITION
3208619 JSCHDER BIGTABL_LG_X_ID P_2007_09 INDEX PARTITION
3208620 JSCHDER BIGTABL_LG_X_CHANGE_DATE P_2007_09 INDEX PARTITION


Now wait just a moment… this isn’t even the object we’re updating!! Well I’ll spare you the details but one of the triggers logs every change to BIGTABLE with about 7 inserts into this one. It’s all PL/SQL so we get bind variables and everything – it’s just the sheer number of accesses that is causing all the contention.



One further thing we can do is actually see which blocks are getting most contended for – the ASH records this too. (Isn’t the ASH great?)



select current_file#, current_block#, count(*) cnt
from dba_hist_active_sess_history
where snap_id between 4486 and 4498
and event_id=1478861578 and sql_id='8v3b2m405atgy'
and current_obj# in (3208618, 3208619, 3208620)
group by current_file#, current_block#
having count(*)>50
order by 3;

CURRENT_FILE# CURRENT_BLOCK# CNT
------------- -------------- ----------
1330 238073 51
1542 22645 55
1487 237914 56
1330 238724 61
1330 244129 76
1487 233206 120


One thing that I immediately noticed is that there does not seem to be a single hot block!!! (What?) Out of 40,000 sessions accessing these three objects no more than 120 ever tried to hit the same block. Let’s quickly check if any of these are header blocks on the segments.



select segment_name, header_file, header_block
from dba_segments where owner='JHEIDER' and partition_name='P_2007_09'
and segment_name in ('PLACEMENTS_LOG','PLCMNTS_LG_X_ID',
'PLCMNTS_LG_X_CHANGE_DATE');

SEGMENT_NAME HEADER_FILE HEADER_BLOCK
------------------------------ ----------- ------------
BIGTABL_LG_X_CHANGE_DATE 1207 204809
BIGTABL_LG_X_ID 1207 196617
BIGTABLE_LOG 1209 16393


No – all seem to be data blocks. Why so much contention? Maybe the RAC and OPS experts out there already have some guesses… but first let’s explore one alternative method to check the same thing and see of the numbers line up.



AWR Segment Statistics



Here’s a handy little query I made up the other day to quickly digest any of the segment statistics from the AWR and grab the top objects for the cluster, reporting on each instance.

As an aside, there is a line in the middle that says “GC_BUFFER_BUSY_DELTA”. You can replace that line with any of these values to see the top objects for the corresponding waits during the reporting period:



LOGICAL_READS_DELTA
BUFFER_BUSY_WAITS_DELTA
DB_BLOCK_CHANGES_DELTA
PHYSICAL_READS_DELTA
PHYSICAL_WRITES_DELTA
PHYSICAL_READS_DIRECT_DELTA
PHYSICAL_WRITES_DIRECT_DELTA
ITL_WAITS_DELTA
ROW_LOCK_WAITS_DELTA
GC_CR_BLOCKS_SERVED_DELTA
GC_CU_BLOCKS_SERVED_DELTA
GC_BUFFER_BUSY_DELTA
GC_CR_BLOCKS_RECEIVED_DELTA
GC_CU_BLOCKS_RECEIVED_DELTA
SPACE_USED_DELTA
SPACE_ALLOCATED_DELTA
TABLE_SCANS_DELTA


col object format a60
col i format 99
select * from (
select o.owner||'.'||o.object_name||decode(o.subobject_name,NULL,'','.')||
o.subobject_name||' ['||o.object_type||']' object,
instance_number i, stat
from (
select obj#||'.'||dataobj# obj#, instance_number, sum(
GC_BUFFER_BUSY_DELTA
) stat
from dba_hist_seg_stat
where (snap_id between 12831 and 12838)
and (instance_number between 1 and 6)
group by rollup(obj#||'.'||dataobj#, instance_number)
having obj#||'.'||dataobj# is not null
) s, dba_hist_seg_stat_obj o
where o.dataobj#||'.'||o.obj#=s.obj#
order by max(stat) over (partition by s.obj#) desc,
o.owner||o.object_name||o.subobject_name, nvl(instance_number,0)
) where rownum<=40;

OBJECT I STAT
------------------------------------------------------------ --- ----------
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 2529540
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 1 228292
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 2 309684
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 3 289147
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 4 224155
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 5 1136822
JSCHDER.BIGTABLE_LOG.P_2007_09 [TABLE PARTITION] 6 341440
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 2270221
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 1 220094
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 2 313038
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 3 299509
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 4 217489
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 5 940827
JSCHDER.BIGTABL_LG_X_CHANGE_DATE.P_2007_09 [INDEX PARTITION] 6 279264
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 1793931
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 1 427482
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 2 352305
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 3 398699
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 4 268045
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 5 269230
JSCHDER.BIGTABLE.P_WAREHOUSE [TABLE PARTITION] 6 78170
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION] 771060
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION] 1 162296
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION] 2 231141
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION] 3 220573
JSCHDER.BIGTABL_LG_X_ID.P_2007_09 [INDEX PARTITION] 4 157050
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 393663
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 1 66277
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 2 10364
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 3 6930
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 4 3484
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 5 266722
JSCHDER.BIGTABLE.P_DEACTIVE [TABLE PARTITION] 6 39886
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 276637
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 1 13750
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 2 12207
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 3 23522
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 4 28336
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 5 99704
JSCHDER.BIGTABLE.P_ACTIVE_APPROVED [TABLE PARTITION] 6 99118

40 rows selected.





Now as you can see, these statistics confirm what we observed from the ASH: the top waits in the system are for the BIGTABLE_LOG table. However this also reveals something the ASH didn’t – that the date-based index on the same table is a close second.



The Real Culprit



Any time you see heavy concurrency problems during inserts on table data blocks there should always be one first place to look: space management. Since ancient versions of OPS it has been a well-known fact that freelists are the enemy of concurrency. In this case, that was exactly the culprit.



select distinct tablespace_name from dba_tab_partitions
where table_name='BIGTABLE_LOG';

TABLESPACE_NAME
------------------------------
BIGTABLE_LOG_DATA

select extent_management, allocation_type, segment_space_management
from dba_tablespaces where tablespace_name='BIGTABLE_LOG_DATA';

EXTENT_MAN ALLOCATIO SEGMEN
---------- --------- ------
LOCAL USER MANUAL

SQL> select distinct freelists, freelist_groups from dba_tab_partitions
2 where table_name='BIGTABLE_LOG';

FREELISTS FREELIST_GROUPS
---------- ---------------
1 1


And there you have it. The busiest table on their OLTP RAC system is using MSSM (Manual segment space management) with a single freelist group. I’m pretty sure this could cause contention problems! But in this case it wasn’t quite what I expected. It looks to me like the single freelist itself wasn’t the point of contention – but it was pointing all of the nodes to the same small number of blocks for inserts and these data blocks were getting fought over. But they were probably filling up quickly and so no single block had a large number of waits reported in the ASH. If anyone has another idea to identify the hot blocks then leave a comment and let me know!

Monday, October 26, 2009

Managing control files in ASM

Copying Control File to Other locations
Just a couple of notes on how to create multiple ASM control files and moving control files to a different diskgroup.
Assuming we already have a ASM control file in the DATA diskgroup and we wish to add one to the FRA diskgroup

SQL>show parameter control_files

+DATA/SID/controlfile/current.730.698514673

SQL>alter system set control_files='+DATA','+FRA' scope=spfile;

SQL>shutdown immediate;

SQL>startup nomount;

SQL>exit

rman target

RMAN>restore controlfile from '+DATAHR/SID/CONTROLFILE/current.730.698514673';

SQL>alter database mount;

SQL>alter database open;

SQL>show parameter control_files

+DATA/SID/controlfile/current.730.698514673, +FRAHR/SID/controlfile/current.766.698514673


Moving Control File to Other locations

Here is an example of how to migrate from one ASM diskgroup to another Add the new disk discovery path in (if needed), create a diskgroup and add disks

SQL>alter system set asm_diskstring='/dev/oracle','/dev/oracle_hr/r*';

SQL>create diskgroup DATANEW  external redundancy disk '/dev/oracle_hr/rdisk1','/dev/oracle_hr/rdisk2';

SQL>show parameter control_files

control_files                 string     

+DATA10/SID/controlfile/current.796.695384211


SQL>alter system set CONTROL_FILES = '+DATANEW' scope=spfile

SQL>shutdown

rman target /

RMAN>startup nomount

RMAN>restore controlfile  from  '/+DATA/SID/controlfile/current.796.695384211';

SQL>alter database mount;

SQL>alter database open;

Wednesday, August 12, 2009

Ordering the sequence of execution plan steps

Reading SQL execution plans has always been difficult, but there are some tricks to help determine the correct order that the explain plan steps are executed.

SQL execution plans are interpreted using a preorder traversal (reverse transversal) algorithm which you will see below.  Preorder traversal is a fancy way of saying:

1. That to read an execution plan, look for the innermost indented statement. That is generally the first statement executed.
2. In most cases, if there are two statements at the same level, the first statement is executed first.

In other words, execution plans are read inside-out, starting with the most indented operation. Here are some general rules for reading an explain plan.

1. The first statement is the one that has the most indentation.
2. If two statements appear at the same level of indentation, the top statement is executed first.

To see how this works, take a look at this plan. Which operation is first to executed?

|   0 | SELECT STATEMENT   |      |    10 |   650 |     7  (15)| 00:00:01 |

|*  1 |  HASH JOIN         |      |    10 |   650 |     7  (15)| 00:00:01 |

|   2 |   TABLE ACCESS FULL| JOB  |     4 |   160 |     3   (0)| 00:00:01 |

|   3 |   TABLE ACCESS FULL| EMP  |    10 |   250 |     3   (0)| 00:00:01 |

The answer is that the full table scan operation on the job table will execute first.  Let’s look at another example plan and read it…

  ID  Par Operation

   0      SELECT STATEMENT Optimizer=FIRST_ROWS

   1    0   TABLE ACCESS (BY INDEX ROWID) OF 'EMP'

   2    1     NESTED LOOPS

   3    2       TABLE ACCESS (FULL) OF 'DEPT'

   4    2       INDEX (RANGE SCAN) OF 'IX_EMP_01' (NON-UNIQUE)

 

By reviewing this hierarchy of SQL execution steps, we see that the order of operations is 3,4, 2, 1.

SEQ  ID  Par Operation

      0      SELECT STATEMENT Optimizer=CHOOSE

 3    1    0   TABLE ACCESS (BY INDEX ROWID) OF 'EMP'

 4    2    1     NESTED LOOPS

 2    3    2       TABLE ACCESS (FULL) OF 'DEPT'

 1    4    2       INDEX (RANGE SCAN) OF 'IX_EMP_01' (NON-UNIQUE)

Understanding the sequence of explain plan steps is a critical skill, so let’s try some more examples:

Consider this SQL query:

select
   a.empid,
   a.ename,
   b.dname

from
   emp a,
   dept b

where
   a.deptno=b.deptno;

We get this execution plan:

Execution Plan

   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=40 Card=150000

               Bytes=3300000)

   1    0   HASH JOIN (Cost=40 Card=150000 Bytes=3300000)

   2    1     TABLE ACCESS (FULL) OF 'DEPT' (Cost=2 Card=1 Bytes=10)

   3    1     TABLE ACCESS (FULL) OF 'EMP' (Cost=37 Card=150000 Bytes=

          1800000)

What is the order of operations here? 

Answer:  Execution plan steps are 2, 3, 1

Consider this query:

select
   a.empid,
   a.ename,
   b.dname

from
   emp a,
   dept b

where
   a.deptno=b.deptno;

We get this execution plan:

Execution Plan

   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=864 Card=150000
               Bytes=3300000)

   1    0   HASH JOIN (Cost=864 Card=150000 Bytes=3300000)

   2    1     TABLE ACCESS (BY INDEX ROWID) OF 'DEPT' (Cost=826 Card=1

                   Bytes=10)

   3    2       INDEX (FULL SCAN) OF 'IX_DEPT_01' (NON-UNIQUE) (Cost=26

                     Card=1)

   4    1     TABLE ACCESS (FULL) OF 'EMP' (Cost=37 Card=150000

                    Bytes=1800000)

What is the order of operations here?   

Answer:  Execution plans steps are 3, 2, 4, 1

Here is the same query, but slightly different plan:

select
   a.empid,
   a.ename,
   b.dname

from
   emp a,
   dept b

where
   a.deptno=b.deptno;

We get this execution plan:

Execution Plan

   0      SELECT STATEMENT Optimizer=CHOOSE
                   (Cost=39 Card=150000 Byte=3300000)

   1    0   NESTED LOOPS (Cost=39 Card=150000 Bytes=3300000)

   2    1     TABLE ACCESS (FULL) OF 'DEPT' (Cost=2 Card=1 Bytes=10)

   3    1     TABLE ACCESS (FULL) OF 'EMP'
                   (Cost=37 Card=150000 Bytes=1800000)

What is the order of operations here? 

Answer:  Execution plans steps are 2, 3, 1

Let’s find the SQL execution steps for a three table join:

select
   a.ename,
   a.salary,
   b.dname,
   c.bonus_amount,
   a.salary*c.bonus_amount

from
   emp a,
   dept b,
   bonus c

where
   a.deptno=b.deptno
and
   a.empid=c.empid;

What is the order of operations here?

Execution Plan

   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=168 Card=82 Bytes=3936)

   1    0   TABLE ACCESS (BY INDEX ROWID) OF 'EMP' (Cost=2 Card=1 Bytes=12)

   2    1     NESTED LOOPS (Cost=168 Card=82 Bytes=3936)

   3    2       MERGE JOIN (CARTESIAN) (Cost=4 Card=82 Bytes=2952)

   4    3         TABLE ACCESS (FULL) OF 'DEPT' (Cost=2 Card=1 Bytes=10)

   5    3         BUFFER (SORT) (Cost=2 Card=82 Bytes=2132)

   6    5           TABLE ACCESS (FULL) OF 'BONUS' (Cost=2 Card=82

                         Bytes=2132)

   7    2       INDEX (RANGE SCAN) OF 'IX_EMP_01' (NON-UNIQUE) (Cost=1

                     Card=1)

This is a little tougher…. 

The execution order is 4,6,5,3,7,2,1.

Final Exam!  What are the steps for this execution plan?

Execution Plan

----------------------------------------------------------

   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=2871 Card=2 Bytes=143)

   1    0   UNION-ALL

   2    1     SORT (GROUP BY) (Cost=2003 Card=1 Bytes=59)

   3    2       FILTER

   4    3         HASH JOIN (Cost=1999 Card=1 Bytes=59)

   5    4           INDEX (FAST FULL SCAN) OF 'XIN8OPS_FLT_LEG' (UNIQUE) 

                         (Cost=489 Card=14436 Bytes=404208)

   6    4           INDEX (RANGE SCAN) OF 'XIN3BAG_TAG_FLT_LEG' (UNIQUE)

                         (Cost=1501 Card=10121 Bytes=313751)

   7    1     SORT (GROUP BY) (Cost=868 Card=1 Bytes=84)

   8    7       FILTER

   9    8         NESTED LOOPS (Cost=864 Card=1 Bytes=84)

  10    9           HASH JOIN (Cost=862 Card=1 Bytes=57)

  11   10             INDEX (FAST FULL SCAN) OF 'XIN1SCHED_FLT_LEG' (UNIQUE)

                           (Cost=371 Card=1426 Bytes=41354)

  12   10             INDEX (FAST FULL SCAN) OF 'XIN8OPS_FLT_LEG' (UNIQUE)

                           (Cost=489 Card=14436 Bytes=404208)

  13    9           INDEX (RANGE SCAN) OF 'XIN2BAG_TAG_FLT_LEG' (UNIQUE)

                         (Cost=2 Card=2 Bytes=54)

Answer:  The order of operations is 5, 6, 4, 3, 2, 11, 12, 10, 13, 9, 8, 7, 1.

Hope this help…………………………………..