Confidential Page 1 of 10
Date Nov'06
DR SWITCHOVER / SWITCHBACK PROCESS
S No ACTIVITY
PRE-SWITCH OVER PROCESS
DESCRIPTION
REMARKS
DR SERVER'S
SWITCH OVER PROCESS
Connect to Primary Database (172.31.1.7)
Stop all Application server's. IP NAME In Short Stop all CITRIX connections. 172.31.1.7 GEMS PRI Stop GEMS listener in Primary DB. 172.31.1.3 GEMSL SBY stop LISTENER (Default) in Primary DB. 172.31.20.12 GEMSR RMT Keep only DG listener's up in all 3 server's (Primay --> lsnrgems Local --> lsnrgemsl Remote --> lsnrgemsr ) . Ensure log_archive_dest_state_1 , log_archive_dest_state_2 & log_archive_dest_state_3 is in 'enable' state in Primary DB. Before issuing switch over commands, ensure that all the 3 server's are in sync. Keep alert log file open in all 3 servers and monitor. COMMANDS Telnet 172.31.1.7 login: oracle password: PRI $ lsnrctl stop GEMS PRI $ lsnrctl stop LISTENER PRI $ sqlplus /as sysdba PRI > show parameter dest To see whether dest_state is enable or defer. PRI> select sequence#,applied from v$archived_log order by 1 ; To check whether all 3 DB's are in sync or not. PRI > select name,switchover_status , database_role from NOTE: Always check the Instance Name v$database ; PRI> alter database backup controlfile to trace; Take the controlfile backup. PRI> alter system switch logfile ; Now one archive will be created and it should be applied in all the 3 servers before you issue the SWITCH over command.
Page 1 of 10
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Confidential Page 2 of 10
Date Nov'06
S No
ACTIVITY
DESCRIPTION
PRI> col DEST_NAME for a20 col error for a20 select dest_id,dest_name , status , error from v$archive_dest ; PRI> alter database commit to switchover to standby with session shutdown ; Telnet 172.31.20.12 login: oracle password: RMT$ sqlplus /as sysdba GEMSR_SBY> alter database recover managed standby database cancel ; GEMSR_SBY> recover standby database ;
REMARKS
To check the status of archive destination.
This will switch Primary DB to StandBy DB.
Connect to Remote Standby Database (172.31.20.12)
GEMSR_SBY> alter database commit to switchover to primary ; SHUTDOWN BOTH THE GEMSR_SBY> shut immediate (172.31.20.12) DATABASE ( i.e New Primary and GEMSR_SBY> exit New Standby also )
Cancelling the automatic recovery mode of SBY DB. Recovering any pending archives mannually. This will make Remote Standby DB as Remote Primary DB .
PRI> shut immediate PRI> exit
( 172.31.1.7)
Startup the New Standby DB(172.31.1.7) in nomount state
Startup the new Primary DB (172.31.20.12)
Shutdown and restart the Local
Telnet 172.31.1.7 login: oracle password: PRI$ sqlplus '/as sysdba' PRI> startup nomount PRI> alter database mount standby database ; PRI> alter database recover managed standby database Keeping it in automatic recovery. disconnect from session ; Telnet 172.31.20.12 login: oracle password: RMT$ sqlplus '/as sysdba' GEMSR_SBY> startup GEMSR_SBY> select name,switchover_status , database_role from v$database ; Telnet 172.31.1.3
Page 2 of 10
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Confidential Page 3 of 10
Date Nov'06
S No
ACTIVITY
Standby Database.
DESCRIPTION
login: oracle password: LSBY$ sqlplus '/as sysdba' GEMSL_SBY> alter database recover managed standby database cancel ; GEMSL_SBY> shut immediate GEMSL_SBY> startup nomount GEMSL_SBY> alter database mount standby database ; GEMSL_SBY> alter database recover managed standby database disconnect from session ;
REMARKS
At New Primary DB (172.31.20.12) GEMSR_SBY> alter system switch logfile;
Duing the switchover process , Local Standby DB goes idle. So you need to restart the process. Now one archive will be created and monitor whether this archive is transmitted through DG listener and applied in all 3 DB's.
POST SWICH OVER PROCESS AT REMOTE SITE
AT REMOTE DB side (Assuming that your Application server & CITRIX server is up & running) So that user's can connect to DR site. (This is to be done only if you are Make the Remote DB's listener up (GEMS, LISTENER) allowing the User's to connect to Remote DB else not required.) Change password in ENVIRON.sh Change password in 2 application server (.DBCON.env) Change password & recompile the fmx in CITRIX (GATIPLL.pll , GATIPLLSIN.pll , GLOGIN.fmb , GLOGINSIN.fmb)
SWITCH BACK COMMANDS
/var/www/apps/conversion/tmp/scratch_2/164802679.doc Page 3 of 10
Confidential Page 4 of 10
Date Nov'06
S No
ACTIVITY
PRE-SWITCH BACK PROCESS AT DR SITE
DESCRIPTION
REMARKS
Stop all Application server's. If it is already started. If it is already started. Stop all CITRIX connections. Stop GEMS listener Stop LISTENER (Default) Keep only DG listener's up in all 3 server's (Primay --> lsnrgems Local --> lsnrgemsl Remote --> lsnrgemsr ) . Ensure log_archive_dest_state_1 , log_archive_dest_state_2 & log_archive_dest_state_3 is in 'enable' state in New Primary DB. Before issuing switch back commands, ensure that all the 3 server's are in sync. Keep alert log file open in all 3 servers and monitor. Connect to New Primary Database Telnet 172.31.20.12 (172.31.20.12) login: oracle password: RMT$ lsnrctl stop GEMS RMT$ lsnrctl stop LISTENER RMT$ sqlplus /as sysdba GEMSR_SBY > show parameter dest To see whether dest_state is enable or defer. GEMSR_SBY> select sequence#,applied from v$archived_log To check whether all 3 DB's are in sync order by 1 ; or not. GEMSR_SBY > select name,switchover_status , database_role NOTE: Always check the Instance Name from v$database ; GEMSR_SBY > alter system switch logfile ; Now one archive will be created and it should be applied in all the 3 servers before you issue the SWITCH over command. GEMSR_SBY > col dest_name for a20 To check the status of archive destination. col error for a20 select dest_id,dest_name , status , error from v$archive_dest ;
/var/www/apps/conversion/tmp/scratch_2/164802679.doc Page 4 of 10
Confidential Page 5 of 10
Date Nov'06
S No
ACTIVITY
DESCRIPTION
REMARKS
This will switch New Primary DB to StandBy DB.
GEMSR_SBY > alter database commit to switchover to standby with session shutdown ; Connect to New Standby Database Telnet 172.31.1.7 (172.31.1.7) login: oracle password: PRI$ sqlplus /as sysdba PRI> alter database recover managed standby database cancel ; PRI> recover standby database ; PRI> alter database commit to switchover to primary ; SHUTDOWN BOTH THE PRI> shut immediate DATABASE ( i.e New Primary and PRI> exit New Standby also )
Cancelling the automatic recovery mode of SBY DB. Recovering any pending archives mannually. This will make New Standby DB as Primary DB .
(172.31.1.7)
GEMSR_SBY> shut immediate ( 172.31.20.12) GEMSR_SBY> exit
Startup the New Standby Telnet 172.31.20.12 DB(172.31.20.12) in nomount state login: oracle password: RMT$ sqlplus '/as sysdba' GEMSR_SBY> startup nomount GEMSR_SBY> alter database mount standby database ; GEMSR_SBY> alter database recover managed standby database disconnect from session ; Startup the New Primary DB Telnet 172.31.1.7 (172.31.1.7) login: oracle password: PRI$ sqlplus '/as sysdba' PRI> startup PRI> select name,switchover_status , database_role from v$database ; Shutdown and restart the Local Telnet 172.31.1.3 Standby Database. (172.31.1.3) login: oracle password: LSBY$ sqlplus '/as sysdba' GEMSL_SBY> alter database recover managed standby
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Keeping it in automatic recovery.
Check the switchover status This DB need to be shutdown and restart its recovery process as during the switchover/switchback process it goes to idle state.
Page 5 of 10
Confidential Page 6 of 9
Date Nov'06
S No
ACTIVITY
DESCRIPTION
database cancel ; GEMSL_SBY> shut immediate GEMSL_SBY> startup nomount GEMSL_SBY> alter database mount standby database ; GEMSL_SBY> alter database recover managed standby database disconnect from session ; PRI> alter system switch logfile;
REMARKS
At New Primary DB (172.31.1.7)
Now one archive will be created and monitor whether this archive is transmitted through DG listener and applied in all 3 DB's.
POST SWICH BACK PROCESS AT PRIMARY SITE (172.31.1.7)
AT PRIMARY DB side Make listener up (GEMS, LISTENER) Start all 5 Application server's Give access to CITRIX user' END OF SWITCHOVER / SWITCHBACK PROCESS
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Page 6 of 9
Confidential Page 7 of 9
Date Nov'06
S No
ACTIVITY GO TO THE STANDBY DATABASE (WHICHEVER SERVER YOU WANT TO BRING UP FROM STANDBY MODE TO PRIMARY , IN CASE OF DISASTER).
DESCRIPTION
REMARKS Note: After issuing the FAIL OVER commands, there will be database incarnation. That means archive log number starts from sequence 1.
FAIL OVER PROCESS
Commands: sql> alter database recover managed standby database cancel ; sql> recover standby database ; sql> alter database activate standby database; sql> shut immediate sql> startup Add temp files as per primary DB.
END OF FAIL OVER PROCESS SOME USEFUL DG COMMANDS
Sql> select name,switchover_status , database_role from To check Switchover status and databsae role. v$database ; Sql> col DEST_NAME for a20 col error for a20 select dest_id,dest_name , status , error from v$archive_dest ; Sql> select sequence#,applied from v$archived_log order by 1 ; Sql> alter database create standby controlfile as 'testdrsby.ctl';
To check the errors in transmitting the archive from primary to standby and also to check the status of archive destination.
To check the archive sequence and applied to standby . To create standby conrol file from primary database in case standby control file is lost OR you want to make a standby database by using Hot or Cold backup of primary DB.
HOW TO SHUT DOWN STANDBY DATABASE:
/var/www/apps/conversion/tmp/scratch_2/164802679.doc Page 7 of 9
Confidential Page 8 of 9
Date Nov'06
S No
ACTIVITY
DESCRIPTION
REMARKS
sql> alter database recover managed standby database cancel ; sql> shut immediate;
HOW TO START STANDBY DATABASE
sql> startup nomount sql> alter database mount standby database ; sql> alter database recover managed standby database disconnect from session ; OR sql> recover automatic standby database;
Bringing the standby DB to mount mode. Putting it in DG taken care automatic recovery mode. Putting it in a manual recovery . (Especially useful when you do manual RCP and then Recovery.
CASE - WHEN YOU WANT TO DEFER REMOTE DB PROCESS
STEPS :1) Telnet to Primary DB 2) Defer the Remote DB Start manually transmitting the archives to Remote DB by RCP command.
This situation may arise especially when you Rebuild the Indexes OR when you start Purging Job. This will stop transmitting & applying the Archives to Remote DB.
3) Once 100 archive is RCP'd to Remote DB, start applying it manually in Remote DB.
Telnet to 172.31.1.7 as oracle user. $ sqlplus '/as sysdba' sql> alter system set log_archive_dest_state_3='defer' scope=both; Go to the Archive destination in Primary DB from where you want to start RCP eg: $ rcp <arch_1_xxxx*.arc> 172.31.20.12:/g09/app/oracle/oradata/GEMS/archive/ & Telnet to 172.31.20.12 as oracle user. $ sqlplus '/as sysdba' sql> alter database recover managed standby database cancel ; sql> recover automatic standby database; You can restart this recovery process once another 100 archives been RCP'd to Remote DB.
Manually copying by RCP command.
Canceling the automatic recovery. Starting manual recovery.
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Page 8 of 9
Confidential Page 9 of 9
Date Nov'06
S No
ACTIVITY
DESCRIPTION
REMARKS
Once the Remote DB came closer to the sync mode, put it in Automatic Recovery mode (172.31.20.12) : sql> alter database recover managed standby database Putting back to automatic recovery mode. disconnect from session ; Enabling the Remote DB's destination , so Make destination enable in Primary DB (172.31.1.7) : sql> alter system set log_archive_dest_state_3='enable' that DG will take care of it. scope=both;
/var/www/apps/conversion/tmp/scratch_2/164802679.doc
Page 9 of 9