Dbvisit : PDBs creation and switchover test

Hi there,

today, i decided to share my discoveries with a nice product called Dbvisit Standby.

My new company (Digora) is a Dbvisit partner so i was able to retrieve a license for version 9.0 with snapshot option.

Dbvisit is mainly used with Oracle Standard Edition to have a taste of automated Dataguard.

Snapshot DB is not the purpose of this review (but it is working as expected if your fs is able to handle snapshot).

You can have a clone opened read/write for testing or reporting.

OK,

but i still want to try the creation of new PDBs and the switchover functionality of Dbvisit.

My installation consists of 2 VMs under Virtualbox (2vCPus 8GBs of RAM)

one is named hol and the other is sandbox.

I have Oracle EE 19.5 installed on both (i know i should have SE instead).

I have one CDB (CDB2) with one PDB (PDB3).

hol : CDB2 / CDB2 as service_name

sandbox : CDB2/ CDB2SB as service_name

I do not explain the installation of Dbvisit and the creation of standby with Dbvisit cli (dbvctl) as you can find very good examples on Dbvisit site.

At start, i have a working Oracle primary/standby configuration.

You can check on primary (hol) that there is no trouble with :

dbvctl -d CDB2 -i (Yes my DDC is CDB2)


=============================================================

Dbvisit Standby Database Technology (9.0.08_0_g99a272b) (pid 17935)

dbvctl started on hol: Tue Dec 31 13:53:27 2019

=============================================================




Dbvisit Standby log gap report for CDB2 at 201912311353:

-------------------------------------------------------------

Description    | SCN     | Timestamp

-------------------------------------------------------------

Source       6301412    2019-12-31:13:53:27 +01:00

Destination     6299877    2019-12-31:13:26:39 +01:00




Standby database time lag (DAYS-HH:MI:SS): +00:26:48




Report for Thread 1

-------------------

SOURCE

Current Sequence 320

Last Archived Sequence 319

Last Transferred Sequence 319

Last Transferred Timestamp 2019-12-31 13:26:44




DESTINATION

Recovery Sequence 320




Transfer Log Gap 0

Apply Log Gap 0




=============================================================

dbvctl ended on hol: Tue Dec 31 13:53:30 2019

=============================================================

To have the apply/transfer automatically scheduled, i executed on both nodes :

dbvctl -d CDB2 -D start
Starting Dbvisit Daemon... 

Started successfully. 

You can check the status with :

dbvctl -d CDB2 -D status


Dbvisit Daemon process is running with pid 19760. 

If you want to stop the apply/transfer :

dbvctl -d CDB2 -D stop


Stopping Dbvisit Daemon... 

Successfully stopped. 

Well,

i still want to try the creation of new PDBs as with Oracle 19c,

we have now the right to use 3 PDBs without licensing for multitenant options.

I have one PDB unplugged self contained waiting somewhere in a tmp directory,

let's try to plug it to CDB2 on primary.

On primary (hol) :

create pluggable database pdb1 using '/home/oracle/tmp/cdb2pdb1.pdb'

file_name_convert=('/home/oracle/tmp','/u02/oradata/CDB2/pdb1');

alter pluggable database pdb1 open;

I checked on alert.log of CDB2 and PDB1 is successfully created and opened.

After several minutes (my schedule is 10 mins),

I checked on primary if apply/transfer was still working.

dbvctl -d CDB2 -i

That was not the case, i had an apply gap, so something went wrong on standby side.

I checked on sandbox (standby) to see what happened with my PDB creation through Dbvisit.

Inside alert.log, i can see a lof of messages like this one :

ALTER DATABASE RECOVER    LOGFILE '/u02/oralog/dbvisit/CDB2/1_215_1006733922.arc'  
Mon Dec 23 13:32:48 2019
Media Recovery Log /u02/oralog/dbvisit/CDB2/1_215_1006733922.arc
Recovery created pluggable database PDB1
PDB1(3):File #43 added to control file as 'UNNAMED00043' because
PDB1(3):the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
PDB1(3):The file should be manually created to continue.
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.34).
Datafiles are recovered to a consistent state at change 5913815 but controlfile could be ahead of datafiles.
stopping change tracking
Mon Dec 23 13:32:48 2019
Media Recovery failed with error 7202
ORA-283 signalled during: ALTER DATABASE RECOVER    LOGFILE '/u02/oralog/dbvisit/CDB2/1_215_1006733922.arc'  ...
ALTER DATABASE RECOVER CANCEL 
ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...
Mon Dec 23 13:33:48 2019
PDB1(3):ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'
PDB1(3):ORA-1524 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'...
Mon Dec 23 13:48:54 2019
PDB1(3):ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'
PDB1(3):ORA-1524 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'
...
Mon Dec 23 13:52:33 2019
PDB1(3):ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'
PDB1(3):ORA-1524 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'...
Mon Dec 23 14:08:06 2019
ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'
ORA-1516 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/system01.dbf'...
Mon Dec 23 14:08:39 2019
ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'
ORA-1516 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'...
Mon Dec 23 14:09:17 2019
PDB1(3):ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'
PDB1(3):ORA-1119 signalled during: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'...
Mon Dec 23 14:10:21 2019
PDB1(3):ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'
PDB1(3):Completed: ALTER DATABASE CREATE DATAFILE '/u01/app/oracle/product/19.0/dbhome_1/dbs/UNNAMED00043' AS '/u02/oradata/CDB2/pdb1/system01.dbf'
Mon Dec 23 14:12:36 2019
ALTER DATABASE RECOVER  standby database  NOPARALLEL  
Media Recovery Start
Mon Dec 23 14:12:36 2019
Serial Media Recovery started
.... (PID:17278): Managed Standby Recovery not using Real Time Apply
max_pdb is 6
ORA-279 signalled during: ALTER DATABASE RECOVER  standby database  NOPARALLEL  ...
ALTER DATABASE RECOVER    CANCEL  
...

It seems that it is not able to create the datafiles in the right directory.

They should be created under /u02/oradata/CDB2/pdb1 and not under /u02/oradata/CDB2.

Ok i can correct it by myself by executing several commands like :

alter database create datafile '..../UNNAMED00043' as '/u02/oradata/CDB2/pdb1/system01.dbf'

alter database create datafile '..../UNNAMED00044' as '/u02/oradata/CDB2/pdb1/sysaux01.dbf'

etc

I had to retrieve the file number and the file name from the primary.

But even with that workaround, the apply was not working are recovery process crashed with a coredump.

Hmmm, that sounds not good.

Doing the alter database recover standby database noparallel and the recover logfile specified,

it crashed with an ORA-10560 and ORA-00600 [4552].

It seems to be corrupted somewhere.

* ERROR at line 1: ORA-00283: recovery session canceled due to errors 
ORA-10562: Error occurred while applying redo to data block (file# 34, block# 225) ORA-10564: tablespace SYSTEM 
ORA-01110: data file 34: '/u02/oradata/CDB2/DBVISIT/system01.dbf' 
ORA-10560: block type '0' 
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []

So it needs recovery.

rman target /

restore pluggable database PDB1 from service CDB2_SRC;

After restoring the PDB, let's start again the apply process. (dbvctl -d CDB2 -D start)

Why did i restore the PDB ? Because all the datafiles on the PDB created had the same trouble.

Looking in alert.log, i can see now that logfile are applied and checking the gap status on primary is OK now.

So after that test, i can tell that creation of PDBs is not handled correctly.

I have opened a SR at Dbvisit support and they confirmed that the creation of PDBs is not supported yet (but it will be very soon).

Ok let's start the second test :

i wanted to try a switchover.

Now, that all is ok.

Let's try it.

First of all, we have to check if there is no gap.

dbvctl -d CDB2 -i (on primary)

if the result is Transfer Log Gap=0 and apply Log Gap=0 then it is ok to switchover.

Let's try this command :

dbvctl -d CDB2 -f switchover_precheck




Running pre-checks    ... 

<<<< Dbvisit Standby terminated >>>> 

PID:21983 

TRACEFILE:21983_dbvctl_f_switchover_precheck_CDB2_201912311451.trc 

SERVER:hol 

ERROR_CODE:1 

Remote execution error on sandbox. 




=====================Remote Output start: sandbox===================== 

<<<< Dbvisit Standby terminated >>>> 

PID:16563 

TRACEFILE:16563_dbvctl_f_gs_precheck_standby_CDB2_201912311452.trc 

SERVER:sandbox 

ERROR_CODE:932 

Dbvisit Daemon is running on sandbox. Please stop it using a command "dbvctl -d CDB2 -D 

stop" 

>>>> Dbvisit Standby terminated <<<< 

======================Remote Output end: sandbox====================== 




>>>> Dbvisit Standby terminated <<<< 

Cool it tells you that you have to stop apply/transfer process.

On both nodes, do :

dbvctl -d CDB2 -D stop

And then retry the precheck command :

dbvctl -d CDB2 -f switchover_precheck




Running pre-checks    ... done 




On completion of Gracefull Switchover new primary database will have tempfiles and redologs as following:




REDOLOGS

Group 1: /u02/oradata/CDB2/redo01.log

Group 2: /u02/oradata/CDB2/redo02.log

Group 3: /u02/oradata/CDB2/redo03.log




TEMPFILES

Id 1: /u02/oradata/CDB2/temp01.dbf

Id 2: /u02/oradata/CDB2/pdbseed/temp01.dbf

Id 3: /u02/oradata/CDB2/pdb1/temp01.dbf

Id 4: /u02/oradata/CDB2/pdb3/temp01.dbf

Id 5: /u02/oradata/CDB2/pdb1copy/temp01.dbf

Yes it seems that we can switchover.

Let's do it (on primary) :

dbvctl -d CDB2 -o switchover

After several seconds,

i see :

Processing primary ... failed

Performing rollback ...

and it stopped with ORA-24964 error.

ORA-24964 means that Dbvisit was not able to alter session set container=PDB1.

But it is working perfectly with SQL*Plus.

There is a MOS note (2494623.1) for the same error.

And it seems that it is due to the client used.

PDB1 was an 11.2.0.4 upgraded to 19c and then described/plugged as a PDB into a new 19c CDB (non_cdb_to_pdb.sql).

The MOS note matches my case.

As i'm Bob the builder,

i decided to download the instant-client 12.2 and replace all the libraries under <DBVISIT_HOME>/standby/lib with the new ones from 12.2 on both nodes.

Dbvisit has 12.1 as libraries for client.

I restarted the apply/transfer on both nodes and made some switch logfiles.

Check the apply lag : OK

So we can stop again.

Precheck switchover : OK

Let's try switchover now :

dbvctl -d CDB2 -o switchover

Guess what ! it worked.

dbvctl -d CDB2 -o switchover
=============================================================
Dbvisit Standby Database Technology (9.0.08_0_g99a272b) (pid 13092)
dbvctl started on hol: Tue Dec 31 12:23:55 2019
=============================================================


>>> Starting Switchover between sandbox and hol 


Running pre-checks       ... done 
Pre processing           ... done 
Processing primary       ... done 
Processing standby       ... done 
Converting standby       ... done 
Converting primary       ... done 
Completing               ... done 
Synchronizing            ... done 
Post processing          ... done 


>>> Graceful switchover completed. 
    Primary Database Server: sandbox 
    Standby Database Server: hol


>>> Dbvisit Standby can be run as per normal: 
    dbvctl -d CDB2 


PID:13092
TRACE:13092_dbvctl_switchover_CDB2_201912311223.trc


=============================================================
dbvctl ended on sandbox: Tue Dec 31 12:27:51 2019
=============================================================

I have asked again the Dbvisit support if it was supported to change the library version, and they answered yes.

I can say that Dbvisit support is very reactive.

So if you are like me, with strange upgrade paths or with strange errors, do not hesitate to change the library version.

Voilà,

Hope it helps !