Oracle Data Guard 12c Switchover Issues

Environment

Primary:

2-nodes 12c RAC
ASM
Super Cluster
Oracle Solaris
Dedicated Network Interface for Data Guard
Dedicated Listener for Data Guard

Standby:

2-nodes 12c RAC
ASM
Super Cluster
Oracle Solaris
Dedicated Network Interface for Data Guard
Dedicated Listener for Data Guard

What is the Issue:

Sometimes, when I do a Switchover I receive an error similar to the following one:

1

But when I checked the environment after this error, I found that the switchover was done successfully and everything is fine.

Sometimes, the switchover proceeds without these error messages.

But in all cases – even with the one with errors – it ended up that the switchover completed successfully in the background.

I tried switchover from primary and from standby and using primary tns connection and using standby tns connection, all combinations may work or may end up with error, but the final result is switchover was complete.

Some Analysis & Research

Review the Log Files:

2

Check the configuration after the switchover:
3
4
5
6

StaticConnectIdentifier

The SERVICE_NAME of the primary database in the StaticConnectIdentifier is : “prod_DGMGRL” , but in the listener.ora it is “PROD_DGMGRL”
I changed it to upper-case in the StaticConnectIdentifier, as follows:

7

I tested the switchover again but the issue is still there.

I did the opposite also, and changed the service_name in the listener.ora to match that in the StaticConnectIdentifier,but this didn’t fix the issue.

Listener Configuration

We have created a dedicated local database listener called LSNR_PROD_DG that uses the dedicated DG network interface.

In the Listener log file of the LSNR_PROD_DG DB local listener I found many errors similar to the following one across all nodes:

8

So, it looks for the _DGB service in the LSNR_PROD_DG listener services.
I found that this service is registered automatically with the local grid listener “LISTENER” on port 1526
Also, it is registered with all the scan listeners on port 1526 across all nodes.
But this service is not registered with our local DB listener LSNR_PROD_DG on port 1540.

So, I added our customized listener to the local_listener entry so as the _DGB could be registered automatically with it:

9

Please note that the above IPs are = node virtual IP + node dedicated DG IP

Now the _DGB services are registered against the LSNR_PROD_DG listener on port 1540

The listener status of the LSNR_PROD_DG listener was changed as follows:
From:

10

To:
11

After switchover to proddr, I changed the same parameters on the new primary:

12

But when you refer to the following MOS document:
Oracle Data Guard Broker and Static Service Registration (Doc ID 1387859.1):
“If there is a requirement that the LOCAL_LISTENER parameter be changed then the Broker configuration needs to be manually updated as follows to restore automatic Broker updates of the StaticConnectIdentifier property values for a database instance.

For a Standby database use the DGMGRL CLI to remove the database from the Broker configuration and then add it back to the configuration.
For a Primary database use the DGMGRL CLI to remove and then recreate the entire broker configuration”

So, if you changed the local_listener for the primary database, we should remove the DG configuration and create it again as follows:

13

After doing the above changes , many errors disappeared from the listener log files, but we still get some errors while switching over to the standby database.

Enable debug option

Now I tried to do the switchover while enabling the debug option for the dgmgrl

14

Let us try to switchover while being in the debug mode

15
16

So, it looks like that the clusterware is trying to restart the old primary database and this process takes time, so, he tried the connection
many times until the database started and he was able to connect. So, it looks like these errors are normal ,but for some reason the dgmgrl exists
– when not using the debug option – exists when he faced the errors. But in the background the clusterware is already restarting the old primary database as a the new standby database.
That is why when I relogined to the dgmgrl and issued the command “show configuration” I found that the switchover completed successfully.
I faced this issue with 12c, I didn’t face similar issue with 11gR2.

Final Fix(workaround)

I found that when I do the switchover from the current primary server and using the current primary database tns string, this issue will not happen.
This may be because restarting the old primary database is the longest step in the switchover process, and I guess connecting local may avoid some delays
and speed up the process and avoid this type of errors.

Now switchover in the two ways is working fine:

17

18

Some Recommendations

– Connect from the primary server using the tns string of the current primary database.
– Service_name in both the StaticConnectIdentifier property and the static listener in listener.ora should match (case sensitive)
– If they are different in case sensitivity, change that in the listener to match the one in the StaticConnectIdentifier property and restart the listener.
– If you did the opposite and changed the one in the StaticConnectIdentifier property to match that in the listener, it will not work.
– You may need to update the local_listener for non-default listeners and ports, but in this case do the change before creating the DG Broker configuration, otherwise you will have to remove the configuration and recreate it again
– Addresses in the Local_Listener & StaticConnectIdentifier property may not be fully matching, but that of listener should be superset.

Donate to support our blogging work

$1.00

Thanks
Ahmed

One thought on “Oracle Data Guard 12c Switchover Issues

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s