GoldenGate Veridata for Data Consistency in WLCG Distributed Database Environment
Reaching the end of my internship as an Openlab student at CERN, I think it’s appropriate to write a summary of my project in our blog. I will try to give a good overview for the past weeks.
Our department provides replication services between CERN and some specific Tier-1s worldwide as part of the WLCG project. Until now, data replication was accomplished with Oracle Streams, but we are now adapting a new tool, Oracle GoldenGate. One of the most challenging tasks during replication procedures is checking the data consistency between CERN and the Tier-1s. Until now if data was unsynchronized we had to replicate whole datasets between the databases that were involved. GoldenGate Veridata was proposed by Oracle for this complication and it seems a promising solution.
My project was to evaluate Oracle GoldenGate Veridata for data consistency between CERN’s and the Tier-1s’ databases that we use to store the data from the experiments. What is GoldenGate Veridata and why do we need it? Veridata is an easy-to-use tool that manages to identify divergences between 2 datasets and in the last release is able to repair them. GoldenGate Veridata is cross-platform and is efficient of comparing data between divergent databases without affecting other ongoing operations.
After installation, first impression about GoldenGate Veridata was that is an easy to use tool even for an average user. The menu is well structured and the configuration steps weren’t complicated. Once everything is configured, we have an overview of GoldenGate Veridata’s web UI as the picture bellow.
After some time to familiarize with GoldenGate Veridata’s environment we started the evaluation. In order to evaluate GoldenGate Veridata properly, we tested it through WLCG conditions in different use cases. In the first use case, CERN’s and 4 Tier-1s’ databases were used for the evaluation process. During the testing process data was replicated from CERN to the 4 Tier-1 while simultaneously we were checking the data with GoldenGate Veridata. Various compare and repair jobs were created for a detailed evaluation of GoldenGate Veridata and its possibilities.
After a month of several test workloads the results have been really positive. GoldenGate Veridata was able to check the consistency in detail without issues. There were no complications or failures. Essentially, the testing process of GoldenGate Veridata was considered successful and the adoption in production systems was decided. GoldenGate Veridata was used in production data of ATLAS experiment through CERN and the 4 Tier-1s.
GoldenGate Veridata was examined further in WLCG conditions in data that were being replicated from a Tier-1 to CERN with Oracle Streams. It must be pointed out that GoldenGate Veridata is possible to operate with different replication tools except Oracle GoldenGate.
Around 70 GB of data was checked within 12 hours with a speed of 1.63 MB/s
- 24 Schemas were inspected.
- Around 600 compare pairs were processed.
- 13 compare pairs failed due to misconfiguration.
- No compare pairs were out-of-sync.
Afterwards, we used GoldenGate Veridata between CERN and a Tier-1 while the replication process wasn’t operating. Apparently, we were expecting a lot of discrepancies between the two databases. The results were as we expected. The size of the source was 1460 GB while the size of the target was only 660 GB.
The 660 GB of data was checked within 32 hours with a speed of 5.84 MB/s
- 33 Schemas were inspected.
- 11659 compare pairs were processed.
- 21 compare pairs failed due to misconfiguration.
- 181 compare pairs were out-of-sync.
The speed here was higher because the database was in idle condition and the replication processes between the databases were disabled, so the comparisons were performed faster.
Finally, production migration from Oracle Streams to Oracle GoldenGate was first performed to ATLAS offline and online databases. After intervention had successfully completed and replication was initialized with Oracle GoldenGate, the coherence of production data was investigated with GoldenGate Veridata.
More than 1.16 TB of data was checked within 61 hours with a speed of 5.57 MB/s
- 25 Schemas were inspected.
- More than 7220 compare pairs were processed.
- 6 compare pairs failed due to misconfiguration.
- 1 compare pair out-of-sync. This alert is considered as false due to high transaction rate and the replication latency between the tables.
In the picture below you can see our 3 cases. Notice that while the size of the data increases the time duration increases too.
The speed and the duration of the compare processes are acceptable for our environment due to the high transaction rate between CERN and the Tier-1s. Moreover, we have to keep in mind the distance between CERN and the Tier-1s, so as a result throughput is throttled.
In detail during the evaluation, more than 34888 of tables were processed with 96.2% being synchronized, 2.4% being cancelled, 1.3% being out-of-sync and only 0.2% having failed.
In conclusion, after evaluation of Oracle GoldenGate Veridata, it appears to be a solid solution for data consistency between remote databases. It has proven its accuracy through the out-of-sync alerts, as it was able to recognize the data discrepancies rapidly. From a user’s point of view, GoldenGate Veridata provides a user friendly environment through its web UI, so there are no complications in using or configuring its components.