{"id":5064,"date":"2023-12-27T07:11:06","date_gmt":"2023-12-27T07:11:06","guid":{"rendered":"https:\/\/test.opensource-db.in\/wp1\/?p=5064"},"modified":"2024-01-23T07:40:55","modified_gmt":"2024-01-23T07:40:55","slug":"mastering-timestamp-based-cdc-hurdles-a-proven-solution","status":"publish","type":"post","link":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/","title":{"rendered":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"538\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1024x538.png\" alt=\"\" class=\"wp-image-5078\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1024x538.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-300x158.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-768x403.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1536x806.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-189x99.png 189w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-152x80.png 152w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-394x207.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-915x480.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1240x651.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png 1800w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Introduction<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Have you experimented with Timestamp-Based Change Data Capture using the Pentaho Data Integration (PDI) tool? Achieving data replication from a source database to a target database through &#8220;Timestamp-Based Change Data Capture&#8221; with Pentaho Data Integration is indeed straightforward. Perhaps you&#8217;ve successfully implemented it. However, there are two drawbacks to achieving comprehensive data replication using this method. Firstly, updating the timestamp column in the source table is required each time rows are inserted or updated. Secondly, &#8216;Delete&#8217; operations face a challenge, as there is no means to retrieve the value of the timestamp column in deleted rows.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As far as the first case is concerned, it would be taken care of as long as you incorporate these two practices in your database design:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Include a Date\/Time column to capture the Last Modified \/ Last Updated Timestamp in each and every table (or, at least the ones that you plan to replicate onto another database instance).<\/li>\n\n\n\n<li>Associate the column with an insert or update trigger that automatically updates the values in the column with the current timestamp whenever an insert or update action is performed. (or)<br>Make sure to set\/update the values in the column with the current timestamp value when performing any insert or update operation within your application&#8217;s code.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Coming to the second case, here is the effective and simple solution to get complete data replication with some modifications to the design of the Timestamp based CDC that we discussed in the previous blog. We highly recommend you to follow the series of topics around the Data Integration with PostgreSQL (the World\u2019s Most Advanced Open Source Relational Database System) for better understanding. We have so far explored the Installation of <a href=\"https:\/\/test.opensource-db.in\/wp1\/pentaho-data-integration-with-postgresql\/\">Pentaho Data Integration (PDI)<\/a> tool and connection to the PostgreSQL database, about <a href=\"https:\/\/test.opensource-db.in\/wp1\/unleashing-the-power-of-change-data-capture\/\">Change Data Capture<\/a> and its methods and the implementation of <a href=\"https:\/\/test.opensource-db.in\/wp1\/timestamp-based-change-data-capture\/\">Timestamp based Change Data Capture<\/a>. If you have already checked these out, you&#8217;re good to go!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We are dividing this topic into two parts to make it easier for you to follow-along:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Design of the CDC pipeline.<\/li>\n\n\n\n<li>Execution of the CDC pipeline(next blog)<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Design of the CDC pipeline<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In this pipeline , we will identify the difference in comparison of two tables (one from Source Database and another from Target Database) and apply changes in the Target database accordingly. Before designing this pipeline , we will save the transformation we used to achieve &#8216;Timestamp based CDC&#8217; (previous blog) as &#8216;Timestamp based CDC&#8217; . Open a new transformation. <\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"937\" height=\"446\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1.png\" alt=\"\" class=\"wp-image-5065\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1.png 937w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-300x143.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-768x366.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-208x99.png 208w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-168x80.png 168w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-394x188.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B1-915x436.png 915w\" sizes=\"(max-width: 937px) 100vw, 937px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">As we need two inputs for the second step , we will export the data from Source Database and also Target Database. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Export Data from Source (Step 1.1)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Input \u2192 Table input<\/li>\n\n\n\n<li>Here, we will extract the data in the table(t1) from Source database by giving SQL statements.<\/li>\n\n\n\n<li>Hop the result to the next step 2.0 as input.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"543\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-1024x543.png\" alt=\"\" class=\"wp-image-5066\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-1024x543.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-300x159.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-768x408.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-1536x815.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-187x99.png 187w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-151x80.png 151w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-394x209.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-915x486.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2-1240x658.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B2.png 1596w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Export Data from Target (Step 1.2)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Input \u2192 Table input<\/li>\n\n\n\n<li>Here, we will extract the data in the table(t2) from Target database by giving SQL statements.<\/li>\n\n\n\n<li>Hop the result to the next step 2.0 as input.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"546\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-1024x546.png\" alt=\"\" class=\"wp-image-5067\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-1024x546.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-300x160.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-768x409.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-1536x818.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-186x99.png 186w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-150x80.png 150w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-394x210.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-915x488.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3-1240x661.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B3.png 1597w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Identify the difference in two tables (Step 2.0)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Joins \u2192 Merge rows (diff)<\/li>\n\n\n\n<li>Here, we will compare the field given in both the tables(t1 &amp; t2) and show the output in flagfield column (Temporary column) as Identical, new ,deleted or changed.<\/li>\n\n\n\n<li>Hop the result to the next step 3.0 as input.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"546\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-1024x546.png\" alt=\"\" class=\"wp-image-5068\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-1024x546.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-300x160.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-768x409.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-1536x819.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-186x99.png 186w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-150x80.png 150w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-394x210.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-915x488.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4-1240x661.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B4.png 1595w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Separate the difference (Step 3.0)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Flow \u2192 Filter rows<\/li>\n\n\n\n<li>Here, we will filter the non-identical rows( from flagfield column) from the previous output by giving condition.<\/li>\n\n\n\n<li>Hop the result to the next step 4.0 as input.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"545\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-1024x545.png\" alt=\"\" class=\"wp-image-5069\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-1024x545.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-300x160.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-768x409.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-1536x817.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-186x99.png 186w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-150x80.png 150w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-394x210.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-915x487.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5-1240x660.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B5.png 1598w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Apply changes in target (Step 4.0)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Output \u2192 Delete<\/li>\n\n\n\n<li>Here, we will delete the non-identical rows( from flagfield column) from the previous output.<\/li>\n\n\n\n<li>Hop the result to the next step 5.0 as input.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"546\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-1024x546.png\" alt=\"\" class=\"wp-image-5070\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-1024x546.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-300x160.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-768x409.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-1536x819.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-186x99.png 186w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-150x80.png 150w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-394x210.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-915x488.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6-1240x661.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B6.png 1598w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Timestamp based CDC (Step 5.0)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Design \u2192 Flow \u2192 Transformation executor<\/li>\n\n\n\n<li>Here, we will execute the transformation named &#8216;Timestamp based CDC&#8217; (discussed in previous blog) . <\/li>\n\n\n\n<li>Save and run the whole transformation.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"542\" src=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-1024x542.png\" alt=\"\" class=\"wp-image-5072\" srcset=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-1024x542.png 1024w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-300x159.png 300w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-768x407.png 768w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-1536x814.png 1536w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-187x99.png 187w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-151x80.png 151w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-394x209.png 394w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-915x485.png 915w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8-1240x657.png 1240w, https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/B8.png 1597w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Summary <\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Pentaho Data Integration (PDI) toolset is vast with numerous options built-in. We have explored just a few of those options for addressing the drawbacks of a simple timestamp based Change Data Capture, and effectively replicating &#8216;Delete&#8217; operations from the source to the target databases. Understanding these two aspects is crucial, as Timestamp based is the most used method of Change Data Capture. In this blog post, we explored the design (part1) of the CDC pipeline. In the next post, we&#8217;ll solidify our understanding further on this topic by going through an example for replicating all of the insert, update and delete operations from the source database to the target database.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We highly recommend you to follow the series of topics around the Data Integration with PostgreSQL (the World\u2019s Most Advanced Open Source Relational Database System) for better understanding. We have so far explored the Installation of <a href=\"https:\/\/test.opensource-db.in\/wp1\/pentaho-data-integration-with-postgresql\/\">Pentaho Data Integration (PDI)<\/a> tool and connection to the PostgreSQL database, learnt about <a href=\"https:\/\/test.opensource-db.in\/wp1\/unleashing-the-power-of-change-data-capture\/\">Change Data Capture<\/a> and its methods and the implementation of <a href=\"https:\/\/test.opensource-db.in\/wp1\/timestamp-based-change-data-capture\/\">Timestamp based Change Data Capture<\/a>, and topped it up by adding the ability to handle &#8216;Delete&#8217; operations. Try designing this pipeline, and connect with us if you need any additional information.<br><br>Stay tuned !!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Have you experimented with Timestamp-Based Change Data Capture using the Pentaho Data Integration (PDI) tool? Achieving data replication from [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":5078,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[23,42,89],"tags":[91,81],"class_list":["post-5064","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-postgresql-14","category-postgresql-15","category-postgresql-16","tag-data-integration","tag-data-engineering"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB<\/title>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB\" \/>\n<meta property=\"og:description\" content=\"Introduction Have you experimented with Timestamp-Based Change Data Capture using the Pentaho Data Integration (PDI) tool? Achieving data replication from [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\" \/>\n<meta property=\"og:site_name\" content=\"OpenSource DB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-12-27T07:11:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-23T07:40:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"945\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Venkat Akhil\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@opensource_db\" \/>\n<meta name=\"twitter:site\" content=\"@opensource_db\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Venkat Akhil\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\"},\"author\":{\"name\":\"Venkat Akhil\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/a37b142ecbf953189a4f9209b0b8d328\"},\"headline\":\"Mastering Timestamp-Based CDC Hurdles: A Proven Solution\",\"datePublished\":\"2023-12-27T07:11:06+00:00\",\"dateModified\":\"2024-01-23T07:40:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\"},\"wordCount\":871,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png\",\"keywords\":[\"data integration\",\"data-engineering\"],\"articleSection\":[\"PostgreSQL 14\",\"PostgreSQL 15\",\"PostgreSQL 16\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\",\"name\":\"Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB\",\"isPartOf\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png\",\"datePublished\":\"2023-12-27T07:11:06+00:00\",\"dateModified\":\"2024-01-23T07:40:55+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png\",\"contentUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png\",\"width\":1800,\"height\":945},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/test.opensource-db.in\/wp1\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Mastering Timestamp-Based CDC Hurdles: A Proven Solution\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#website\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/\",\"name\":\"OpenSource DB\",\"description\":\"Your Trusted OpenSource Databases partner\",\"publisher\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/test.opensource-db.in\/wp1\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\",\"name\":\"OPENSOURCE DB PRIVATE LIMITED\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png\",\"contentUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png\",\"width\":368,\"height\":120,\"caption\":\"OPENSOURCE DB PRIVATE LIMITED\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/\",\"https:\/\/x.com\/opensource_db\",\"https:\/\/www.youtube.com\/channel\/UCmTI5h\",\"https:\/\/www.linkedin.com\/company\/opensource-db\",\"https:\/\/www.instagram.com\/opensource_db\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/a37b142ecbf953189a4f9209b0b8d328\",\"name\":\"Venkat Akhil\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/522c0947d21b2c2c89f10698fd9131f9db4110f65c8d02b53d9b4559c7650865?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/522c0947d21b2c2c89f10698fd9131f9db4110f65c8d02b53d9b4559c7650865?s=96&d=mm&r=g\",\"caption\":\"Venkat Akhil\"},\"url\":\"https:\/\/test.opensource-db.in\/wp1\/author\/sudheer-sopensource-db-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB","og_description":"Introduction Have you experimented with Timestamp-Based Change Data Capture using the Pentaho Data Integration (PDI) tool? Achieving data replication from [&hellip;]","og_url":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/","og_site_name":"OpenSource DB","article_publisher":"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/","article_published_time":"2023-12-27T07:11:06+00:00","article_modified_time":"2024-01-23T07:40:55+00:00","og_image":[{"width":1800,"height":945,"url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png","type":"image\/png"}],"author":"Venkat Akhil","twitter_card":"summary_large_image","twitter_creator":"@opensource_db","twitter_site":"@opensource_db","twitter_misc":{"Written by":"Venkat Akhil","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#article","isPartOf":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/"},"author":{"name":"Venkat Akhil","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/a37b142ecbf953189a4f9209b0b8d328"},"headline":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution","datePublished":"2023-12-27T07:11:06+00:00","dateModified":"2024-01-23T07:40:55+00:00","mainEntityOfPage":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/"},"wordCount":871,"commentCount":0,"publisher":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#organization"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage"},"thumbnailUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png","keywords":["data integration","data-engineering"],"articleSection":["PostgreSQL 14","PostgreSQL 15","PostgreSQL 16"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/","url":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/","name":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution - OpenSource DB","isPartOf":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#website"},"primaryImageOfPage":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage"},"thumbnailUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png","datePublished":"2023-12-27T07:11:06+00:00","dateModified":"2024-01-23T07:40:55+00:00","breadcrumb":{"@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#primaryimage","url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png","contentUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png","width":1800,"height":945},{"@type":"BreadcrumbList","@id":"https:\/\/test.opensource-db.in\/wp1\/mastering-timestamp-based-cdc-hurdles-a-proven-solution\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/test.opensource-db.in\/wp1\/"},{"@type":"ListItem","position":2,"name":"Mastering Timestamp-Based CDC Hurdles: A Proven Solution"}]},{"@type":"WebSite","@id":"https:\/\/test.opensource-db.in\/wp1\/#website","url":"https:\/\/test.opensource-db.in\/wp1\/","name":"OpenSource DB","description":"Your Trusted OpenSource Databases partner","publisher":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/test.opensource-db.in\/wp1\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/test.opensource-db.in\/wp1\/#organization","name":"OPENSOURCE DB PRIVATE LIMITED","url":"https:\/\/test.opensource-db.in\/wp1\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/","url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png","contentUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png","width":368,"height":120,"caption":"OPENSOURCE DB PRIVATE LIMITED"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/","https:\/\/x.com\/opensource_db","https:\/\/www.youtube.com\/channel\/UCmTI5h","https:\/\/www.linkedin.com\/company\/opensource-db","https:\/\/www.instagram.com\/opensource_db\/"]},{"@type":"Person","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/a37b142ecbf953189a4f9209b0b8d328","name":"Venkat Akhil","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/522c0947d21b2c2c89f10698fd9131f9db4110f65c8d02b53d9b4559c7650865?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/522c0947d21b2c2c89f10698fd9131f9db4110f65c8d02b53d9b4559c7650865?s=96&d=mm&r=g","caption":"Venkat Akhil"},"url":"https:\/\/test.opensource-db.in\/wp1\/author\/sudheer-sopensource-db-com\/"}]}},"rttpg_featured_image_url":{"full":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",1800,945,false],"landscape":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",1800,945,false],"portraits":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",1800,945,false],"thumbnail":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-150x150.png",150,150,true],"medium":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-300x158.png",300,158,true],"large":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1024x538.png",1024,538,true],"1536x1536":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges-1536x806.png",1536,806,true],"2048x2048":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",1800,945,false],"ultp_layout_landscape_large":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",1200,630,false],"ultp_layout_landscape":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",870,457,false],"ultp_layout_portrait":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",600,315,false],"ultp_layout_square":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2023\/12\/blog-cdc-challenges.png",600,315,false]},"rttpg_author":{"display_name":"Venkat Akhil","author_link":"https:\/\/test.opensource-db.in\/wp1\/author\/sudheer-sopensource-db-com\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-14\/\" rel=\"category tag\">PostgreSQL 14<\/a> <a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-15\/\" rel=\"category tag\">PostgreSQL 15<\/a> <a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-16\/\" rel=\"category tag\">PostgreSQL 16<\/a>","rttpg_excerpt":"Introduction Have you experimented with Timestamp-Based Change Data Capture using the Pentaho Data Integration (PDI) tool? Achieving data replication from [&hellip;]","_links":{"self":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5064","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/comments?post=5064"}],"version-history":[{"count":5,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5064\/revisions"}],"predecessor-version":[{"id":5081,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5064\/revisions\/5081"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/media\/5078"}],"wp:attachment":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/media?parent=5064"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/categories?post=5064"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/tags?post=5064"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}