{"id":5375,"date":"2024-01-24T09:43:29","date_gmt":"2024-01-24T09:43:29","guid":{"rendered":"https:\/\/test.opensource-db.in\/wp1\/?p=5375"},"modified":"2024-01-24T10:06:13","modified_gmt":"2024-01-24T10:06:13","slug":"open-source-data-engineering-with-postgresql-3","status":"publish","type":"post","link":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/","title":{"rendered":"Open-source Data Engineering with PostgreSQL"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Blog-3: Data Loading with Apache Spark<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">INTRODUCTION:<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Welcome to the next installment of our series on Open-source Data Engineering with PostgreSQL. In this blog, we\u2019ll delve into the practicalities of transforming table data from PostgreSQL into the Parquet format and storing it in an S3 bucket. Leveraging Python, Pandas, Apache Spark, and the power of open-source tools, we aim to guide you through a seamless data transformation process.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Installing Required Libraries:<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Firstly, we ensure that you have the necessary libraries installed to run the following commands to install Pandas, Apache Spark, and boto3<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install pandas\npip install pyspark\npip install pyarrow\npip install psycopg2\npip install sqlalchemy\npip install boto3<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Python code for Transformation:<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the Python code for transforming PostgreSQL data into the Parquet format locally along with the timestamp and compression of data to Gzip<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import pandas as pd\nimport pyarrow as pa\nimport pyarrow.parquet as pq\nfrom sqlalchemy import create_engine\nimport datetime\n\n# Connection parameters for PostgreSQL\ndb_params = {\n    'host': 'localhost',\n    'port': 5432,\n    'database': 'spark_db',\n    'user': 'postgres',\n    'password': 'postgres'\n}\n\n# Table name in PostgreSQL\npostgres_table_name = 'trail_tab'\n\n# Parquet file path\nparquet_output_path = 'table1.parquet'\n\n# Chunk size\nchunk_size = 1000000\n\n# Start measuring time\ntotal_starttime = datetime.datetime.now()\n\n# Create a connection to PostgreSQL\nengine = create_engine(f\"postgresql:\/\/{db_params&#91;'user']}:{db_params&#91;'password']}@{db_params&#91;'host']}:{db_params&#91;'port']}\/{db_params&#91;'database']}\")\n\n# Get the total number of rows in the PostgreSQL table\ntotal_rows = pd.read_sql_query(f\"SELECT COUNT(*) FROM {postgres_table_name}\", engine).iloc&#91;0, 0]\n\n# Initialize an empty DataFrame to hold the combined data\ncombined_df = pd.DataFrame()\n\n# Loop through chunks\nfor chunk_index, chunk_start in enumerate(range(0, total_rows, chunk_size)):\n\n    # Read data from PostgreSQL table into a Pandas DataFrame for the current chunk\n    query = f\"SELECT * FROM {postgres_table_name} OFFSET {chunk_start} LIMIT {chunk_size}\"\n    postgres_df = pd.read_sql_query(query, engine)\n\n    # Concatenate the current chunk to the combined DataFrame\n    combined_df = pd.concat(&#91;combined_df, postgres_df], ignore_index=True)\n\n# Write the combined DataFrame to a single Parquet file\npq.write_table(pa.Table.from_pandas(combined_df), parquet_output_path)\n\n# End measuring time\ntotal_endtime = datetime.datetime.now()\n\n# Calculate and print the total time taken\ntotal_time_taken = total_endtime - total_starttime\nprint(f\"Total time taken: {total_time_taken}\")\nprint(f\"Data from PostgreSQL table '{postgres_table_name}' has been successfully written to a single Parquet file.\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the output for the above code execution:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><img decoding=\"async\" width=\"602\" height=\"44\" src=\"https:\/\/lh7-us.googleusercontent.com\/HHZAA0g71MHiXVCJZloIZCItm0gxMNYMjSlFGBY4CD6777x00o05TJfhGZiqMvxx6-QkNu0s19qab_IWElxA2_9QihaJBFdR1rAqvCFbehpAzMfirH86jlmiEuotpF6eYY6Gu1acK_uPPMudP59llf4\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Python code for transforming PostgreSQL data to S3 bucket:<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code># This Python code to parse PostgreSQL data to S3 in Parquet\nimport pandas as pd\nimport pyarrow as pa\nimport pyarrow.parquet as pq\nfrom sqlalchemy import create_engine\nimport datetime\nimport boto3\nfrom io import BytesIO\n\n# Connection parameters for PostgreSQL\ndb_params = {\n    'host': 'localhost',\n    'port': 5432,\n    'database': 'spark_db',\n    'user': 'postgres',\n    'password': 'postgres'\n}\n\n# Table name in PostgreSQL\npostgres_table_name = 'trail_tab'\n\n# Parquet file path (local temporary path)\nparquet_local_path = 'table1_combined.parquet'\n\n# S3 bucket and key\ns3_bucket = 'table-parquetbucket'\ns3_key = 'table1_combined.parquet'\n\n# Chunk size\nchunk_size = 1000000\n\n# Start measuring time\ntotal_starttime = datetime.datetime.now()\n\n# Create a connection to PostgreSQL\nengine = create_engine(f\"postgresql:\/\/{db_params&#91;'user']}:{db_params&#91;'password']}@{db_params&#91;'host']}:{db_params&#91;'port']}\/{db_params&#91;'database']}\")\n\n# Get the total number of rows in the PostgreSQL table\ntotal_rows = pd.read_sql_query(f\"SELECT COUNT(*) FROM {postgres_table_name}\", engine).iloc&#91;0, 0]\n\n# Initialize an empty DataFrame to hold the combined data\ncombined_df = pd.DataFrame()\n\n# Loop through chunks\nfor chunk_index, chunk_start in enumerate(range(0, total_rows, chunk_size)):\n\n    # Read data from PostgreSQL table into a Pandas DataFrame for the current chunk\n    query = f\"SELECT * FROM {postgres_table_name} OFFSET {chunk_start} LIMIT {chunk_size}\"\n    postgres_df = pd.read_sql_query(query, engine)\n\n    # Concatenate the current chunk to the combined DataFrame\n    combined_df = pd.concat(&#91;combined_df, postgres_df], ignore_index=True)\n\n# Write the combined DataFrame to a local Parquet file\npq.write_table(pa.Table.from_pandas(combined_df), parquet_local_path)\n\n# Upload the local Parquet file to S3\ns3_client = boto3.client('s3', region_name='ap-south-1')\nwith open(parquet_local_path, 'rb') as data:\n    s3_client.upload_fileobj(data, s3_bucket, s3_key)\n\n# End measuring time\ntotal_endtime = datetime.datetime.now()\n\n# Calculate and print the total time taken\ntotal_time_taken = total_endtime - total_starttime\nprint(f\"Total time taken: {total_time_taken}\")\nprint(f\"Data from PostgresSQL '{postgres_table_name}' has been successfully written to the S3 Bucket.\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the output for the above code execution<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><img decoding=\"async\" width=\"602\" height=\"45\" src=\"https:\/\/lh7-us.googleusercontent.com\/KS09FV_w92UfxR1wnz3eHhUMbsENQhegoFDLQgv_1tEWGpMGk44Qq5wZQLGNRffaz_D0TowlFcVXFRSM-ijfGoRsi7tTiXgzD-3s6P4uVU-rAXrsTq5sw62a0YjFHAjzd7zHlwRiDpPbZCvVz8Uh-vc\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Python code for transforming Parquet format data to a new PostgreSQL table :<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code> # This Python code to parse Parquet data to PostgreSQL new table\nimport pandas as pd\nimport pyarrow.parquet as pq\nfrom sqlalchemy import create_engine\nimport datetime\n# Connection parameters for PostgreSQL\ndb_params = {\n    'host': 'localhost',\n    'port': 5432,\n    'database': 'db_1',\n    'user': 'postgres',\n    'password': 'postgres'\n}\n# Table name in PostgreSQL\npostgres_table_name = 'parquetinput'\n# Parquet file path\nparquet_input_path = 'compressedtable1.parquet'\n\n# Chunk size for insertion into PostgreSQL\ninsert_chunk_size = 100000\n# Start measuring time\ntotal_starttime = datetime.datetime.now()\n# Create a connection to PostgreSQL\nengine = create_engine(f\"postgresql:\/\/{db_params&#91;'user']}:{db_params&#91;'password']}@{db_params&#91;'host']}:{db_params&#91;'port']}\/{db_params&#91;'database']}\")\n# Read the Parquet file into a Pandas DataFrame\nparquet_data = pq.read_table(parquet_input_path).to_pandas()\n# Chunked insertion into PostgreSQL table\nfor chunk_start in range(0, len(parquet_data), insert_chunk_size):\n    chunk_end = min(chunk_start + insert_chunk_size, len(parquet_data))\n    chunk_df = parquet_data.iloc&#91;chunk_start:chunk_end]\n   # Use pandas to_sql function to insert the chunk into the PostgreSQL table\n    chunk_df.to_sql(postgres_table_name, engine, if_exists='append', index=False)\n# End measuring time\ntotal_endtime = datetime.datetime.now()\n# Calculate and print the total time taken\ntotal_time_taken = total_endtime - total_starttime\nprint(f\"Total time taken: {total_time_taken}\")\nprint(f\"Parquet data from file '{parquet_input_path}' has been successfully loaded into the new PostgreSQL table '{postgres_table_name}'.\")\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the Output for this code execution:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><img decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/hiLi-oe9wvlvWCzhXGheT5OBgt7jSRVPmR55nzsziXxvA3LTxEVbI3-erI7yxzpHymV03bcjo44PvWw76NHHLZhIAoVCD_CLDmQ54Ds34GjzgOnQeNIgJuzM83V3VpXlc3ufhOogBcwsxBCAMQjKseM\" width=\"602\" height=\"32\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Python code for parsing S3 bucket data back to a PostgreSQL table:<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code># This Python code to parse S3 data back to new PostgreSQL table\nimport pandas as pd\nimport pyarrow.parquet as pq\nfrom sqlalchemy import create_engine\nimport datetime\nimport boto3\nfrom io import BytesIO\n\n# Connection parameters for PostgreSQL\ndb_params = {\n    'host': 'localhost',\n    'port': 5432,\n    'database': 'db_1',\n    'user': 'postgres',\n    'password': 'postgres'\n}\n\n# S3 bucket and key\ns3_bucket = 'table-parquetbucket'\ns3_key = 'table1_combined.parquet'\n\n# Parquet file path (local temporary path)\nparquet_local_path = 'table1_combined.parquet'\n\n# Table name in PostgreSQL\npostgres_table_name = 't1'\n\n# Chunk size for insertion into PostgreSQL\ninsert_chunk_size = 100000\n\n# Start measuring time\ntotal_starttime = datetime.datetime.now()\n\n# Download the Parquet file from S3\ns3_client = boto3.client('s3', region_name='ap-south-1')\ns3_client.download_file(s3_bucket, s3_key, parquet_local_path)\n\n# Read the Parquet file into a Pandas DataFrame\nparquet_data = pq.read_table(parquet_local_path).to_pandas()\n\n# Create a connection to PostgreSQL\nengine = create_engine(f\"postgresql:\/\/{db_params&#91;'user']}:{db_params&#91;'password']}@{db_params&#91;'host']}:{db_params&#91;'port']}\/{db_params&#91;'database']}\")\n\n# Chunked insertion into PostgreSQL table\nfor chunk_start in range(0, len(parquet_data), insert_chunk_size):\n    chunk_end = min(chunk_start + insert_chunk_size, len(parquet_data))\n    chunk_df = parquet_data.iloc&#91;chunk_start:chunk_end]\n\n    # Use pandas to_sql function to insert the chunk into the PostgreSQL table\n    chunk_df.to_sql(postgres_table_name, engine, if_exists='append', index=False)\n\n# End measuring time\ntotal_endtime = datetime.datetime.now()\n\n# Calculate and print the total time taken\ntotal_time_taken = total_endtime - total_starttime\nprint(f\"Total time taken: {total_time_taken}\")\nprint(f\"Parquet data from S3 has been successfully loaded into the new PostgreSQL table '{postgres_table_name}'.\")\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the Output<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/3_JAmxgDgNhXT59eB_2gpC17ut_7Fgv-A_a8t7mGDdogt52mWSSaCr8G7lJOyoCXo0-p_rz1NyD-n9fw6ZGjpxJAUAIeOwbzM0QkH8xt3y2ZxBUlPkFQfhA9od5XFklglulwzaNIQ_9bwtyM4ijH07A\" width=\"602\" height=\"45\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What&#8217;s Next!<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We are in no mood to wrap up this blog series, with the steps outlined in this blog post, you can successfully learn the intricate process of seamlessly transferring data from PostgreSQL to Parquet files, storing them on Amazon S3, and bringing them back into PostgreSQL tables. The journey doesn&#8217;t end here, in our next series we\u2019ll shift our focus to the exciting realm of querying using Apache Drill. Stay tuned as we dive into empowering business analysts with the versatile Apache Drill.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Blog-3: Data Loading with Apache Spark INTRODUCTION: Welcome to the next installment of our series on Open-source Data Engineering with [&hellip;]<\/p>\n","protected":false},"author":10,"featured_media":5385,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[23,42,89],"tags":[107,81,26],"class_list":["post-5375","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-postgresql-14","category-postgresql-15","category-postgresql-16","tag-apache-spark","tag-data-engineering","tag-postgresql"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Open-source Data Engineering with PostgreSQL - OpenSource DB<\/title>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Open-source Data Engineering with PostgreSQL - OpenSource DB\" \/>\n<meta property=\"og:description\" content=\"Blog-3: Data Loading with Apache Spark INTRODUCTION: Welcome to the next installment of our series on Open-source Data Engineering with [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\" \/>\n<meta property=\"og:site_name\" content=\"OpenSource DB\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-24T09:43:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-24T10:06:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"945\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Brahmini Ratnam\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@opensource_db\" \/>\n<meta name=\"twitter:site\" content=\"@opensource_db\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Brahmini Ratnam\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\"},\"author\":{\"name\":\"Brahmini Ratnam\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/867a56696ebca29a93489ed53609f301\"},\"headline\":\"Open-source Data Engineering with PostgreSQL\",\"datePublished\":\"2024-01-24T09:43:29+00:00\",\"dateModified\":\"2024-01-24T10:06:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\"},\"wordCount\":278,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png\",\"keywords\":[\"apache spark\",\"data-engineering\",\"postgresql\"],\"articleSection\":[\"PostgreSQL 14\",\"PostgreSQL 15\",\"PostgreSQL 16\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\",\"name\":\"Open-source Data Engineering with PostgreSQL - OpenSource DB\",\"isPartOf\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png\",\"datePublished\":\"2024-01-24T09:43:29+00:00\",\"dateModified\":\"2024-01-24T10:06:13+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png\",\"contentUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png\",\"width\":1800,\"height\":945},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/test.opensource-db.in\/wp1\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Open-source Data Engineering with PostgreSQL\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#website\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/\",\"name\":\"OpenSource DB\",\"description\":\"Your Trusted OpenSource Databases partner\",\"publisher\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/test.opensource-db.in\/wp1\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#organization\",\"name\":\"OPENSOURCE DB PRIVATE LIMITED\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png\",\"contentUrl\":\"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png\",\"width\":368,\"height\":120,\"caption\":\"OPENSOURCE DB PRIVATE LIMITED\"},\"image\":{\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/\",\"https:\/\/x.com\/opensource_db\",\"https:\/\/www.youtube.com\/channel\/UCmTI5h\",\"https:\/\/www.linkedin.com\/company\/opensource-db\",\"https:\/\/www.instagram.com\/opensource_db\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/867a56696ebca29a93489ed53609f301\",\"name\":\"Brahmini Ratnam\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/05bcc95d3352ddae501c8cdae6b20b8e4e59c373cb7e506042427bca7fe13a3b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/05bcc95d3352ddae501c8cdae6b20b8e4e59c373cb7e506042427bca7fe13a3b?s=96&d=mm&r=g\",\"caption\":\"Brahmini Ratnam\"},\"url\":\"https:\/\/test.opensource-db.in\/wp1\/author\/brahmini-ratnam-meruva\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Open-source Data Engineering with PostgreSQL - OpenSource DB","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Open-source Data Engineering with PostgreSQL - OpenSource DB","og_description":"Blog-3: Data Loading with Apache Spark INTRODUCTION: Welcome to the next installment of our series on Open-source Data Engineering with [&hellip;]","og_url":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/","og_site_name":"OpenSource DB","article_publisher":"https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/","article_published_time":"2024-01-24T09:43:29+00:00","article_modified_time":"2024-01-24T10:06:13+00:00","og_image":[{"width":1800,"height":945,"url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png","type":"image\/png"}],"author":"Brahmini Ratnam","twitter_card":"summary_large_image","twitter_creator":"@opensource_db","twitter_site":"@opensource_db","twitter_misc":{"Written by":"Brahmini Ratnam","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#article","isPartOf":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/"},"author":{"name":"Brahmini Ratnam","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/867a56696ebca29a93489ed53609f301"},"headline":"Open-source Data Engineering with PostgreSQL","datePublished":"2024-01-24T09:43:29+00:00","dateModified":"2024-01-24T10:06:13+00:00","mainEntityOfPage":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/"},"wordCount":278,"commentCount":0,"publisher":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#organization"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage"},"thumbnailUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png","keywords":["apache spark","data-engineering","postgresql"],"articleSection":["PostgreSQL 14","PostgreSQL 15","PostgreSQL 16"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/","url":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/","name":"Open-source Data Engineering with PostgreSQL - OpenSource DB","isPartOf":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#website"},"primaryImageOfPage":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage"},"thumbnailUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png","datePublished":"2024-01-24T09:43:29+00:00","dateModified":"2024-01-24T10:06:13+00:00","breadcrumb":{"@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#primaryimage","url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png","contentUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png","width":1800,"height":945},{"@type":"BreadcrumbList","@id":"https:\/\/test.opensource-db.in\/wp1\/open-source-data-engineering-with-postgresql-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/test.opensource-db.in\/wp1\/"},{"@type":"ListItem","position":2,"name":"Open-source Data Engineering with PostgreSQL"}]},{"@type":"WebSite","@id":"https:\/\/test.opensource-db.in\/wp1\/#website","url":"https:\/\/test.opensource-db.in\/wp1\/","name":"OpenSource DB","description":"Your Trusted OpenSource Databases partner","publisher":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/test.opensource-db.in\/wp1\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/test.opensource-db.in\/wp1\/#organization","name":"OPENSOURCE DB PRIVATE LIMITED","url":"https:\/\/test.opensource-db.in\/wp1\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/","url":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png","contentUrl":"https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2021\/10\/osdb-logo-tm-2.png","width":368,"height":120,"caption":"OPENSOURCE DB PRIVATE LIMITED"},"image":{"@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/people\/OpenSource-DB\/100072970755470\/","https:\/\/x.com\/opensource_db","https:\/\/www.youtube.com\/channel\/UCmTI5h","https:\/\/www.linkedin.com\/company\/opensource-db","https:\/\/www.instagram.com\/opensource_db\/"]},{"@type":"Person","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/867a56696ebca29a93489ed53609f301","name":"Brahmini Ratnam","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/test.opensource-db.in\/wp1\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/05bcc95d3352ddae501c8cdae6b20b8e4e59c373cb7e506042427bca7fe13a3b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/05bcc95d3352ddae501c8cdae6b20b8e4e59c373cb7e506042427bca7fe13a3b?s=96&d=mm&r=g","caption":"Brahmini Ratnam"},"url":"https:\/\/test.opensource-db.in\/wp1\/author\/brahmini-ratnam-meruva\/"}]}},"rttpg_featured_image_url":{"full":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",1800,945,false],"landscape":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",1800,945,false],"portraits":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",1800,945,false],"thumbnail":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1-150x150.png",150,150,true],"medium":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1-300x158.png",300,158,true],"large":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1-1024x538.png",1024,538,true],"1536x1536":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1-1536x806.png",1536,806,true],"2048x2048":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",1800,945,false],"ultp_layout_landscape_large":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",1200,630,false],"ultp_layout_landscape":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",870,457,false],"ultp_layout_portrait":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",600,315,false],"ultp_layout_square":["https:\/\/test.opensource-db.in\/wp1\/wp-content\/uploads\/2024\/01\/image-1-1.png",600,315,false]},"rttpg_author":{"display_name":"Brahmini Ratnam","author_link":"https:\/\/test.opensource-db.in\/wp1\/author\/brahmini-ratnam-meruva\/"},"rttpg_comment":0,"rttpg_category":"<a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-14\/\" rel=\"category tag\">PostgreSQL 14<\/a> <a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-15\/\" rel=\"category tag\">PostgreSQL 15<\/a> <a href=\"https:\/\/test.opensource-db.in\/wp1\/category\/postgres\/postgresql-16\/\" rel=\"category tag\">PostgreSQL 16<\/a>","rttpg_excerpt":"Blog-3: Data Loading with Apache Spark INTRODUCTION: Welcome to the next installment of our series on Open-source Data Engineering with [&hellip;]","_links":{"self":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5375","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/comments?post=5375"}],"version-history":[{"count":8,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5375\/revisions"}],"predecessor-version":[{"id":5389,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/posts\/5375\/revisions\/5389"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/media\/5385"}],"wp:attachment":[{"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/media?parent=5375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/categories?post=5375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/test.opensource-db.in\/wp1\/wp-json\/wp\/v2\/tags?post=5375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}