Databricks Data Cleaning & Loading

₹100-400 INR / hour

Posted half a minute ago

•

Ends in 6 days

₹100-400 INR / hour

I have several relational database tables that need to be extracted into Databricks, cleaned, and reshaped before being loaded into a separate database / data-warehouse. The entire workflow should be scripted in Python (PySpark and SQL) and version-controlled in GitHub, with development ideally done from VS Code so that I can review and extend the notebooks locally. Key points you’ll handle • Connect to the source database from Databricks and pull the required tables. • Use PySpark DataFrame APIs and SQL to remove duplicates, harmonise column types, apply business rules, and create the final fact and dimension structures. • Write the transformed data back to the target warehouse in an efficient, partition-friendly format (Parquet or Delta Lake preferred). • Push all notebooks / scripts and any helper modules to a GitHub repo with clear commit messages so that CI can be added later. Acceptance criteria – All transformations run end-to-end in Databricks without manual intervention. – Resulting tables match the agreed schema and row counts between source and target reconcile within 0.1 %. – Code is readable, modular, and PEP 8 compliant; Git history shows logical commits. If this sounds straightforward to you and you’re comfortable with Databricks, PySpark, SQL, GitHub, and VS Code, let’s get started—clean code and reliable loads are what matter most.

Project ID: 40815026

About the project

Open for bidding

Remote project

Active 57 yrs ago

Place your bid

Bid amount Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

It's free to sign up and bid on jobs

内容来自 Freelancer News,版权归原作者所有。

阅读原文 ↗

把喜欢的内容,留在你的新标签页。

打开我的订阅