Databricks Data Cleaning & Loading
₹100-400 INR / hour
Posted half a minute ago
•
Ends in 6 days
₹100-400 INR / hour
I have several relational database tables that need to be extracted into Databricks, cleaned, and reshaped before being loaded into a separate database / data-warehouse. The entire workflow should be scripted in Python (PySpark and SQL) and version-controlled in GitHub, with development ideally done from VS Code so that I can review and extend the notebooks locally. Key points you’ll handle • Connect to the source database from Databricks and pull the required tables. • Use PySpark DataFrame APIs and SQL to remove duplicates, harmonise column types, apply business rules, and create the final fact and dimension structures. • Write the transformed data back to the target warehouse in an efficient, partition-friendly format (Parquet or Delta Lake preferred). • Push all notebooks / scripts and any helper modules to a GitHub repo with clear commit messages so that CI can be added later. Acceptance criteria – All transformations run end-to-end in Databricks without manual intervention. – Resulting tables match the agreed schema and row counts between source and target reconcile within 0.1 %. – Code is readable, modular, and PEP 8 compliant; Git history shows logical commits. If this sounds straightforward to you and you’re comfortable with Databricks, PySpark, SQL, GitHub, and VS Code, let’s get started—clean code and reliable loads are what matter most.
Project ID: 40815026
About the project
Open for bidding
Remote project
Active 57 yrs ago
Place your bid
Bid amount Email addressBenefits of bidding on Freelancer
Set your budget and timeframe
It's free to sign up and bid on jobs
内容来自 Freelancer News,版权归原作者所有。
阅读原文 ↗