ETL + Databricks

Job Description for your review:

• Design and implement scalable data pipelines and architectures on Azure Databricks.
• Optimize ETL/ELT workflows, ensuring efficiency in data processing, storage, and retrieval.
• Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions.
• Ensure best practices in data governance, security, and compliance within Azure environments.
• Troubleshoot and fine-tune Spark jobs for optimal performance and cost efficiency.
• Lead the migration of Azure SQL to Azure Databricks, ensuring a seamless transition of data workloads.
• Design and implement scalable data pipelines to extract, transform, and load (ETL/ELT) data from Azure SQL into Databricks Delta Lake.
• Optimize Azure SQL queries and indexing strategies before migration to enhance performance in Databricks