Azure Databricks Technical Lead

We are looking for a highly skilled Senior Azure Databricks (ADB) Developer to join our Data Engineering team. This role involves developing large-scale batch and streaming data pipelines on Azure Cloud. The ideal candidate will have strong expertise in Python, Databricks Notebooks, Apache Spark (including Structured Streaming), and real-time integration with Kafka. You will work with both relational databases like DB2 and NoSQL systems such as MongoDB, focusing on performance optimization and scalable architecture.
Key Responsibilities:
• Design and Develop: Create real-time and batch data pipelines using Azure Databricks, Apache Spark, and Structured Streaming.
• Data Processing: Write efficient ETL scripts and automate workflows using Python.
• Data Integration: Integrate with various data sources and destinations, including DB2, MongoDB, and other enterprise-grade data systems.
• Performance Optimization: Tune Spark jobs for optimal performance and cost-effective compute usage on Azure.
• Collaboration: Work with platform and architecture teams to ensure secure, scalable, and maintainable cloud data infrastructure.
• CI/CD Support: Implement CI/CD for Databricks pipelines and notebooks using tools like GitHub and Azure DevOps.
• Stakeholder Communication: Interface with product owners, data scientists, and business analysts to translate data requirements into production-ready pipelines