Key Responsibilities:
• Design, develop, and maintain robust ETL pipelines using Databricks and Apache Spark.
• Optimize big data workflows to improve performance and reduce processing times.
• Architect and implement scalable data solutions on cloud platforms
• Ensure data quality and governance by implementing best practices and frameworks.
• Mentor and guide junior engineers, providing technical leadership and support.
• Participate in code reviews and contribute to the continuous improvement of development processes.
Required Skills and Qualifications:
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with a focus on Databricks and Apache Spark.
• Proficiency in programming languages such as Python and SQL.
• Strong understanding of cloud platforms and tools, particularly AWS, Databricks,.
• Experience with data warehousing, ETL pipelines, and data orchestration.
• Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
• Strong communication and interpersonal skills.
• Experience with Git.
Preferred Qualifications:
• Experience with real-time data ingestion and streaming using tools like Apache Kafka.
• Knowledge of machine learning frameworks and model deployment strategies.
• Experience with CI/CD pipelines and agile development methodologies.
• Familiarity with data visualization tools such as Power BI , Tableau.
Bachelor's or Master's degree in Computer Science, Engineering