Description

Job Description:

  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Azure Data Factory, and other Azure services
  • Data Processing: Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks
  • Integration: Integrate data from various sources, such as databases, APIs, and streaming data, into the Databricks environment, often using tools like Azure Data Factory or Databricks Workflows
  • Performance Optimization: Enhance the performance of data processing tasks by optimizing Spark jobs, managing cluster resources, and implementing best practices for data storage and retrieval
  • Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure the solutions meet business needs
  • Data Governance: Implement data governance policies and ensure data security and compliance with relevant regulations
  • CI/CD Integration: Utilize Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure
  • Technical Proficiency: Expertise in Azure Databricks, Azure Data Factory, Spark, Python, and SQL


 

Key Skills
Education

Any Graduate

Back To Jobs