Azure Data Engineer

JD:

We are seeking a highly experienced and motivated Senior Data Engineer to join our growing team.
In this role, you will be responsible for designing, building, and maintaining robust and scalable data pipelines and solutions on the Azure cloud platform. You will leverage your deep expertise in big data technologies and Azure data services to empower our organization with actionable insights. The ideal candidate will possess strong leadership skills, a passion for data, and a proven track record of delivering high-quality data solutions.

Responsibilities:

Design, develop, and maintain scalable and efficient ETL/ELT pipelines using PySpark, SQL, and Azure Data Factory.

Build and optimize data models, partitioning strategies, and performance tuning for large datasets using Azure SQL Database and Azure Synapse Analytics.

Implement and manage data solutions using Azure Data Lake Storage (ADLS), Azure Databricks, Azure Functions, and Synapse Pipelines.

Develop and maintain infrastructure as code using Terraform, ARM templates, or Bicep.

Utilize Apache Spark (PySpark) and distributed computing principles on Azure Databricks to process and analyze large datasets.

Implement CI/CD pipelines for data engineering workflows using Azure DevOps.

Ensure data governance, security, and compliance through the implementation of Azure RBAC, Key Vault, and Purview.

Develop and maintain data solutions with integration into tools such as Power BI, Azure Logic Apps, or Data Factory triggers.

Implement real-time/streaming data solutions using Azure Event Hubs, Azure Stream Analytics, or Kafka on Azure.

Write and maintain high-quality, modular code in Python, SQL, and Shell Scripting.

Collaborate with stakeholders to understand data requirements and deliver effective, scalable solutions.

Provide technical leadership and mentorship to junior engineers and contribute to overall architectural decisions.

Required Skills:

6+ years of hands-on experience in data engineering roles.

Strong experience with Azure Databricks, Azure SQL, Azure Data Factory, and ADLS.

Proficient in Python for data manipulation and workflow scripting.

Deep understanding of SQL performance tuning, data modeling, and partitioning strategies.

Experience with infrastructure as code and automation tools (Terraform, Bicep, ARM templates).

Knowledge of Azure security practices, including access management and encryption.

Familiarity with streaming technologies and real-time data ingestion pipelines.

Experience in working with large, complex datasets and distributed data processing frameworks.

Strong communication skills and ability to work collaboratively in cross-functional team