Data Engineer with (Scala or Spark)

Job description

We are seeking a highly experienced and motivated Senior Data Engineer to lead the design development and optimization of data pipelines and architectures This role is critical in enabling scalable data solutions that support analytics reporting and machine learning initiatives across the organization

Key Responsibilities

Design build and maintain robust scalable and efficient data pipelines using tools such as Azure Data Factory Databricks and Apache Spark

Develop and optimize ETLELT workflows to ingest transform and store structured and unstructured data from various sources

Collaborate with data scientists analysts and business stakeholders to understand data requirements and deliver high quality datasets

Implement data governance quality and security best practices across all data engineering processes

Lead the integration of cloud native data services eg Azure Synapse ADLS Azure SQL Cosmos DB into enterprise data platforms

Monitor and troubleshoot data pipelines and infrastructure to ensure reliability and performance

Mentor junior engineers and contribute to architectural decisions and code reviews

Participate in Agile ceremonies and contribute to sprint planning and delivery

Required Skills

Strong programming skills in Python SQL and Spark

Handson experience with Azure data services ADF Synapse Databricks ADLS and Azure DevOps

Deep understanding of data modeling warehousing and distributed computing

Experience with CICD pipelines and version control systems Git GitHub Actions

Familiarity with data lake house architecture and modern data stack principles

Strong problem solving and communication skills

Preferred Qualifications

Bachelors or Masters degree in Computer Science Engineering or related field

Microsoft Certified Azure Data Engineer Associate or equivalent

Experience with MLOps Power BI and realtime data streaming Kafka Event Hubs

Exposure to data cataloging tools and metadata management

Skills

Mandatory Skills : Azure Storage, Azure Web Job, PowerBI, Azure Cosmos DB, Azure Data Factory, Azure Functions, Azure SQL, Azure Synapse Analytics, AZURE DATA LAKE, C#, Flexcube technical PL/SQL, Generative AI/Open AI/Vector DB, Databricks
Good to Have Skills : Azure Synapse Analytics, AZURE DATA LAKE