Data Engineering Lead

What You'll Do:

Work with business to understand business requirements and translate into low level design
Design and implement robust, fault tolerant, scalable, and secure data pipelines using pyspark, notebooks in MS Fabric
Review code of peers and mentor junior team members
Participate in sprint planning and other agile ceremonies
Drive automation and efficiency in Data ingestion, data movement and data access workflow
Contribute ideas to help ensure that required standards and processes are in place and actively look for opportunities to enhance standards and improve process efficiency.

Expertise You'll Bring:

Around 8 to 12 years of experience, at least 1 year in MS fabric and Azure cloud
Leadership: Ability to lead and mentor junior data engineers, help with planning and estimations
Data migration: Experience on migrating and re-modeling large enterprise data from legacy warehouse to Lakehouse (Delta lake) on MS Fabric or Databricks.
Strong Data Engineering Skills: Proficiency in data extraction, transformation, and loading (ETL) processes, data modeling, and database management. Also experience around setting up pipelines using Notebooks and ADF, setting up monitoring and alert notifications.
Experience with Data Lake Technologies: MS Fabric, Azure, Databricks, Python, Orchestration tool like Apache Airflow or Azure Data Factory, Azure Synapse along with stored procedures, Azure data lake storage.
Data Integration Knowledge: Familiarity with data integration techniques, including batch processing, streaming, and real-time data ingestion, auto-loader, change data capture, creation of fact and dimension tables.
Programming Skills: Proficiency in SQL, Python, Pyspark for data manipulation and transformation.
DP-700 certification will be preferred