Data Engineer

Job Description
Mandatory Skills: Microsoft Azure, Py spark, Data Bricks, Hadoop, Spark, Airflow, Kafka

Requirments:

Experience working with distributed technology tools for developing Batch and Streaming pipelines using.

SQL, Spark, Python

Airflow

Scala

Kafka

Experience in Cloud Computing, e.g., AWS, GCP, Azure, etc.

Able to quickly pick up new programming languages, technologies, and frameworks.

Strong skills building positive relationships across Product and Engineering.

Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, build Docker images, etc.

Working knowledge of Data warehousing, Data modelling, Governance and Data Architecture

Experience working with Data platforms, including EMR, Airflow, Data bricks (Data Engineering & Delta Lake components)

Experience working in Agile and Scrum development process.

Experience in EMR/ EC2, Data bricks etc.