Data integration engineer

Roles & Responsibility

Data Integration Engineer

Role Overview

This role will be responsible for conducting ingestion and systems integration activities on the Databricks platform This role requires a deep understanding of

Data Lake ingestion and integration processes and best practices

ETLELT design

CICD

System integration tools

Data pipeline management

About the role

Integrate data from a variety of source systems and knowledge to tailor integration approaches on a persystem basis

Liaise with Technical Data Steward and understanding business requirements for ingestion and integration

Document ingestion and integration flows and pipelines

Use Airflow to schedule and automate ingestion jobs

Manage metadata and master data in technical data catalog

Manage maintain and oversee ETLELT pipelines on the Databricks platform

Ensure ELTETL design meets required security and compliance guidelines and ensure PII management flagging and risk assessment during ingestion and integration

Conduct pipeline monitoring

Ensure adherence to SDLC best practices

About you

Bring 2 years of experience in data engineering ingestion pipelining and ETLELT

Hold a bachelors degree in computer science engineering statistics or related field

Have handson experience with and understanding of the following

SparkScala

SQL

PythonPySpark or similar programming language

Databricks

ETLELT development monitoring and pipelining using tools such as Apache Airflow

mibo

Data quality guidelines

CICD pipelines

Agile

Git and version control