Data Engineer

Skills:

– Databricks

– ETL pipelines

– Docker for containerization

– Utilizing REST API in Python

– Python

Must have:

– Expertise in implementing batch and real-time data processing solutions using Azure Data Lake storage, Azure Data Factory, and Databricks.

– Strong experience in Databricks.

– Experience in building ETL pipelines for ingesting, transforming, and loading data from multiple sources into Cloud data warehouses.

– Proficient in Docker for containerization, utilizing REST API in Python for seamless system integration, and applying containerization concepts to improve deployment efficiency and scalability.

– Experience in data extraction, data acquisition, transformation, data manipulation, performance tuning, and data analysis.

– Experience in Python libraries to build efficient data processing workflows and streamline ETL operations across large data sets and similar distributed systems.

– Expertise in automating data quality checks, reducing data errors by 40%, and ensuring more reliable reporting and analytics with data marts.

– Expertise in data orchestration and automation tools such as Apache Airflow, Python, and PySpark, supporting end-to-end ETL workflows.

– Experience in deployment activities.