Big Data Engineer

Key Responsibilities:

Design and implement scalable big data solutions tailored to NDUS’s needs.

Maintain and enhance existing big data infrastructures to meet NDUS’s unique requirements.

Test and evaluate new big data technologies and frameworks for compatibility with NDUS systems and goals.

Collect, store, process, manage, analyze, and visualize large datasets to derive actionable insights.

Collaborate with team members to integrate big data solutions with existing NDUS systems.

Ensure data integrity and security across all platforms used within NDUS.

Develop and optimize data pipelines for ETL/ELT processes specific to NDUS’s data needs.

Document technical solutions and maintain comprehensive records in line with NDUS standards and protocols.

Stay updated with the latest trends and advancements in big data technology relevant to NDUS’s strategic initiatives.

Required Qualifications:

Thorough understanding of cloud computing technologies, including IaaS, PaaS, and SaaS implementations.

Skilled in exploratory data analysis (EDA) to support ETL/ELT processes.

Proficiency with Microsoft cloud products, including Azure and Fabric.

Experience with tools such as Data Factory and Databricks.

Ability to script in multiple languages, with a strong emphasis on Python and SQL.

Preferred Qualifications:

Experience with data visualization tools.

Proficiency with Excel and Power BI.

Familiarity with Delta Lake.

Knowledge of Lakehouse Medallion Architecture.

IaaS, PaaS, or SaaS data or AI implementations within Microsoft Azure

Required

3

Years

Exploratory Data Analysis (EDA): Proficiency in EDA techniques to support ETL/ELT processes

Required

3

Years

Implementation of Development and Production Workflows in Azure Data Factory and Databricks

Required

2

Years

Strong scripting skills in Python and SQL

Required

3

Years

Expert proficiency of data engineering creation in Microsoft Fabric

Required

1

Years

Expert proficiency with Excel and Power BI

Highly desired

2

Years

Expert proficiency of Delta Lake format and protocol

Highly desired

1

Years

Expert understanding of the Data Lakehouse Medallion Architecture