Data Engineer

Key Responsibilities:

•                     Data Pipeline Design and Implementation:

Develop and implement scalable and reliable data pipelines using GCP services like BigQuery, Dataflow, Cloud Storage, and Composer.

•                     Medallion Architecture Implementation:

Design and implement the bronze, silver, and gold layers within the Medallion architecture to organize and process data effectively.

•                     Data Storage and Processing:

Implement data storage solutions and optimize data processing workflows using appropriate GCP tools and technologies.

•                     Data Quality and Security:

Ensure data quality, integrity, and security throughout the data lifecycle, including data validation, cleansing, and access control.

•                     Collaboration and Communication:

Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions.

•                     Monitoring and Maintenance:

Monitor and maintain the health of the data infrastructure, troubleshoot and resolve data-related issues, and ensure optimal performance.

•                     Staying Updated:

Keep up to date with the latest GCP features, data engineering best practices, and advancements in data architecture.

•                     Data Governance:

Implement data governance strategies to ensure compliance with industry standards and best practices.

Skills and Qualifications:

•                     GCP Expertise:

Strong knowledge of Google Cloud Platform (GCP) services, including BigQuery, Dataflow, Cloud Storage, and Composer.

•                     Medallion Architecture:

Solid understanding of the Medallion architecture principles and best practices.

•                     Data Engineering:

Proven experience in data engineering, including data warehousing, ETL processes, and data modelling.

•                     Programming Languages:

Proficiency in programming languages like Python, SQL, and potentially Apache Beam (for Dataflow).

•                     Problem-Solving:

Strong problem-solving and analytical skills to troubleshoot and resolve data-related issues.

•                     Communication:

Excellent communication and collaboration skills to work effectively with various teams.

•                     Cloud Architecture:

Knowledge of cloud architecture best practices, including scalability, reliability, and security.

•                     Data Modelling:

Experience with data modelling techniques and tools