Data Engineer with Snowflake and Apache Spark

Key Responsibilities

• Design, develop, and maintain robust, scalable ETL/ELT pipelines using Apache Spark and Snowflake.
• Leverage Databricks for data processing, transformation, and analytics in distributed environments.
• Develop efficient SQL and Spark applications to process and analyze large volumes of data.
• Implement and maintain data warehousing solutions using Snowflake with best practices for performance, cost, and security.
• Collaborate with data scientists, analysts, and business stakeholders to meet data needs.
• Ensure data quality and integrity through unit testing, data validation, and monitoring.
• Optimize and troubleshoot Spark jobs, SQL queries, and Snowflake data workflows.
• Integrate with various data sources (cloud storage, APIs, RDBMS) and tools (Airflow, DBT, etc.).
• Apply data governance and compliance policies in data pipeline design and execution.