Data Engineer with Snowflake and Apache Spark

Key Responsibilities:

• Design, develop, and maintain robust, scalable ETL/ELT pipelines using Apache Spark and Snowflake.
• Leverage Databricks for data processing, transformation, and analytics in distributed environments.
• Develop efficient SQL and Spark applications to process and analyze large volumes of data.
• Implement and maintain data warehousing solutions using Snowflake with best practices for performance, cost, and security.
• Collaborate with data scientists, analysts, and business stakeholders to meet data needs.
• Ensure data quality and integrity through unit testing, data validation, and monitoring.
• Optimize and troubleshoot Spark jobs, SQL queries, and Snowflake data workflows.
• Integrate with various data sources (cloud storage, APIs, RDBMS) and tools (Airflow, DBT, etc.).
• Apply data governance and compliance policies in data pipeline design and execution.

Required Qualifications:

• Certifications:
  • SnowPro Core / Advanced Certification (e.g., SnowPro Advanced: Architect, Data Engineer, etc.)
  • Databricks Certified Associate Developer for Apache Spark (latest version preferred)
• Experience:
  • 3+ years of experience working with Snowflake, including schema design, query optimization, and Snowpipe/Streams/Tasks.
  • 2+ years of hands-on development with Apache Spark (PySpark, Scala, or Java) in Databricks or open-source environments.
  • Strong understanding of distributed computing, data lakes, and modern data architectures.
• Technical Skills:
  • Proficient in SQL, Spark (RDD/DataFrame APIs), and Python or Scala
  • Experience with cloud platforms (AWS, Azure, or GCP), especially integrating Snowflake and Databricks
  • Familiarity with data modeling, data quality, and orchestration tools (e.g., Airflow, Prefect)
  • Knowledge of CI/CD pipelines and version control (e.g., Git, GitHub Actions)