Scala Developer with Databricks

Job Description:-

Key Responsibilities:

Design, develop, and maintain big data applications using Scala and Apache Spark on the Databricks platform.

Build ETL/ELT pipelines to ingest, transform, and deliver data from various structured and unstructured sources.

Collaborate with data scientists, analysts, and other engineers to understand business requirements and translate them into scalable solutions.

Optimize Spark jobs for performance and cost-effectiveness in Databricks.

Implement best practices in coding, testing, and data engineering workflows.

Monitor data pipelines and troubleshoot production issues.

Integrate solutions with cloud platforms such as Azure, AWS, or GCP, depending on project architecture.

Write unit and integration tests, and ensure data quality using tools like Great Expectations, Delta Lake, or custom validations.

Required Qualifications:

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

3+ years of professional experience in Scala development.

Strong experience with Apache Spark and distributed data processing.

Hands-on experience with Databricks (not just notebooks—also workflows/jobs, cluster management, Delta Lake, etc.).

Experience with cloud platforms (Azure preferred; AWS/GCP is a plus).

Proficiency with data formats (Parquet, Avro, JSON, CSV) and file systems (HDFS, S3, ADLS).

Solid understanding of functional programming principles and design patterns.

Familiarity with Git, CI/CD pipelines, and job orchestration tools (e.g., Airflow, Azure Data Factory, or similar).