Databrick Platform Admin

Key Skills: Databricks Support, SQL, Integration, Python/ Scala/ Java Programming, Cloud AWS/ Azure/ GCP

Job Description:
We are seeking a highly skilled and experienced Databricks Engineer to join our growing team and play a crucial role in enhancing our data processing and analytics capabilities.

As a Databricks Engineer you will be responsible for dealing with the most complex situations related to development, design, implementation, and maintenance of our Databricks environment to ensure optimal performance, scalability, and reliability while aligning with our audit and security standards. You will work closely with cross-functional teams, including data scientists, analysts, and software engineers to build robust data pipelines and deliver actionable insights.

Key Responsibilities:

1. Databricks Environment Management:

• Set up and configure Databricks clusters to meet specific project requirements
• Optimize Databricks configurations for performance and cost-effectiveness
• Ensure the security and integrity of the Databricks environment

1. Data Pipeline Development:

• Collaborate with data scientists and analysts to understand data requirements
• Design and implement scalable and efficient data processing pipelines using Databricks
• Troubleshoot and optimize existing data pipelines for improved performance

1. Integration and Collaboration:

• Work closely with cross-functional teams to integrate Databricks solutions into the overall data architecture
• Collaborate with software engineers to incorporate Databricks functionality into custom applications

1. Monitoring and Maintenance:

• Implement robust monitoring and logging solutions for Databricks clusters
• Conduct regular maintenance tasks, including updates and patches, to ensure system stability
• Generate/ review consumption reports, identify abnormal consumption and take corrective action.

1. Documentation and Training:

• Create and maintain comprehensive documentation for Databricks configurations and processes, document best practices and standards to be followed
• Provide training and support to internal teams on using Databricks effectively

Qualifications:

• Bachelor's degree in computer science, Engineering, or a related field; Master's Degree preferred
• Proven experience as a Databricks Engineer or similar role
• Strong proficiency in Apache Spark and Databricks
• Experience with cloud platforms such as AWS, Azure, or Google Cloud
• Proficient in programming languages such as Python, Scala, or Java
• Solid understanding of data architecture and ETL processes.

Preferred Qualifications:

• Databricks Certification
• Experience with big data technologies and distributed computing
• Familiarity with machine learning frameworks