Key Responsibilities
- Design and implement robust, scalable, and secure data architectures using Databricks, Spark, Delta Lake, and cloud-native tools.
- Collaborate with data engineers, analysts, and business stakeholders to define data models, pipelines, and governance strategies.
- Develop and maintain data lakehouses, ensuring optimal performance and cost-efficiency.
- Define best practices for data ingestion, transformation, and storage using Databricks notebooks, jobs, and workflows.
- Architect solutions for real-time and batch data processing.
- Ensure data quality, lineage, and compliance with internal and external standards.
- Lead migration efforts from legacy systems to modern cloud-based data platforms.
- Mentor junior team members and evangelize data architecture principles across the organization.
Required Skills & Qualifications
- 12+ years of experience in data architecture, with 5+ years hands-on in Databricks.
- Strong Experience in Snowflake
- Experience in cloud platforms AWS, especially AWS Databricks.
- Strong proficiency in Apache Spark, Delta Lake, and PySpark.
- Experience with data modelling, ETL/ELT pipelines, and data warehousing.
- Familiarity with CI/CD, DevOps, and Infrastructure as Code (Terraform, ARM templates).
- Knowledge of data governance, security, and compliance frameworks.
- Excellent communication and stakeholder management skills.
Preferred Qualifications
- Databricks Certified Data Engineer or Architect.
- Experience with MLflow, Unity Catalog, and Lakehouse architecture.
- Background in machine learning, AI, or advanced analytics.
- Experience with tools like Apache Airflow, dbt, or Power BI/Tableau