Lead Data Engineer

JOB RESPONSIBILITIES:

Tracks the various Machine learning projects and their data needs.

Tracks and improves Kanban process of product maintenance

Drives complex technical discussions both within company and outside data partners

Actively Contributes to the design of machine learning solutions by having a deep understanding of how the data is used and how new sources of data can be introduced

Advocates for investments in tools and technologies to streamline data workflows and reduce technical debt

Continuously explores and adopts emerging technologies and methodologies in data engineering and machine learning

Develops and maintains scalable data pipelines to support machine learning models and analytics

Collaborates with data scientists to ensure efficient data processing and model deployment

Ensures data quality, integrity, and security across all stages of the data pipeline

Implements monitoring and alerting systems to detect anomalies in data processing and model performance

Enhances data versioning, data lineage, and reproducibility practices to improve model

transparency and auditing

QUALIFICATION

5+ years of experience in data engineering or related fields, with a strong focus on building scalable data pipelines to support machine learning workflows.

Bachelor’s or Master’s degree in Computer Science, Engineering, Mathematics, or other relevant fields.

Specific experience in Kafka needed . Snowflake and data bricks would be huge plus.

Proven expertise in designing, implementing, and maintaining large-scale, high-performance data architectures and ETL processes managing 1TB a day.

Strong knowledge of database management systems (SQL and NoSQL), distributed data processing (e.g., Hadoop, Spark), and cloud platforms (AWS, GCP, Azure).

Experience working closely with data scientists and machine learning engineers to optimize data flows for model training and real-time inference with latency requirements.

Hands-on experience with data wrangling, data preprocessing, and feature engineering to ensure clean, high-quality data for machine learning models.

Solid understanding of data governance, security protocols, and compliance requirements (e.g., GDPR, HIPAA) to ensure data privacy and integrity.

Preferred

Experience in data pipelines and analytics for video-game development

Experience in Advertising industry

Experience in online businesses where transactions happen without human intervention