Senior Data Engineer

Tech Lead

• Privileged technical point of contact for the Product Owner and business sponsors
• Ability to take data engineering responsibility of a data project
• Listening skills and very good communication skills
• Ability to manage technical debt (variable according to use cases)
• Technical architect, lead design meetings and tech breakdowns
• Ability to mentor, guide the work of a more junior Data Engineer

Data Ingestion & Processing:

• Design and implement batch and real-time data ingestion pipelines.
• Use Databricks (with PySpark) for big data processing tasks.
• Use DBT to transform data.
• Cleanse, transform, and enrich raw data to make it analytics ready.
• Optimize queries and data processing for speed and cost efficiency.

Data Storage & Management:

• Design and implement database schemas, tables, and views.
• Optimize storage formats for querying, such as Parquet or Delta Lake.
• Enforce data quality checks and data lineage documentation.
• Implement partitioning, bucketing, and indexing strategies for efficient data retrieval.

Collaboration with data experts (data analyst, data scientists):

• Work closely with data scientists to provide data in appropriate formats for machine learning and advanced analytics.
• Collaborate with Platform Engineer teams to comply with Platform good practices and escalate common needs
• Assist data analysts with SQL queries, views, and report generation.
• Collaborate on the deployment of machine learning models to production.

Security & Compliance (with Azure security experts):

• Implement role-based access controls and data encryption (at-rest and in-transit).
• Comply with industry and organizational data standards, privacy regulations, and best practices.
• Regularly audit data access and usage.

Infrastructure & Configuration (with Azure infrastructure experts):

• Set up and maintain Azure cloud infrastructure.
• Configure and optimize Azure Data Lake Storage, Blob Storage, and other Azure storage solutions.
• Deploy and manage Databricks clusters for processing tasks.
• Implement and maintain data pipelines using Azure Data Factory.
• Monitor and troubleshoot infrastructure-related issues.

Documentation & Training:

• Onboard new team members, providing access and initial training on tools.

• Create documentation and knowledge bases for data pipelines, best practices, and tooling.

Continuous Improvement:

• Stay updated with the latest advancements in data engineering technologies.
• Propose and implement optimizations for current workflows and systems.
• Proactively identify areas of improvement and automation.
• Regularly update team on new features or changes in Azure, Databricks, or related technologies.

EDUCATION

• Engineering Master's degree or PhD
• 5 years+ experience in a Data engineering role into large corporate organizations
• Experience in a Data/AI environment into a Cloud ecosystem

SOFT SKILLS

Leading by example
 

• Initiative: Taking the lead on projects or tasks without being asked, demonstrating proactive behavior and inspiring others to do the same.
• Consistency: Regularly delivering high-quality work, showing others the standards and expectations through one's own behavior.
• Empathy: Understanding and valuing team members' feelings and perspectives, fostering a supportive work environment.
• Accountability: Owning up to mistakes, learning from them, and making amends, demonstrating to the team the importance of taking responsibility.

Communication and Collaboration:
 

• Clarity: Ability to explain complex technical concepts in simple terms to non-technical stakeholders such as data analysts, business users, and management.
• Active Listening: Paying keen attention to others' ideas and feedback, ensuring that all voices are heard and considered.
• Documentation: Creating clear, concise, and comprehensive documentation that enables other team members to understand and use the systems and tools effectively.
• Teamwork: Building positive working relationships and working seamlessly with cross-functional teams, including data scientists, data analysts, IT, and businesses.
• Feedback Reception: Welcoming constructive criticism and using it for continuous improvement.

Problem Solving and Troubleshooting:
 

• Analytical Thinking: Breaking down complex problems into smaller, manageable components to find root causes.
• Creativity: Thinking outside the box and approaching challenges from different angles to find innovative solutions.
• Resilience: Persisting in the face of setbacks and not getting discouraged when confronted with challenging issues.