Data Engineering

• Engineer - Data Engineering in Financial Services with 6-10 years of experience
• Lead design and implementation of scalable data architectures
• Collaborate with stakeholders to define optimal data models
• Develop data pipelines and optimize for performance
• Oversee database management, ETL processes, and data quality standards
• Provide technical leadership, mentor junior team members, and participate in code reviews
• Collaborate with cross-functional teams to deliver data solutions
• Communicate technical concepts to non-technical stakeholders
• Implement monitoring systems for data pipeline performance
• Good to have experience with Databricks, CI/CD practices, AWS Certified Big Data - Specialty, and Databricks Certified Professional Data Engineer certifications

Roles & Responsibilities

• Lead the design and implementation of scalable, efficient, and robust data architectures to meet business needs and analytical requirements.
• Collaborate with stakeholders to understand data requirements, build subject matter expertise, and define optimal data models and structures.
• Design and develop data pipelines, ETL processes, and data integration solutions for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Optimize data pipelines for performance, reliability, and scalability.
• Oversee the management and maintenance of databases, data warehouses, and data lakes to ensure high performance, data integrity, and security.
• Implement and manage ETL processes for efficient data loading and retrieval.
• Establish and enforce data quality standards, validation rules, and data governance practices to ensure data accuracy, consistency, and compliance with regulations.
• Drive initiatives to improve data quality and documentation of data assets.
• Provide technical leadership and mentorship to junior team members, assisting in their skill development and growth.
• Lead and participate in code reviews, ensuring best practices and high-quality code.
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand their data needs and deliver solutions that meet those needs.
• Communicate effectively with non-technical stakeholders to translate technical concepts into actionable insights and business value.
• Implement monitoring systems and practices to track data pipeline performance, identify bottlenecks, and optimize for improved efficiency and scalability.
• Actively contribute to the end-to-end delivery of complex software applications, ensuring adherence to best practices and high overall quality standards.
• Provide technical leadership and expertise in making sound architectural decisions and solving challenging technical problems.
• Conduct code reviews, provide constructive feedback, mentor and coach junior engineers, fostering a culture of continuous learning, growth, and technical excellence within the team.
• Orchestrate work that spans multiple engineers within the team and keep all relevant stakeholders informed.
• Support the lead/EM in sharing work with stakeholders and escalating issues when necessary.

Our ideal candidate

• Extensive experience in AWS, PySpark, Spark, and Hadoop, showcasing advanced proficiency in big data technologies.
• Thorough understanding and application of ETL frameworks, Java, Python, Scala, SQL, NoSQL, PostgreSQL, and Agile methodologies in developing data pipelines and managing databases.
• Proven expertise in data integration, ensuring data quality and building data warehouses.
• Strong capabilities in data security, stakeholder management, data governance, migration, data science, data architecture, and code quality.
• Collaborative approach in requirements gathering, integrating GCP, Talend, S3, Jenkins, and working with big data technologies.
• Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
• AWS Certified Big Data - Specialty and Databricks Certified Professional Data Engineer certifications preferred