Senior Data Infrastructure/Data Platform Engineer

The Senior Data Infrastructure/Data Platform Engineer will be a Senior software engineer in the Data Engineering, Analytics & BIDW team. This role has a combination of different skills enabling them to design, build and maintain a scalable data infrastructure. This is a senior individual contributor role with a focus on hands on development, collaborative problem solving, and mentorship of junior engineers.

• Lead the design, development, and maintenance of scalable data pipelines, ETL/ELT processes, and data models.  
• Focus on Kafka, HDFS, HBase, Parquet, Spark, and Flink to ensure efficient data ingestion, storage, and processing.  
• Build and manage data ingestion frameworks using big data technologies, ensuring high efficiency and reliability.  
• Develop and optimize data storage solutions, including data warehousing, data lakes, and NoSQL databases on cloud platforms such as on-prem data center, AWS, Azure, or Google Cloud.  
• Design, develop, and maintain data pipelines and workflows using Airflow, Kubeflow to ensure efficient and reliable data ingestion, processing and storage. 
• Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions that meet those needs. 
• Ensure data quality, governance, and security through the implementation of best practices and robust monitoring systems.  
• Work with various data formats, including Parquet, Avro, and others, to ensure efficient data processing and storage

Must-have Requirements –

• Bachelor’s degree in computer science, Engineering, or a related field; Master’s degree preferred. 
• 7+ years of proven experience as a Data Engineer or in a similar role, with a strong focus on data warehousing, ETL/ELT processes, and data modeling.  
• Proficiency in programming languages such as Python, Shell scripting, Java, or Scala.
• Hands-on experience with big data technologies such as Kafka, Spark, HDFS, HBase, Flink, DBT, SQL Mesh, etc.
• Hands-on experience with Containerization & Orchestration technologies such as Docker and Kubernetes for managing scalable data applications.
• Hands-on experience with Hybrid cloud data governance and data quality frameworks.
• Experience with building and managing data pipelines in a multi-cloud environment such as AWS, Azure, or Google Cloud.  
• Strong SQL skills and experience working with both relational and NoSQL databases. 

Preferred Requirements -

• Familiarity with data visualization tools such as Tableau, Power BI, or Looker.
• Excellent problem-solving, communication, collaborative and analytical skills, and experience in mentoring or leading a team of data engineers