Bigdata Engineer

• Design and implement scalable data architectures, including data lakes, data warehouses, and data pipelines, to support diverse data requirements.
• Develop and maintain ETL processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Design and implement data models and schemas for efficient storage, retrieval, and analysis of data, considering business requirements and data governance principles.
• Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow across systems.
• Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
  Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.
• Utilize skills in Apache Spark, Database, AB Testing, Apache Hadoop, API Development, AWS Technologies, Big Data, AWS CloudFormation to enhance job performance and outcomes.

Qualifications:

• Possess a minimum of 10 years of experience as a Big Data Engineer or in a similar role.
• Proficient in Apache Spark and Apache Hadoop.
• Strong knowledge of Database Management and AB Testing.
• Demonstrated experience with API Development.
• Proficient in AWS Technologies including AWS CloudFormation.
• Solid understanding and experience with Big Data tools and frameworks