Description

Responsibilities

  • Design, develop test, deploy, maintain and improve data integration pipeline.
  • Design, develop, test, deploy, support, enhance data integration solutions seamlessly to connect and integrate enterprise systems in our Enterprise Data Platform.
  • Innovate for data integration in Apache Spark-based Platform to ensure the technology solutions leverage cutting edge integration capabilities.
  • Facilitate requirements gathering and process mapping workshops, review business/functional requirement documents, author technical design documents, testing plans and scripts.
  • Assist with implementing standard operating procedures, facilitate review sessions with functional owners and end-user representatives, and leverage technical knowledge and expertise to drive improvements.

Required Skills

  • Batch & stream data processing using AWS platform, Snowflake, Apache Spark, EMR, Hadoop, Kafka, KStream, Data as a service (API).
  • Extremely fluent with Programming : Python, SQL, Shell.
  • Senior knowledge of ETL tools : Informatica.
  • Orchestration : Airflow, Nifi, Autosys Must have been part of a Data Migration project from on-premise to cloud using the above technologies.
  • Excellent articulation and communication skills.

Required Experience

  • 2+ years of Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems.
  • Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
  • Experience in Python and common python libraries.
  • Performance & Cost optimization experience for data pipeline in the cloud.
  • Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes etc.
  • Strong experience with source control systems such as Git, Bitbucket, and Jenkins build and continuous integration tools.
  • Databricks or Apache Spark Experience is a plus.

Education Requirements

  • Bachelor’s Degree in Computer Science, Computer Engineering or a closely related field.


 

Key Skills
Education

Any Graduate

Back To Jobs