Software Engineer 3 – Big Data

Job Description:

JOB DESCRIPTION:
Minimum 4 years of hand on experience with
• Building data pipeline using big-data stack (Hadoop, Hive, pySpark, python)
• Amazon AWS S3 – Object storage, security, data service integration with S3
• Data modelling and database design.
• Job Scheduler – Autosys
• PowerBI, Dremio
• Unix/shell scripting, CICD pipeline
• Exposure in GCP cloud data engineering is a plus

Manager Notes:
-The contractors need to be proactive, they can't wait to be told what to do
-Must be accountable along with the technical skills
-The tech stack mentioned, these are the technologies being used to build data pipelines
-They need to model, design the data, build pipelines, applying logic to the data to transform the data and troubleshoot
-They should have strong understanding and implementation of Autosys
-Ability to automate using spark, Python, Hadoop/Hive
-Should have a fundamental background in database design (MySQL or any standard database)
-Exposure to Cloud data engineering is a big plus, not required
-Financial services experience is a plus but not required-having domain knowledge is helpful