Spark ETL Developer

Responsibilities

• Review current environment that is used for product launch.
• Design and recommend the go forward architecture using Spark, Python and platform including migration strategies to AWS.
• Design, implement and execute a strategy to migrate to a public cloud or hybrid cloud environment that utilizes AWS.
• Migrate to and implement the new rules and jobs to support Operational Intelligence.
• Set up and configure alerting and monitoring.

Required Skills

• Able to prioritize and execute tasks in a high-pressure environment under tight schedules.
• Proficient in SQL, Scripting, etc.
• Should have skills in Data Analysis/Triage.
• Senior knowledge of Data Quality.
• Strong expertise in Database Management.
• Extremely fluent with Tableau & reporting/visualization.
• Good knowledge of Jira.
• Superb communication skills with an emphasis on writing and interpreting abilities as this is a leadership position in a new organization.
• Ability to think analytically and should be a problem solver.
• AWS Big Data or Solutions Architect certification a plus.

Required Experience

• Minimum of 10+ years operations experience.
• 7+ years hands on experience with Apache Spark.
• Minimum of 3 years of experience in utilizing Hadoop (Hive, General HDFS knowledge) specifically Cloudera.
• 1-3 years hands on experience with Python/scala.
• Experience in Teradata and Informatica is required.
• 1-3 years of experience as an AWS architect with specific implementations of architecting of systems to solve real problems.
• 1-3 years of experience in a hybrid cloud environment.
• Data Governance experience a plus.

Education Requirements

• Bachelor’s Degree in Computer Science, Computer Engineering or a closely related field.