Big Data- Spark Scala

Job Description

Mandatory Skills

• Should have 10 years of experience in testing and test automation with Robot framework and python
• Knowledge of data manipulation using various data formats XML CSV JSON plain text etc and experience with SQL queries preparation
• Experience of Designing Editing effective reusable automated tests
• Collaborating with QA Analysts and Software Developers to develop test automation solutions
• Designing and developing test automation scripts
• Contribute in defining the overall automated test strategy approach for team and following test automation guidelines strategy
• Analyse and comment requirements and technical designs from a test perspective
• Designing implementing and executing test cases by automated test methods
• Gathering and managing test data Monitoring and documenting test results Report debug and verify defects
• Collaboration with the team to prepare setup monitor CICD delivery pipelines
• Good Knowledge on Tools like JIRAALMBIT BucketGIT
• Excellent communication and teamwork skills Great attention to detail
• Coordinates with various functions within the company and client organization and ensure timely delivery Own the deliverable task

What you'll be doing

• Building distributed and highly parallelized big data processing pipeline which process massive amount of data both structured and unstructured data in near real-time
• Leveraging Spark to enrich and transform corporate data to enable searching data visualization and advanced analytics
• Working closely with analysts and business stakeholders to develop analytics models
• Continuous delivering on Hadoop and other Big Data Platforms
• Automating processes where possible and are repeatable and reliable
• Working closely with QA team

As an expert Data Engineer, you are expected to have

• Excellent Programming language skills Spark Scala BashUnix Scripting SQL Bash
• Excellent understanding of Hadoop and HDFS Architecture
• Excellent understanding of file types their pros and cons
• Excellent understanding of Build Tools especially SBTGradle
• Very good understanding and good experience in implementing CICD eg Building and Maintaining Jenkins Pipelines
• Working experience in SAFeAgile
• As an expert you are also responsible for Juniors writing maintainable code
• Are collaborative and achieve your expectations through communication and teamwork
• Are curious responsive and can understand the needs of others to ensure delivery of the desired results
• Work qualitatively and strive to always do things better

Skills

• Spark Scala Scala with a focus on the programming
• Apache Spark 2x
• Spark query tuning and performance optimization
• Understanding on Hadoop Architecture
• Experience working with Hive SQL and HDFS
• Deep understanding of distributed systems eg CAP theorem partitioning replication consistency and consensus
• Good to have experience in Scala
• Experience working with tools like Jenkins Jira Bitbucket and Git
• Experience in writing shell scripts and working with Linux platforms
• As an advantage familiarity with one or more of the below technologies
• Experience working in AgileSAFe
• Experience certification with AWS Cloud
• ETL Testing