Senior Data Engineer with Databricks, Python, Pyspark experience

• Project Description:
As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine learning and artificial intelligence, and intuitive workflows that allow our users to derive insights from big data that in turn drive their businesses. 
• Responsibilities:
• Hands-on developer who writes good quality, secure code that is modular, functional, and testable.
• Drive the evolution of Data & Services products/platforms with an impact-focused on data science and engineering.
• Design and implement scalable data architecture and data pipelines
• Solving complex problems with multi-layered data sets, as well as optimizing existing machine learning libraries and frameworks.
• Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
• Discover, ingest, and incorporate new sources of real-time, streaming, batch, and API-based data into our platform to enhance the insights we get from running tests and expand the ways and properties on which we can test 
problem, taking into account usability, technical feasibility, timelines, and differing stakeholder opinions to make a decision
Break large solutions into smaller, releasable milestones to collect data and feedback from product managers, clients, and other stakeholders
• Evangelize releases to users, incorporating feedback, and tracking usage to inform future development
Work with small, cross-functional teams to define the vision, establish team culture and processes
Consistently focus on key drivers of organization value and prioritize operational activities accordingly
• Escalate technical errors or bugs detected in project work
• Maintain awareness of relevant technical and product trends through self-learning/study, training classes, and job shadowing.
• Support the building of scaled machine learning production systems by designing pipelines and engineering infrastructure.
• Mandatory Skills Description:
• Working proficiency in using Python/Scala, Spark (tuning jobs), SQL, Hadoop platforms to build Big Data products & platforms.
• Good understanding of Databricks security, clusters, user management, deployment and performance tuning.
• Strong PySpark skills
• Good programming skills in Java and spring boot and Junit.
• Knowledge in software development test approaches & frameworks
• Familiarity with RESTful APIs and micro-services architectures
• Experience in working with CI/CD