Big Data/Data Engineer

Responsibilities:

• Develop and maintain big data applications using Spark and Scala for processing and analyzing large datasets related to banking operations.
• Design and implement scalable data pipelines using Spark to ingest, transform, and load data from various sources (e.g., relational databases, message queues, log files).
• Utilize Spark libraries (e.g., Spark SQL, MLlib) to perform data cleansing, transformation, feature engineering, and potentially machine learning tasks.
• Collaborate with data engineers to optimize Spark jobs for performance and efficiency.
• Work closely with data scientists to understand their analytical needs and translate them into production-ready Spark applications.
• Write clean, maintainable, and well-documented code adhering to best practices.
• Participate in code reviews and knowledge-sharing sessions with the team.
• Stay updated on the latest advancements in big data technologies (Spark, Scala) and the banking domain.

Qualifications:

• 6-8 years of experience in software development with a focus on big data technologies.
• Proven experience developing big data applications using Apache Spark and Scala.
• Strong understanding of distributed computing frameworks and concepts (MapReduce, functional programming).
• Experience with data ingestion techniques (e.g., Kafka, Flume) is a plus.
• Familiarity with relational databases (e.g., MySQL, PostgreSQL) and SQL querying.
• Experience working with data pipelines and data warehousing concepts.
• Excellent problem-solving and analytical skills.
• Ability to work independently and as part of a team.
• Strong communication and collaboration skills.
• Understanding of the banking domain (a plus).

Desirable Skills:

• Experience with cloud platforms (AWS, Azure, GCP) for deploying Spark applications.
• Knowledge of machine learning libraries (e.g., MLlib, TensorFlow) is a plus.
• Experience with containerization technologies (Docker, Kubernetes).
• Experience with DevOps tools and methodologies (CI/CD pipelines).
• Experience working in an Agile development environment