Big Data Engineer

JOB DESCRIPTION:
Technical Skills:
• Data Structures and Algorithms: Expertise in designing efficient data structures and algorithms for big data processing.
• Distributed Systems: Experience with distributed computing frameworks and parallel processing.
• Data Integration: Proficiency in integrating data from various sources and ensuring data consistency.
• Performance Optimization: Skills in optimizing data processing and storage for high performance.
• Containerization: Experience with Docker and Kubernetes for deploying big data applications.
• Version Control: Proficiency in using Git for collaborative development.
• Security: Knowledge of data security practices and compliance with regulatory standards.
• Machine Learning: Familiarity with machine learning techniques and their application in big data environments

Responsibilities:
• Design, develop, and maintain scalable big data architectures and systems.
• Implement data processing pipelines using technologies such as Hadoop, Spark, and Kafka.
• Optimize data storage and retrieval processes to ensure high performance and reliability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Perform data modeling, mining, and production processes to support business needs.
• Ensure data quality, governance, and security across all data systems.
• Stay updated with the latest trends and advancements in big data technologies.
• Experience with real-time data processing and stream analytics.
• Knowledge of advanced analytics and data visualization tools.
• Knowledge of DevOps practices and tools for continuous integration and deployment
• Experience in managing big data projects and leading technical teams.