Kafka Consultant

Job Description:

Identifying and rectifying the Kafka Messaging issues within justifying time

Working with the business and IT team to understand business problems, design, implement, and deliver an appropriate solution using Agile methodology across the larger program

Working independently to implement solutions on multiple platforms (DEV, QA, UAT, PROD)

Providing technical direction, guidance, and reviews to other engineers working on the same project

Administering Distributed kafka cluster in Dev, QA, UAT, PROD environments and troubleshooting performance issue

Implementing and debugging subsystems/microservice and components

Following automate-first/automate-everything philosophy

Programming languages

Requirements:

Thorough knowledge of Kafka concepts like producers, consumers, topics, partitions, brokers, and replication mechanisms

Programming language proficiency, primarily Java or Scala, with potential for Python depending on the project

Ability to design robust and scalable Kafka-based data pipelines, considering factors like data throughput, fault tolerance, and latency

Understand data serialization formats like JSON, Avro, and Protobuf, and how to manage data schema evolution

Knowledge of Kafka Streams for real-time data processing within the Kafka ecosystem

Monitoring and troubleshooting

Familiarity with tools to monitor Kafka cluster health, identify performance bottlenecks, and troubleshoot issues

Cloud integration experience deploying and managing Kafka on cloud platforms like AWS, Azure, or GCP

Distributed systems concepts