Big Data Development Engineer

Job Summary:
We are looking for a skilled and motivated Big Data Development Engineer to join our team. As a Big Data Development Engineer, you will be responsible for designing, developing and maintaining our Big Data platform. You will work with cross-functional teams to ensure efficient and accurate data processing, storage and retrieval. The ideal candidate should have a solid background in Big Data technologies such as Hadoop, Spark and NoSQL databases.

Job Description:
1. Design, develop and implement big data solutions to process high volume data from various data sources.
2. Develop and maintain data pipelines to transform and load data into Hadoop or other Big Data platforms.
3. Optimise performance, scalability and reliability of Big Data solutions.
4. Monitor and troubleshoot Big Data jobs to ensure data accuracy and integrity.
5. Keep abreast of industry trends and best practices in the field of Big Data development and data processing.
6. Able to read, write and speak in English.

Job Requirements:
1. Bachelor's degree in Computer Science, Information Systems, or related field.
2. 3+ years of hands-on experience as a Big Data Development Engineer or similar position.
3. Familiarity with Big Data technologies such as Hadoop, Spark and NoSQL databases.
4. Proficiency in programming languages such as Java, Spring Framework, Python or Scala.
5. Proficiency in task scheduling tools such as airflow, xxljob.
6. Familiar with common monitoring work, such as zabbix, grafana, ELK, understand Prometheus best practices.
7. Have the ability to write scripts independently, familiar with shell is preferred.
8. Experience with GRPC protocols, asynchronous patterns and messaging frameworks such as Kafka.
9. Experience with data modelling and ETL tools.
10. Ability to work independently and collaborate in a team environment.
11. Good communication and interpersonal skills.