Data Engineer

• Your responsibilities will encompass the full lifecycle of data engineering, from development and automation to integration and optimization.
• You will build and manage data pipelines that support business intelligence, reporting, and crucially, the training and deployment of LLMs. This includes ensuring data quality and integrity for AI applications, collaborating on feature engineering, and exploring the use of LLMs for data augmentation or synthesis.
• You will also work closely with IT and DevOps to manage the underlying data infrastructure, ensuring scalability and security, particularly for demanding AI/ML workloads.
• The ideal candidate possesses expertise in Python, distributed computing frameworks (Spark, Databricks, DBT), and both SQL and NoSQL databases, including vector databases relevant to LLMs.
• Collaboration with consulting teams and clients will be key as you integrate diverse data sources, develop real-time processing solutions, and optimize cloud infrastructure on platforms like AWS, Azure, or GCP.
• A critical aspect of this role involves working with large language models (LLMs), including data preparation, feature engineering, and integration into data pipelines to enable innovative AI-driven solutions.
• Your analytical mindset and ability to align data strategies with business needs, including the innovative application of LLMs, will be crucial to success

Qualifications:

• YOE- 6 to 9 years
• Data Engineer will be pivotal in architecting and maintaining intelligent data solutions driving business operations, client engagements, and advanced analytics.
• Leveraging over 5 years of experience, you'll design scalable data pipelines, automate ETL/ELT processes, and ensure robust data quality and compliance.
• Experience with ETL tools and cloud platforms (AWS, Azure, GCP) is essential. Furthermore, a strong understanding of machine learning fundamentals and direct experience working with large language models, including data preprocessing, fine-tuning, and deployment considerations, is a key requirement