Data Engineer

As an Data Engineer, you will be a part of an Agile team to build healthcare applications and implement new features while adhering to the best coding development standards.

Responsibilities: -

• Design and develop end-to-end business intelligence solutions using AWS and Databricks.
• Build and maintain data models, ETL pipelines, and data warehousing solutions to support reporting and analytics.
• Develop interactive dashboards, reports, and data visualizations that provide meaningful insights to stakeholders.
• Develop and optimize ETL processes using Databricks and Apache Spark to ingest, clean, and transform large datasets.
• Integrate data from various sources, including relational databases, APIs, and cloud storage, into a centralized data warehouse.
• Ensure data accuracy, consistency, and availability across all BI platforms.
• Work closely with business analysts, data engineer, and other stakeholders to gather requirements and translate them into technical solutions.
• Collaborate with cross-functional teams to ensure BI solutions align with business objectives and data governance policies.
• Provide technical guidance and support to stakeholders on best practices for data visualization and reporting.
• Implement and configure AWS services, such as S3 and AWS DMS, to support CDC and analytics workloads.
• Utilize Databricks to build scalable data processing workflows and enhance the performance of BI solutions.
• Leverage Databricks' Delta Lake for real-time data processing and analytics.
• Monitor and optimize the performance of ETL processes, data models, and BI reports to ensure efficiency and scalability
• Troubleshoot and resolve performance issues, ensuring minimal downtime and optimal user experience.
• Implement caching, indexing, and other performance-enhancing techniques in BI tools.
• Develop and maintain visually compelling dashboards and reports using Power BI.
• Implement best practices for data visualization, ensuring that reports are intuitive, interactive, and actionable.
• Automate the delivery of reports and alerts to stakeholders, ensuring timely access to critical insights.
• Ensure that BI solutions comply with data governance policies, including data quality, security, and privacy requirements.
• Implement role-based access controls (RBAC) and data encryption to protect sensitive information.
• Maintain accurate documentation of data sources, ETL processes, and reporting logic.
• Strong proficiency in SQL, Python, and Apache Spark. Experience with data modeling, ETL development, and data warehousing. Hands-on experience with Power BI.
• Knowledge of AWS services such as S3, Lambda, AWS DMS, API Gateway.
• Strong analytical and problem-solving skills, with a keen eye for detail.
• Experience with Databricks’ Delta Lake and real-time data processing.
• Familiarity with data governance and data quality frameworks. Understanding of machine learning concepts and their application in BI.
• Experience with AzureDevops for CI/CD.
• Experience with Terraform.
• Experience with on-prem to AWS data connectivity