AWS Data Engineer

Responsibilities: -

• Design and develop end-to-end business intelligence solutions using AWS and Databricks. Build and maintain data models, ETL pipelines, and data warehousing solutions to support reporting and analytics. Develop interactive dashboards, reports, and data visualizations that provide meaningful insights to stakeholders.
• Develop and optimize ETL processes using Databricks and Apache Spark to ingest, clean, and transform large datasets. Integrate data from various sources, including relational databases, APIs, and cloud storage, into a centralized data warehouse. Ensure data accuracy, consistency, and availability across all BI platforms.
• Work closely with business analysts, data engineer, and other stakeholders to gather requirements and translate them into technical solutions.
• Collaborate with cross-functional teams to ensure BI solutions align with business objectives and data governance policies. Provide technical guidance and support to stakeholders on best practices for data visualization and reporting.
• Implement and configure AWS services, such as S3 and AWS DMS, to support CDC and analytics workloads. Utilize Databricks to build scalable data processing workflows and enhance the performance of BI solutions. Leverage Databricks' Delta Lake for real-time data processing and analytics.
• Monitor and optimize the performance of ETL processes, data models, and BI reports to ensure efficiency and scalability. Troubleshoot and resolve performance issues, ensuring minimal downtime and optimal user experience.
• Implement caching, indexing, and other performance-enhancing techniques in BI tools.
• Develop and maintain visually compelling dashboards and reports using Power BI. Implement best practices for data visualization, ensuring that reports are intuitive, interactive, and actionable.
• Automate the delivery of reports and alerts to stakeholders, ensuring timely access to critical insights.
• Ensure that BI solutions comply with data governance policies, including data quality, security, and privacy requirements. Implement role-based access controls (RBAC) and data encryption to protect sensitive information. Maintain accurate documentation of data sources, ETL processes, and reporting logic.
• Strong proficiency in SQL, Python, and Apache Spark. Experience with data modeling, ETL development, and data warehousing. Hands-on experience with Power BI.
• Knowledge of AWS services such as S3, Lambda, AWS DMS, API Gateway. Strong analytical and problem-solving skills, with a keen eye for detail.
• Experience with Databricks’ Delta Lake and real-time data processing. Familiarity with data governance and data quality frameworks. Understanding of machine learning concepts and their application in BI.
• Experience with AzureDevops for CI/CD. Experience with Terraform.
• Experience with on-prem to AWS data connectivity