Data Engineer- Databricks

Key Result Areas and Activities:

 Demonstrate leadership abilities to ensure the smooth delivery of team outcomes.

 Spearhead efforts to gather requirements from clients.

 Work closely with onsite counterparts and act as the representative for the offshore team.

 Collaborate with business users and other IT teams to ensure smooth operations and optimal performance of

ETL processes.

 Implement enhancements, monitor performance, troubleshoot issues, and provide continuous support for

Informatica PowerCenter workflows and Databricks jobs in the production environment.

 Conduct unit testing, integration testing, and troubleshooting of ETL processes to ensure data accuracy and

reliability.

 Manage the deployment of Informatica and Databricks ETL processes from development to production.

 Assist in version control and release management processes for ETL code.

 Identify and resolve job failures or performance issues in the production environment.

 Respond to incidents reported by users or monitoring tools, ensuring swift resolution to minimize downtime.

 Maintain detailed logs of incidents, resolutions, and troubleshooting steps for knowledge sharing and future

Work and Technical Experience

Must-have Skills:

 Hands-on experience on working with Informatica PowerCenter, Databricks with PySpark, Snowflake with a strong

emphasis on support and troubleshooting in production environments

 Expertise in performance tuning and job optimization using PowerCenter and Databricks

 Proficient in incident management tools and methodologies like ServiceNow or Jira

 Strong knowledge of SQL, data warehousing concepts, and relational databases like Oracle, SQL Server, etc.

 Strong understanding of data integration, ETL best practices, and batch processing

Good to have Skills:

 Hands-on experience or knowledge on Alteryx

 Good understanding of the CI/CD pipeline

 Working experience with GitHub

 Experience in working with Job scheduler and workload automation tools like Tidal job scheduler, Control-M, etc.

 Familiar with Unix/Linux scripting for automating tasks and managing job failures

 Experience in providing data engineering solutions while maintaining balance between architecture requirements,

required efforts and customer specific needs in other tools

reference.

 Provide regular reports on job status, system performance, and support activities.

 Demonstrate strong technical expertise and a solid understanding of Data Warehouse concepts and SQL.