Data Engineer

• Work with Azure Synapse Analytics using PySpark for data processing and analytics.
• Develop and manage Synapse Pipelines, Data Flows, and Data Integration tasks.
• Utilize Azure Synapse Studio for data development and orchestration.
• Perform data transformation and ETL/ELT processes using Synapse Pipelines, Data Factory, and PySpark.
• Integrate Azure Synapse Analytics with other Azure data services such as Azure Data Lake Storage, Azure Data Factory, Azure Databricks, and Azure SQL Database.
• Conduct data ingestion from sources like Azure Blob Storage, Data Lake Storage, and on-premises databases.
• Use Synapse SQL and Spark SQL for data analysis and transformation.
• Automate tasks using PowerShell or other scripting languages.
• Troubleshoot and resolve issues in data pipelines and workflows.
• Debug data processing and transformation problems.

Qualifications

• 5 to 7 years of experience with Azure Synapse Analytics and PySpark.
• Hands-on experience using Azure Synapse Studio with PySpark.
• Experience working with Spark integrated within Synapse Analytics.
• Proficiency in Python for data processing using PySpark.
• Strong experience in integrating with Azure data services such as Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Azure SQL Database, etc.
• Strong SQL skills for querying and managing relational databases.
• Experience in performance tuning and optimization of SQL queries.
• Solid understanding of database design principles and normalization.
• Familiarity with big data processing using Apache Spark.
• Experience with scripting languages such as PowerShell for automation.
• Hands-on experience in migrating data from Oracle to PostgreSQL