Big Data Engineer

Job Duties :

Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases. Experience building and optimizing ‘big data’ data pipelines, architectures and data sets. Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement. Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency and workload management. A successful history of manipulating, processing and extracting value from large disconnected datasets. Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores. Strong project management and organizational skills. Experience supporting and working with cross-functional teams in a dynamic environment. Implement the use case with Bigdata open source tools and technologies like SQOOP, HADOOP Ecosystems, HIVE, Unix Scripting, SPARK, SCALA, PYTHON, PYSPARK and SCALA to achieve the requirements. Ability to utilize the bigdata tools and techniques to integrate traditional databases like Oracle, SQL Server and Mysql into Hadoop Datalake. Create the Hive databases and store the processed data into Hive tables and then perform the aggregate operations to generate the new datasets and analyze the data. Develop an Enterprise Data Lake Application using Bigdata tools like Hive, Spark Streaming, Kafka, Sqoop, Oozie, Spark, Shell Scripting, Python, SQL, HDFS, Streamsets and Tidal.Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies. Provide automation solutions for the jobs which are run manually to generate frequent reports which are utilized by the business team for data analysis.

Work Locations :

Various unanticipated work locations throughout the United States; relocation may be required. Must be willing to relocate.

Minimum Qualifications Education :

Bachelor – Computer Science