We are seeking an experienced Data Engineer with a strong background in AWS toolsets to advise, develop, and maintain data engineering solutions within the AWS Cloud ecosystem. This role involves designing, building, and managing both batch and real-time data pipelines using various AWS services.Key Responsibilities:Design, build, and operate batch and real-time data pipelines using AWS services like AWS EMR, AWS Glue, Glue Catalog, and Kinesis.Develop data layers on AWS RedShift, Aurora, and DynamoDB.Migrate data using AWS DMS and manage data integration using AWS Data Platform components such as S3, RedShift, RedShift Spectrum, AWS Glue with Spark, AWS Glue with Python, Lambda functions with Python, AWS Glue Catalog, and AWS Glue Databrew.Build data pipelines for Data Warehouses and Data Lakes using AWS Kinesis and Managed Streaming for Apache Kafka.Utilize open-source technologies, including Apache Airflow and dbt, as well as Spark/Python or Spark/Scala on the AWS Platform.Schedule and manage data services on AWS, ensuring efficient operation and integration.Mandatory Requirements:Experience working on traditional Data Warehousing with any database (Oracle, DB2, or SQL Server; Redshift experience is a plus).Strong SQL skills with the ability to write complex queries using analytical functions.Prior experience working on the AWS platform.Proficient in Python programming for data engineering.Experience with PySpark/Spark.Working knowledge of data pipeline tools like Apache Airflow.Preferred Skills and Expertise:Experience with DBT.Familiarity with Agile working environments.Proven ability to troubleshoot and resolve production issues under a DevOps model.A consistent track record of identifying opportunities to improve performance and quality within data ecosystems.Experience monitoring system performance and ensuring high-quality data operations.If the above looks of interest, please make an application and Experis will be in contact with you.Rowan MollisonSenior Consultant, Experis
#J-18808-Ljbffr