The Data Engineer is responsible for building and supporting the big data platform. As a Data Engineer, your responsibilities include:
Develop, test and support future-ready data solutions for customers across industry verticals.Develop, test and support end-to-end batch and near real-time data flows/pipelines.Demonstrate understanding in data architectures, modern data platforms, big data, ML/AI, analytics, cloud platforms, data governance and information management and associated technologies.Develop and demonstrate Proof of Concepts and Working Demos.Lead or collaborate with other internal/external consultants in consulting, workshops and delivery engagements.Mentor junior consultants in the practice and delivery engagements.Required ExperienceTo ensure success in the role you will possess the following skills:
6-10 years of hands-on data engineering expertise and experience in building at least one Python/Spark/Scala based data ingestion, transformation, and egress framework.Working experience on Data Lake Implementations on Azure cloud/Big Data technologies.Cloud experience in one or more of Azure, AWS, or GCP environment.Expertise with modern data warehouse design patterns and tools.Good understanding of data architecture principles including data modeling.Ability to tune Spark and Scala jobs for performance.Deep understanding of distributed computing systems.Familiarity with Data Warehousing concepts and physical Data modeling techniques.Able to stitch together a Real-time, Streaming, Kappa and Lambda solution.Proficient with the latest Big Data tools/technologies like Hive, Hadoop, Yarn, Kafka, Spark Streaming.Good knowledge in Azure cloud data technologies (Spark, ADLS2, CosmosDB, AKS, AEH). Experience not mandatory.Good temperament - building relationships and stakeholder management skills with the ability to interact with a diverse group of stakeholders and proactively provide actionable data.Familiarity with DevSecOps tools, methodologies - CI/CD (i.e., Jenkins, Bitbucket, GitHub), Azure DevOps i.e., best practices for code deployment.Working knowledge in job orchestration (Control-M, Airflow).Work with Architects, Technical Leads, and Business Teams and contribute to the development of technical designs.Strong knowledge of various data structures and the ability to extract data from various data sources.Provides technical database consultation on application development, global infrastructure, and other database administration efforts related to specific DBMS.Experience in writing complex SQL queries preferred.Ability to conduct data profiling, cataloguing, and mapping for technical design and construction of technical data flows.
#J-18808-Ljbffr