Information Technology - Lead Data Engineer

Job Description

The lead data engineer is a senior software developer with strong softwareengineering skills who is responsible for building custom open-source-baseddata ingestion and MLOps platforms. He/she has deep appreciation of thecomplexity of the data engineering process, such as the challenges of dataingestion involving large or near-real-time datasets, the maintenance of highdata quality, and the importance of automation for increasing pipelinerobustness and reducing the need for human intervention.

Responsibilities

• Be an effective distributed-system implementer in the following coreactivities:

o Design and develop data engineering services and their ecosystem usingdistributed databases (relational, columnar, graph, in-memory); orchestration(Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis,Spark).

oDesign and develop MLOps production pipelines; provide technical support todata scientists/ML engineers by getting their ML/DL models deployed at scaleand meeting SLAs on both cloud and on-premises GPU and CPU instances.

o Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestionmore fault tolerant.

o Design and develop intuitive, highly automated, self-service data platformfunctions for business users.

o Design, build, and operate scalable and reliable data pipelines on theDatabricks platform.

• Explore, evaluate and champion the introduction of next-generationtechnologies in the data-ingestion workflow. Participate in project planningand provide technical guidance on cloud architecture for data projects.

Requirements

• BS in Computer Science or other related discipline is required. Advanceddegrees in Computer Science (PhD, MS) are highly desirable.

•5+ years of relevant industry experience in some or most of the followingtechnical areas:

o Advanced programming skills in Python. Conversant with data structures andalgorithm design.

o Experience in building data pipelines (including data collection,warehousing, processing, analysis, monitoring, and governance) usingopen-source data ingestion platforms.

o Intermediate-level knowledge and experience with AWS cloud components andbest practices. Good understanding in deploying data stores such as S3,RedShift, Elasticache, PostgreSQL, and EMR.

o Hands on experience with Databricks workspace, cluster management, AI Agentcapabilities, and job orchestration

o Prior experience in modern software development is required (such as webfrontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming(using e.g. Python, Java, Scala, or C#).