Data Engineer

Job Description & Requirements

Role Overview

This role is responsible for designing, building, and deploying a scalable and reliable data lakehouse platform on AWS. The Data Engineer will architect end-to-end data pipelines, establish data quality frameworks, and deliver production-ready solutions that form the foundation of the organisation's data infrastructure.

• Key Responsibilities
  Data Pipeline & Architecture Design
  Design and implement scalable ETL/ELT pipelines using AWS services including Glue, Step Functions, Lambda, and S3. Architect and build data lakehouse solutions leveraging Apache Iceberg or S3 tables, incorporating features such as schema evolution, partition evolution, and ACID transactions. Ensure pipelines are optimised for performance, cost, and reliability at scale.
• Data Quality & Governance
  Define, implement, and maintain automated data quality validation frameworks, establishing metrics and monitoring to ensure data accuracy, completeness, and consistency. Enforce data governance standards and ensure compliance across the data platform.
• Application Development & Deployment
  Write production-quality code and deploy solutions on AWS cloud infrastructure. Work with DevOps practices including CI/CD pipelines and infrastructure-as-code tools such as Terraform or CloudFormation to ensure repeatable and auditable deployments.
• Collaboration & Handover
  Work closely with cross-functional teams and communicate technical designs clearly to non-technical stakeholders. Produce thorough technical documentation to support the handover of developed solutions to the Day 2 operations team.
• Qualifications & Experience
  A degree in Computer Science, Data Engineering, Information Systems, or a related field is required. At least 7–10 years of experience in data engineering, ETL/ELT development, or data platform roles is expected. Strong hands-on experience with AWS services and proficiency in Apache Iceberg or similar open table formats is essential. Experience in a Government Commercial Cloud (GCC) environment is strongly preferred. AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are an advantage.
• Skills & Competencies
  Data Engineering & Architecture — Ability to design and build scalable, reliable data pipelines using modern data lakehouse architectures including Apache Iceberg and S3 tables. Data Quality & Governance — Strong proficiency in defining, measuring, and monitoring data quality metrics, implementing automated validation frameworks, and ensuring compliance with governance standards. Application Development — Skilled in writing production-quality code, optimising performance, and deploying solutions on cloud platforms. Technical — Deep understanding of Apache Iceberg features and S3 table optimization techniques. Collaboration & Communication — Ability to work effectively in cross-functional teams and communicate technical concepts clearly to non-technical stakeholders.