AWS Data Engineer

Roles & Responsibilities

• Design and architect AWS Data Lake/Lakehouse solutions across Landing, Transformed, and Curated/Consumption zones.
• Define and govern data architecture standards, patterns, and best practices.
• Design reusable ingestion pipelines for REST APIs, JDBC databases, S3, and SaaS connectors such as Salesforce via AWS AppFlow.
• Define data storage strategies covering hot, warm, and cold storage, encryption, and lifecycle policies.
• Develop and deploy pipelines using AWS Glue, Lambda, Step Functions, EventBridge, and API Gateway.
• Build batch and streaming transformation workflows using AWS Glue and Amazon Redshift.
• Implement pipeline orchestration, monitoring, logging, and notification frameworks.
• Develop and maintain the AWS Glue Data Catalogue, including schema evolution and metadata tagging.
• Implement data security and access controls using AWS Lake Formation, IAM, and Secrets Manager.
• Ensure compliance with data classification, retention, and audit requirements.
• Support data quality and observability frameworks.
• Monitor platform health, performance, and pipeline reliability, and troubleshoot failures and data quality issues.
• Maintain architecture documentation, pipeline configurations, and operational runbooks.
• Optimise AWS platform performance and cost efficiency.

Requirements

Essential

• 3–5 years of experience in data engineering, data architecture, or cloud infrastructure.
• Hands-on experience with Amazon S3, AWS Glue, Amazon Redshift, Lambda, Kinesis, Step Functions, EventBridge, AppFlow, and Lake Formation.
• Strong SQL skills and proficiency in Python or Scala.
• Experience designing and implementing Data Lake/Lakehouse architectures.
• Strong understanding of data governance, data cataloguing, and metadata management.
• Experience with batch and streaming data processing.
• AWS Certified Data Engineer – Associate, AWS Certified Solutions Architect, or equivalent.

Preferred

• Experience integrating Tableau or similar BI tools through Redshift or S3.
• Familiarity with MLOps and AI/ML deployment using AWS SageMaker.
• Salesforce integration experience using AWS AppFlow.
• Knowledge of CDC and incremental data loading patterns.
• Experience in a government or public sector data environment.