M20 - Data Engineer

Responsibilities

• Design and develop scalable backend services integrating with external vendor systems via RESTful APIs
• Implement robust data ingestion services including pagination and incremental/time-window retrieval
• Handle API error handling, retries, and idempotent processing to ensure reliable integrations
• Build and maintain data pipelines for processing large volumes of transactional and event data
• Perform data transformation, validation, and enrichment for downstream systems
• Support batch and near real-time data processing workflows
• Develop and deploy applications using Microsoft Azure services such as Azure Functions, App Services, Storage, Key Vault, APIM, and VNET
• Design scalable, resilient, and highly available cloud-native solutions
• Implement observability practices including logging, monitoring, and alerting
• Ensure data accuracy, consistency, and reconciliation across multiple data sources
• Support auditability and traceability of data flows for regulatory requirements
• Collaborate with external vendors on API contracts and integration designs
• Deliver high-quality, maintainable, and reusable software solutions
• Contribute to engineering standards, best practices, and continuous improvement initiatives
• Support CI/CD pipelines, automated testing, and continuous delivery practices

Requirements

Core Technical Skills

• Strong proficiency in Python (preferred) or backend languages such as Java or Node.js
• Strong SQL proficiency including PostgreSQL, MySQL, or T-SQL
• Solid experience building backend services and API integrations
• Experience working with RESTful APIs and JSON payloads
• Knowledge of secure integration patterns including API keys, TLS, certificates, and PGP

Good understanding of:

• Pagination, filtering, and incremental data retrieval
  Idempotency, retries, and distributed system failure handling
• Experience with data visualisation tools such as Power BI, including connectivity and configuration

Data Engineering & Processing

• Experience building batch or streaming data pipelines
• Familiarity with data transformation, validation, and data modelling
• Experience handling large datasets and high-volume ingestion systems
• Understanding of data reconciliation and audit requirements is an advantage

Databases

• Experience with relational databases such as PostgreSQL or MS SQL
• Knowledge of query optimisation and large-scale data handling

Cloud & DevOps

• Experience with cloud platforms, preferably Microsoft Azure
• Experience setting up and managing CI/CD pipelines using GitLab CI/CD or similar tools
• Familiarity with Docker/containerisation is an advantage

Observability & Reliability

• Experience implementing logging, monitoring, metrics, and alerting solutions
• Understanding of system reliability and fault-tolerant design principles