Databricks Engineer - 12 month agency Contract

Duration: 1 year

Working Hours: 8.30am - 6pm, Monday - Fridays. Standby and shift work may be required

Location: Central

Responsibilities

• Design, develop, and maintain scalable data solutions for data generation, collection, processing, and analytics.
• Develop and maintain production-grade data pipelines using Databricks, PySpark, Python, and SQL.
• Perform data migration, data transformation, data processing, and integration across multiple systems.
• Build and optimize ETL/ELT pipelines to ensure reliable and efficient data movement across systems.
• Query, clean, transform, and validate large datasets using SQL and PySpark on Databricks.
• Construct complex SQL statements including joins, subqueries, CTEs, aggregations, window functions, and data transformation logic.
• Implement data quality checks, validation rules, and reconciliation processes to ensure data accuracy and consistency.
• Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.
• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and reliability.
• Identify and resolve data pipeline failures, performance bottlenecks, and data quality issues.
• Develop innovative data solutions to improve data accessibility, usability, and operational efficiency.
• Collaborate with data engineers, analysts, application teams, and business stakeholders to understand requirements and translate them into technical solutions.
• Participate in data migration and integration activities across source and target systems.
• Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics
• Explore and prototype automation solutions to streamline repetitive or manual data workflows.
• Participate in system integration testing (SIT), user acceptance testing (UAT), QA activities, and production deployment.
• Develop and maintain technical documentation, including data mappings, technical specifications, pipeline documentation, and test cases.
• Create user stories and acceptance criteria, manage JIRA tickets, and participate in Agile/Scrum ceremonies and sprint activities.
• Work closely with cross-functional teams to investigate and resolve data-related issues.
• Stay updated with emerging trends and best practices in Databricks, Spark, data engineering, and cloud data platforms.

Requirements:

• Bachelor’s Degree in Computer Science, Information Technology, Data Science, Engineering, or a related discipline.
• At least 4 years of relevant hands-on Data Engineering experience, with strong practical experience in developing and maintaining data pipelines.
• Must have real industry/production-grade experience with Databricks and PySpark, such as developing, deploying, and supporting live production data solutions.
• Strong proficiency in SQL, with the ability to independently construct and optimize SQL statements for data extraction, transformation, validation, and analysis.
• Strong hands-on experience with PySpark for large-scale/distributed data processing.
• Hands-on experience developing ETL/ELT data pipelines and performing data migration/transformation.
• Strong understanding of Apache Spark architecture and distributed data processing concepts.
• Proficiency in Python for data processing, automation, and scripting.
• Experience working with Databricks notebooks, jobs/workflows, clusters, and data processing environments.
• Experience troubleshooting and optimizing production data pipelines.
• Understanding of data quality, data validation, reconciliation, and monitoring practices.
• Experience with Git or other version control systems.
• Experience with SIT/UAT, system testing, QA, and production deployment.
• Hands-on experience with Power BI, Tableau, or equivalent dashboard/visualisation tools.
• Good understanding of Agile development methodologies, including JIRA, user stories, acceptance criteria, and sprint management.

Interested parties, please click the "Apply Now" below.

We regret that only shortlisted applicants would be notified.

Evonne Png Wen Xuan | REG No : R22105531

PERSOL SINGAPORE PTE LTD | EA License No : 01C4394

By sending us your personal data and curriculum vitae (CV), you are deemed to consent to PERSOL Singapore Pte Ltd (""PERSOL"") and its local and overseas subsidiaries and affiliates to collect, use and disclose your personal data to prospective employers/companies based in any country for purposes of evaluating suitability for employment, conducting reference checks, administering employment related services, complying with Government’s health advisories and such other purposes stated in our Privacy Policy.

We may use artificial intelligence (""AI"") and machine learning technologies to support our recruitment and placement services, including screening resumes, assessing qualifications and experience, and identifying skills and job matches. Where permitted by applicable law, we may also use such personal data to develop, train, test, improve and/or fine-tune AI and machine learning technologies that support our recruitment and business operations.

For more information on how we collect, use, disclose and protect your personal data, please refer to our Privacy Policy: https://www.persolsingapore.com/policies. If you wish to withdraw your consent, including consent relating to the use of your personal data for AI-enabled recruitment activities and AI development activities, please email us at [email protected].