Senior Data Engineer - PySpark & SQL

Role: Senior Data Engineer - PySpark & SQL

Relevant Experience Required: 5 Years

Job Description:

Data Transformation

• Query, clean, and transform datasets using SQL and Python on the Databricks platform.
• Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.

Analytics & Dashboard Development

• Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics.
• Partner with business stakeholders to understand data requirements and translate them into impactful visual insights.

Documentation & Knowledge Management

• Develop clear and comprehensive user guides for dashboards, charts, and data workflows in Databricks
• Maintain technical documentation (e.g., on Confluence) for analytics processes, standards, and best practices.

Additional Opportunities

• Explore and prototype automation solutions to streamline repetitive or manual workflows.
• Develop project management competencies, including:
• Creating user stories and acceptance criteria
• Managing JIRA tickets and sprint tasks
• Writing and executing QA test cases

Competencies & Requirements

Required Skills

• Proficiency in SQL and Python for data processing and automation.
• Hands-on experience in dashboard tools such as Tableau, Power BI, or equivalent visualisation tools.
• Familiarity with Databricks or similar big data/analytics platforms.
• Ability to produce high-quality technical documentation.

Additional Skills Preferred

• Experience in system testing, SIT/UAT, or quality assurance processes.
• Knowledge of Spark for distributed data processing.
• Understanding of Git or other version control systems.

Soft Skills

• Strong analytical mindset and problem-solving abilities.
• Good communication skills, especially in explaining technical concepts to non-technical users.
• Ability to work independently as well as collaboratively in cross-functional teams.
• Attention to detail and commitment to delivering high-quality work.