Big Data Engineering Lead

Job Description

We are seeking an experienced Big Data Engineering Lead to design, develop and lead scalable enterprise data platforms, real-time data pipelines and analytics solutions. The successful candidate will work with large-volume datasets, cloud platforms, distributed processing technologies and modern AI/GenAI solutions.

Key Responsibilities

• Lead the design and development of scalable Big Data Engineering and Data Analytics platforms.
• Design and implement batch and real-time data pipelines using Python, PySpark, Spark Structured Streaming and Kafka.
• Develop data ingestion, transformation and processing workflows using AWS, S3, Airflow and orchestration tools.
• Design data models, data warehouses, data lakes/lakehouse architectures and high-performance analytics solutions.
• Develop and optimize complex SQL queries, ETL processes and data processing workflows.
• Build REST APIs and backend services using FastAPI/Flask for data and analytics applications.
• Implement data replication, archival, reconciliation, quality monitoring and governance processes.
• Deploy scalable data applications on AWS, including containerized environments and CI/CD pipelines.
• Lead development of GenAI/LLM, RAG and Agentic AI solutions for enterprise analytics and natural-language data querying.
• Provide technical leadership, mentoring and guidance to data engineering teams.
• Collaborate with business and technology stakeholders to translate requirements into scalable data solutions.

Requirements

• Degree in Computer Science, Information Technology, Engineering, Data Science or a related discipline.
• 8+ years of experience in Data Engineering / Big Data / Data Analytics.
• Strong hands-on experience with Python, SQL, PySpark, Apache Spark and Kafka.
• Experience in AWS cloud data services, including S3 and related data engineering technologies.
• Strong experience with Airflow or equivalent workflow orchestration tools.
• Experience with Snowflake, Hadoop/HDFS, Hive or other enterprise data platforms.
• Strong understanding of ETL, data warehousing, data modelling and distributed data processing.
• Experience developing REST APIs using FastAPI or Flask.
• Exposure to GenAI, LLM, RAG, LangChain/LangGraph or Agentic AI is an advantage.
• Strong analytical, problem-solving and technical leadership skills.
• Experience working in Agile software development environments.