Lead AI Engineer

Job Description

• Build and maintain end-to-end training data pipelines for Large Language Models (LLMs), covering data cleaning, deduplication, format conversion, quality filtering, data mixture/formulation, and version management.
• Design and dynamically adjust the data mix ratio between SFT training data and general-purpose Replay data.
• Manage the Prompt pool for distillation training, including prompt collection, deduplication, and balanced sampling across different domains.
• Establish a comprehensive data quality assurance framework, implementing a two-level automated and manual sampling-based quality control process covering format validation, semantic deduplication, and persona consistency checks.
• Collaborate with Agent Algorithm Engineers to convert ReAct multi-turn reasoning trajectories and Function Calling workflows into training-ready data formats.
• Manage training data versioning and traceability, ensuring that data snapshots and change records for each training iteration are fully documented and traceable.

Job Requirements

• Bachelor’s degree or above in Computer Science, Artificial Intelligence, Data Science, or a related field.
• Proficient in Python and familiar with data processing frameworks such as pandas, with hands-on experience processing large-scale text datasets.
• Experience building LLM SFT/RLHF data pipelines, with a solid understanding of best practices in data deduplication, quality filtering, and data mixture/formulation.
• Familiarity with ChatML/Chat formats, Function Calling data formats, and multi-turn conversational data structures.
• Experience in data quality measurement, monitoring, and data version management.