AI Systems Engineer

AI Systems Engineer

Role Overview｜岗位概述

We are looking for an AI Systems Engineer (Model Routing) with strong hands-on experience in automated model routing, multi-model systems, AI evaluation, and production AI infrastructure.

我们正在寻找一位 AI Systems Engineer (Model Routing)，候选人需要在自动化模型路由、多模型系统、AI 评估及生产级 AI 基础设施方面具备扎实的实际经验。

The core responsibility of this role is to design and build systems that can automatically determine which underlying foundation model should handle each incoming task or request, based on factors such as task type, complexity, model capability, quality requirements, latency, cost, and reliability.

该岗位的核心职责是设计并建设能够针对每一个输入任务或请求，自动判断应由哪个底层基础模型处理的系统，并综合考虑任务类型、复杂度、模型能力、质量要求、延迟、成本及可靠性等因素。

You will also build the evaluation and production infrastructure required to measure model capabilities, translate evaluation signals into routing decisions, and continuously improve routing performance in production.

你还将负责建设相应的评估及生产基础设施，对不同模型能力进行量化，并将评估结果转化为模型选择及路由决策，持续优化生产环境中的路由表现。

Key Responsibilities｜岗位职责

Model Routing & Decision Systems｜模型路由与决策系统

• Design and implement automated model-routing and dynamic model-selection systems across multiple proprietary and open-source foundation models.设计并实现跨多个闭源及开源基础模型的自动化模型路由及动态模型选择系统。
• Determine the most appropriate model for each request based on signals such as task type, complexity, model capability, expected quality, latency, inference cost, reliability, and operational constraints.根据任务类型、复杂度、模型能力、预期质量、延迟、推理成本、可靠性及运营约束等信号，为每个请求自动选择最适合的模型。
• Develop model-selection, arbitration, escalation, fallback, and ensemble mechanisms for different production scenarios.针对不同生产场景建设模型选择、模型仲裁、能力升级、智能降级及多模型组合机制。
• Explore and apply approaches such as classification, ranking, learned routing, contextual bandits, or other data-driven decision methods where appropriate.根据实际场景探索并应用分类、排序、学习型路由、Contextual Bandit 或其他数据驱动决策方法。
• Continuously improve routing policies using offline evaluation, online experimentation, production feedback, and controlled A/B testing.综合离线评估、线上实验、生产反馈及受控 A/B 测试，持续优化模型路由策略。
• Measure routing performance against fixed-model baselines and quantify improvements in quality, latency, cost, and reliability.将路由系统与固定模型方案进行比较，并量化其在质量、延迟、成本及可靠性方面带来的实际收益。

This role focuses on model-level routing based on task requirements and model capabilities, rather than simple load balancing, availability-based failover, static configuration switching, or agent/tool routing.

本岗位关注的是基于任务需求及模型能力进行的模型级自动化路由，而非简单的负载均衡、基于可用性的故障切换、静态配置切换或 Agent / Tool Routing。

Evaluation & Benchmarking｜评估与基准体系

• Define task taxonomies, evaluation dimensions, scoring criteria, and acceptance thresholds required to compare models and support routing decisions.建立任务分类、评估维度、评分标准及验收门槛，为模型比较及路由决策提供可靠依据。
• Design and maintain benchmark suites, golden datasets, annotation standards, and regression test sets.设计并维护 Benchmark、Golden Dataset、标注标准及回归测试集。
• Build automated evaluation pipelines using deterministic checks, LLM-as-a-Judge, rubric-based scoring, pairwise comparison, and human evaluation where appropriate.根据场景建设自动化评估流程，综合使用确定性校验、LLM-as-a-Judge、Rubric 评分、Pairwise Comparison及人工评估。
• Validate evaluation methods against human judgments and monitor judge consistency, bias, and drift.通过人工判断验证评估方法，并持续监测 Judge 的一致性、偏差及漂移。
• Establish continuous evaluation and regression mechanisms for changes to models, prompts, data, and routing policies.针对模型、Prompt、数据及路由策略变更建立持续评估和回归机制。
• Translate evaluation results into actionable model-selection and routing policies rather than treating evaluation as an isolated benchmarking exercise.将评估结果真正转化为可执行的模型选择及路由策略，而非将 Evaluation 作为独立的 Benchmark 工作。

Multi-model & Production AI Infrastructure｜多模型与生产级 AI 基础设施

• Design scalable infrastructure for integrating and operating multiple proprietary, open-source, and self-hosted models.设计可扩展的多模型基础设施，支持闭源、开源及自托管模型的统一接入和运行。
• Build model gateways, unified APIs, version-management mechanisms, routing infrastructure, and model lifecycle management capabilities.建设模型网关、统一 API、版本管理、路由基础设施及模型生命周期管理能力。
• Support traffic governance, model rollout, rollback, replacement, and controlled experimentation.支持流量治理、模型灰度发布、回滚、替换及受控实验。
• Build monitoring, logging, tracing, failure analysis, and performance diagnostics for both model calls and routing decisions.针对模型调用及路由决策建设监控、日志、链路追踪、故障分析及性能诊断能力。
• Establish feedback loops capturing model performance, routing outcomes, failure cases, user signals, and human-review results.建立数据反馈闭环，持续采集模型表现、路由结果、失败案例、用户信号及人工审核结果。
• Build production-grade AI systems with strong standards for reliability, scalability, security, testing, and maintainability.按照可靠性、可扩展性、安全性、测试性及可维护性要求建设生产级 AI 系统。

Qualifications｜任职要求

• Hands-on experience building production AI, machine-learning, or distributed systems.具有生产级 AI、机器学习系统或分布式系统的实际建设经验。
• Demonstrated hands-on experience with automated model routing, dynamic model selection, model arbitration, ensemble systems, or multi-model decision systems.具有自动化模型路由、动态模型选择、模型仲裁、多模型组合或多模型决策系统的实际经验。
• Experience making model-selection decisions based on task requirements and differences in model capability, quality, latency, cost, or reliability.具有根据任务需求以及不同模型在能力、质量、延迟、成本或可靠性方面的差异进行模型选择的实际经验。
• Experience designing evaluation frameworks, benchmark datasets, scoring methodologies, regression pipelines, or continuous evaluation systems.具有评估框架、Benchmark Dataset、评分方法、回归流程或持续评估体系的设计及建设经验。
• Strong understanding of trade-offs among model quality, latency, inference cost, reliability, and operational risk.深入理解模型质量、延迟、推理成本、可靠性及运营风险之间的权衡关系。
• Experience integrating and operating multiple foundation models through third-party APIs, self-hosted deployments, or open-source serving infrastructure.具有通过第三方 API、自托管部署或开源 Serving Infrastructure 接入并运行多个基础模型的经验。
• Ability to translate ambiguous product or business requirements into measurable evaluation criteria, technical designs, experiments, and production systems.能够将模糊的产品或业务需求转化为可量化的评估标准、技术方案、实验及生产系统。
• Strong analytical and problem-solving skills, with the ability to validate technical decisions using data and experiments.具备较强的分析及问题解决能力，能够通过数据和实验验证技术决策。