AI Engineer – LLM Ops & Evaluation

Darum lohnt es sich

Responsibilities Own the LLMOps pipeline: evaluate infrastructure, prompt optimization loop, and the production integration that turns experiments into reliable customer-facing features Design evaluation strategy per output type: decide when to use deterministic evals (exact match, schema validation, embeddings) versus LLM-as-judge, and build the rubrics, test datasets, and human‑review loops that make the system trustworthy Drive prompt engineering and optimization across all LLM operations in the product: moving from hand‑tuned prompts to a measurable, iterative process Pick the right tool for each problem: some things are LLM problems, some are embedding + classical NLP problems, some are deterministic logic Run the production side of AI features: observability (Langfuse/LangSmith/ similar), cost and latency engineering, incident response when an LLM feature degrades Build human‑in‑the‑loop workflows: review queues, feedback ingestion, labeling, so production signal feeds back into evals and prompt iteration Mentor our AI & Analytics Intern and contribute to how we build the AI team over time Requirements 3+ years of hands‑on experience building and shipping ML/AI systems in production (we care more about what you've shipped than years on a CV) Have shipped an LLM evaluation or prompt optimization pipeline, not just used LLMs in a project, but owned the loop Strong hands‑on experience with LLM‑as‑judge, including its variance problems and concrete techniques for controlling them Solid foundation in classical NLP and ML ops: embeddings, semantic similarity, entity matching, classification, fuzzy matching Informed opinions on deterministic vs.

You're familiar with prompt regression and have strategies for managing it Strong Python Excellent English communication, written and verbal: we discuss nuanced technical tradeoffs daily with the founding team and customers Comfort with ambiguity: you can run experiments on real data, build intuition for this domain, and know when to stop iterating #J-18808-Ljbffr LLM-based evals, from experience Production judgment: you've owned cost and latency tradeoffs, observability, and incident response for an LLM‑powered feature.