data scientist for AI quality

Описание:

Finom is a European tech startup headquartered in Amsterdam that develops an all-in-one financial B2B solution for entrepreneurs. Its mobile-first platform integrates banking functions, accounting, financial management, invoicing, and AI-enabled accounting, and operates across key EU markets.

Задачи:

• Own and extend the offline evaluation suite across products, including datasets, judges, and metrics;
• Build and maintain online quality dashboards covering resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, and latency;
• Mine failure patterns from real traffic, convert them into regression cases, and propose fixes to Product and domain experts;
• Harden evaluation methodology through judge stability and non-determinism handling;
• Translate metrics into decisions through weekly syncs and clear trade-offs;
• Collaborate closely with AI engineers, Product, and domain experts across the company.

Требования:

• Build end-to-end analyses with Python and SQL;
• Have a solid foundation in statistics, including sampling, hypothesis testing, variance, and noisy metrics;
• Start from the business question rather than the tool;
• Have 3+ years of experience in analyst or data scientist roles, including at least one year in a product context;
• Be curious about and fluent with AI-assisted coding, or genuinely excited to become fluent quickly;
• Nice to have: experience in quality analytics for ML systems such as ranking, recommendations, or classification;
• hands-on experience evaluating LLM applications including RAG, agents, tool use, or judges;
• experience building LLM agents through side projects, toy builds, or personal experiments.

Условия:

• Work in the EU;
• Remote or hybrid work and travel across Europe are available;
• Stock Options Program available to every team member;
• Resources and opportunities for continuous personal and professional development;
• Work & Swim Program with one month in a corporate apartment in Cyprus;
• Equal opportunity employer with support for employee well-being.