Senior Machine Learning Engineer (Senior) - Cover

Cover is a fast-growing Insurtech startup making insurance accessible, transparent, and easy — for individuals and businesses.

We are looking for a driven, highly skilled

Senior Machine Learning Engineer (AI Evaluation & Reliability)

to join our AI team. This is not a traditional model-training role — we are seeking an engineering and evaluation expert to build automated systems that inspect, benchmark, and ensure the reliability of complex AI and Agent-based workflows at scale. If you have a rigorous analytical mindset, strong statistical foundations, and a passion for making production AI systems bulletproof, this role is for you.

What You'll Do:

Design, build, and scale automated evaluation pipelines combining deterministic tests, model-based grading (LLM-as-a-Judge), and Ground Truth validation

Implement trace and behavior analysis to catch failures, validate tool-calling workflows, and ensure real-world task execution

Build and maintain continuous evaluation frameworks, benchmark datasets, and automated regression suites

Develop rigorous evaluation metrics for domain accuracy, data layer integrity, and quality of generated insights

Establish production quality monitoring, real-time error classification, and statistical anomaly detection

Build human-in-the-loop workflows to continuously benchmark, calibrate, and improve automated evaluation engines

Partner closely with engineering teams to establish quality standards, prevent regressions, and govern production releases

What We're Looking For:

7+ years of software engineering and machine learning experience, including 3+ years hands-on with production LLM and AI Agent systems — Must

Proven track record designing and deploying automated evaluation frameworks in production — Must

Strong mathematical, statistical, and data validation background (hypothesis testing, error analysis, calibration, anomaly detection) — Must

Advanced proficiency in Python, SQL, and core ML/data libraries

Practical experience with cloud infrastructure (AWS/GCP), Docker, and CI/CD pipelines

B.Sc. in Computer Science, Mathematics, Statistics, or a related quantitative field (M.Sc. — strong advantage)

Experience in FinTech, InsurTech, or enterprise SaaS — strong advantage

A proactive problem-solver with high attention to detail who thrives in a fast-paced, dynamic startup environment

View job on HireMe →