Cover is a fast-growing Insurtech startup making insurance accessible, transparent, and easy — for individuals and businesses.
We are looking for a driven, highly skilled
Senior Machine Learning Engineer (AI Evaluation & Reliability)
to join our AI team. This is not a traditional model-training role — we are seeking an engineering and evaluation expert to build automated systems that inspect, benchmark, and ensure the reliability of complex AI and Agent-based workflows at scale. If you have a rigorous analytical mindset, strong statistical foundations, and a passion for making production AI systems bulletproof, this role is for you.
What You'll Do:
Design, build, and scale automated evaluation pipelines combining deterministic tests, model-based grading (LLM-as-a-Judge), and Ground Truth validation
Implement trace and behavior analysis to catch failures, validate tool-calling workflows, and ensure real-world task execution
Build and maintain continuous evaluation frameworks, benchmark datasets, and automated regression suites
Develop rigorous evaluation metrics for domain accuracy, data layer integrity, and quality of generated insights
Establish production quality monitoring, real-time error classification, and statistical anomaly detection
Build human-in-the-loop workflows to continuously benchmark, calibrate, and improve automated evaluation engines
Partner closely with engineering teams to establish quality standards, prevent regressions, and govern production releases
What We're Looking For:
7+ years of software engineering and machine learning experience, including 3+ years hands-on with production LLM and AI Agent systems — Must
Proven track record designing and deploying automated evaluation frameworks in production — Must
Strong mathematical, statistical, and data validation background (hypothesis testing, error analysis, calibration, anomaly detection) — Must
Advanced proficiency in Python, SQL, and core ML/data libraries
Practical experience with cloud infrastructure (AWS/GCP), Docker, and CI/CD pipelines
B.Sc. in Computer Science, Mathematics, Statistics, or a related quantitative field (M.Sc. — strong advantage)
Experience in FinTech, InsurTech, or enterprise SaaS — strong advantage
A proactive problem-solver with high attention to detail who thrives in a fast-paced, dynamic startup environment