Eray Turkel

Senior ML Research Engineer · LLM Post-Training & Evaluation · Causal ML · Statistics · Stanford Ph.D.

I’m an ML researcher working on post-training and evaluation for AI systems. Most of my work is focused on measurement, evaluations, and data: building evaluations and benchmarks whose results can be trusted, quantifying uncertainty in LLM-judges, and designing datasets and reward functions for RL post-training. I approach these with tools from statistics and causal inference.

At Roblox, I work on end-to-end LLM post-training for agentic coding models focused on game development: synthetic data generation, reinforcement learning with verifiable and rubric-based rewards, and the evaluation infrastructure surrounding all these efforts. I maintain Open Game Eval, Roblox’s open-source LLM evaluation framework for game-development tasks.

Previously, at Google Search (AI Overviews), I built LLM-as-judge systems for factuality and groundedness: judge calibration, uncertainty quantification, and hybrid human-LLM pipelines deciding when to trust an automated judge versus escalate to human review. This work fed into fine-tuning and reward design for Search’s generative AI products.

Before that, on Google’s Causal Inference team, I built ML and experimentation systems across Maps, Ads, YouTube, and Play: the Bayesian models and statistical tooling behind a novel crossover experiment on Maps, measuring routing-algorithm interventions across the 10 largest US cities (published in Nature Cities), sales-intervention models for Ads affecting millions of dollars in operations (Doubly robust dose-response modeling, presented in Joint Statistical Meetings), the YouTube Hype small-creator bonus mechanism, and a hierarchical Bayesian system for price experimentation on Google Play. Earlier, at Uber, I built variance-reduction methods for marketplace experimentation and worked on identifying spillovers in switchback experiments.

My Stanford Ph.D. (with an M.S. in Statistics) combined machine learning, statistics, and causal inference with published work in PNAS and ACM WWW.

Current independent projects

Conformal inference and risk control for LLM-as-a-judge uncertainty Distribution-free, finite-sample uncertainty quantification and risk control for LLM-judge scores using frozen-encoder embeddings. Showed embeddings sharpen interval efficiency (tighter prediction intervals) and improve error triage: catching more errors within a fixed human-review budget, with a formal guarantee bounding the error rate of everything auto-accepted, using 9 datasets across different domains.

Sensitivity framework for Bradley-Terry LLM preference leaderboards Developed a Rosenbaum-style sensitivity framework using Chatbot Arena’s public data releases, showing top model rankings flip under very small shifts in prompt or judge composition.