Eray Turkel

Senior ML Research Engineer · LLM Post-Training & Evaluation · Causal ML · Stanford Ph.D.

I’m an ML researcher working on post-training and evaluation for modern AI systems. I apply statistical rigor to the most difficult measurement and attribution problems in modern AI.

My career has been a progression through increasingly ambiguous versions of measurement and attribution problems. My current work targets what determines whether post-training actually produces the capabilities we intend: building evaluation systems whose measurements can be trusted, designing reward signals that generate genuine capability rather than reward hacking, and assigning credit correctly across long training trajectories. I approach these as ML problems informed by the tools of statistics and causal inference.

At Roblox, I work on end-to-end LLM post-training for agentic coding models focused on game development: synthetic data generation, supervised fine-tuning, reinforcement learning with verifiable and rubric-based rewards, and the evaluation infrastructure surrounding all these efforts. Part of this work is open sourced: I maintain Open Game Eval, Roblox’s open-source LLM evaluation framework for game-development tasks.

Previously, at Google Search (AI Overviews), I built LLM-as-judge evaluation systems, uncertainty quantification and calibration methods for LLM judges, and hybrid human-LLM evaluation algorithms that decided when to trust an automated judge versus escalate to human review, focusing on factuality and groundedness. This work fed directly into fine-tuning, reward design, and RLHF for Search’s generative AI products.

Before that, on Google’s Causal Inference team, I built ML and experimentation systems across Maps, Ads, YouTube, and Play: the Bayesian models and statistical tooling behind a novel crossover experiment on Maps, measuring routing-algorithm interventions across the 10 largest US cities (published in Nature Cities), sales-intervention models for Ads affecting millions of dollars in operations (Doubly robust dose-response modeling, presented in Joint Statistical Meetings), the YouTube Hype small-creator bonus mechanism, and a hierarchical Bayesian system for price experimentation on Google Play.

Earlier, at Uber, I built variance-reduction methods for marketplace experimentation and worked on identifying spillovers in switchback experiments. My Stanford Ph.D. combined machine learning, statistical modeling, and causal inference, with published work in PNAS and ACM WWW.

The questions I find most interesting these days: designing reward functions that correctly assign credit across long trajectories, and improving the quality and quantifying the uncertainty of LLM-as-judge evaluation methods.

A few current independent projects:

Conformal inference and risk control for LLM-as-a-judge uncertainty Distribution-free, finite-sample uncertainty quantification and risk control for LLM-judge scores using frozen-encoder embeddings. Showed embeddings sharpen interval efficiency (tighter prediction intervals) and improve error triage: catching more errors within a fixed human-review budget, with a formal guarantee bounding the error rate of everything auto-accepted, using 9 datasets across different domains.

Sensitivity framework for Bradley-Terry LLM preference leaderboards Developed a Rosenbaum-style sensitivity framework using Chatbot Arena’s public data releases, showing top model rankings flip under very small shifts in prompt or judge composition.