Speakers: Max Niederman (head of quality, host), Stephen Yang (senior software engineer), Ege Erdil (CTO). Max [00:00:00] I’m Max Niederman, head of quality at Mechanize, and I’m here today with Stephen Yang, a senior software engineer here, and Ege Erdil, our CTO, to talk about evals. I want to start off with a very basic question, which is just: what is an eval exactly? And how does it relate to benchmarks and to RL environments? Stephen [00:00:18] An eval, as well as an RL environment and benchmarks, are just various ways to quantitatively measure the performance of an LM at a task. And this is very important whether you’re a user of a model trying to decide what model to use for a specific use case, or a researcher at a lab trying to improve your models at a task. The way that an eval works is that it consists of multiple test cases, and each of these test cases has three components: a prompt, which is the task that the model needs to complete; an environment that the model is situated in; and a grader that takes the work that the model completes and assigns a quantitative score to it. This is the same three components as what makes up an RL environment. So the difference is largely in how it is used. An RL environment is used during training with various RL methods like GRPO, for example, and that’s directly used for updating the weights of the model. Whereas an eval is used by researchers at a lab, for example, to pick what RL method to use or to tune the hyperparameters. This is one of the most fundamental differences between the two: how they’re used. And this has a lot of cascading effects as well. For example, one is the price that you might be willing to pay for an eval versus an RL environment. Because RL environments are run during training, you need much more of them, because the RL method is going to be much more sample inefficient than a researcher. And because you need so many more of them, you end up wanting to buy cheaper RL environments and just buying a very large quantity of them. But on the other hand, human researchers, their time is very scarce, and they’re also relatively sample efficient learners. So at the stage where the human researcher is involved, you want higher quality test cases with less noise in them. And so labs are willing to invest more in evals than in RL environments on a per test case basis, for example. Ege [00:02:05] I think that’s not the only reason why you might want to spend more money on evals. The other difference is that in a typical RL run, a single task usually will be used maybe a couple of times. You don’t want to reuse the same task too many times in RL, because if there’s not enough diversity, exactly because of the sample efficiency being poor, you won’t get enough generalization. So you really care about diversity. But that means that the amount of compute that you spend on inferencing any single task is kind of small. While if you have an eval suite that you really trust, then you’re probably going to be running it very frequently, much more frequently than you will be running a single task in an RL environment. So you’re spending way more compute on this task, which also means that the quality of the task is kind of complementary to that, which means you should also be willing to spend more money on it. It’s the same kind of thing where if you’re buying an expensive car, then you don’t want it to have very cheap tires, right? If you’re buying a very cheap car, that might make sense because the rest of the car is also very cheap, and so saving on costs by making the tires very cheap makes sense. But if you’re buying something expensive, which is the entire bundle of you take the eval and then it produces a score, then trying to save money by making the data part really cheap, when you’re going to spend a very large amount of money on the compute part, doesn’t really make a lot of sense. Stephen [00:03:31] And this is something that might be surprising to a lot of people. Because when people think about evals, they might think about public benchmarks like SWE-bench Verified, for example, and they might only see a new public announcement about a score once a month. So could you talk more about why labs need to run evals so many times? Ege [00:03:47] Yeah. So even for public evals, it’s not a matter of whether an eval is public or private, but rather whether the model it’s being run on is public or private. Labs do very few model releases compared to how many different checkpoints of a model they have. Because the way a model gets built is that you start from some initial state, some initial model. Maybe it’s a base model, maybe it’s a model you’ve already post-trained in some way and you’re doing further post-training on top of it. But it just gradually changes over time. People come up with many different ways of training the model. Those are applied maybe in parallel. Maybe there are merge steps where the results of those training steps get merged and somehow you merge the weights. And over time, every day, you have a slightly different checkpoint of the model. And every day people are doing different things and changing things around. And then you want to make sure that those changes people are making are not making the model worse and are indeed making it better, so you’re sort of gradually getting better. And once you reach a performance threshold that you’re satisfied with, it makes sense to spend a lot of resources on optimizing the inference and rolling this out to your inference clusters, because there is a cost to doing that. You can’t serve too many different checkpoints at once because you want the batch sizes to be large. There are other overheads with serving like 200 different models. Labs usually want to have, for various reasons, a small number of models that they release. This means the number of scores that a lab releases is going to be way less than the number of scores they measure internally, because for every model they release, they’re going to have like hundreds of checkpoints internally that they have never released. But they still need the scores on those checkpoints, because otherwise, if they are experimenting, say, with a new training method, they might go totally off the rails. They might think it’s working, but if they have no measurement of whether it’s working well or not, they might just go totally off the rails, for example. And that’s the reason why evals are run so many more times internally than results will be announced on them externally. Max [00:05:56] Yeah. So as Stephen alluded to, within labs there’s kind of this two-level optimization structure where researchers are trying to optimize the model at the high level, and then they use this lower level inner optimizer that’s like an RL algorithm that automatically updates a huge number of weights. And the outer optimization loop runs much less frequently. So you have maybe on the order of like hundreds of samples from an eval, whereas you have many, many orders of magnitude more for the RL environment. And then you also have users as maybe even a third tier. They’re trying to optimize over which lab’s model they choose, but they really aren’t going to spend that much time on this. They’re going to select from maybe a few different, maybe six or ten different models, maybe at the very high end. And so they really do not need to run their benchmarks very many times. And in practice, they just look at the benchmarks that either the labs or some other organization publishes. Ege [00:07:03] Yeah, I think the point about evals being run at a much lower frequency than RL environments might not be true. Because if you think about the wall clock time you spend, you’re running many, many tasks in parallel when you’re doing RL, but you’re not necessarily running many more tasks in sequence, because each rollout takes the same amount of time. So if you’re interpreting frequency literally, I don’t think that’s true. Max [00:07:31] Yeah. So I guess it’s more the throughput of samples rather than the number of times that any given task will be sampled from. I mean, actually, is that true? Because during an RL run, any given task will be run at a much higher parallelism than during evals. So if you’re thinking how many times does a task run, I don’t know if that’s true. Ege [00:07:56] Because when you do an eval, you often want to get the-- it depends on how large your eval suite is. But even if it’s hundreds of tasks, usually you’ll want to run the same task many dozens of times to get a low amount of noise. Otherwise, in an incremental checkpoint, it’s very difficult to tell if there was actually an improvement or not. If the model quality difference is very large, then the signal to noise ratio is high enough, even with a small number of samples. But usually with incremental checkpoints, where you only perturb it a very small amount, that’s not going to be true. So I think you’re still going to want to do a lot of parallel samples, even in eval. Max [00:08:38] Yeah. I guess what’s definitely true is that you will spend more compute on all of your tasks during an RL environment than during evals. Across all of your tasks, I think that’s true. Ege [00:08:45] Across any one task, that’s not true. Because the same one task in an eval will probably be used way more frequently than during an RL run. Max [00:08:57] Yeah. I do think it’s valuable to zoom out a little bit, though, and just reiterate this point that all three of these things are very similar. I think this is kind of underappreciated. But really, it’s the same problem to make a good eval, to make a good benchmark, to make a good RL environment, in a bunch of ways. Actually, I want to hear you guys talk about how exactly you go about this problem. Say you want to measure a very concrete ability, a very
