Using simulation to teach robots would unblock the massive scale of robotics data that seems to be missing right now. The issue is usually sim-to-real transfer; MolmoBOT aims to solve this with sufficient diversity of simulation data, so that no real data is necessary for robot deployment. It has a few parts: MolmoBot-Engine produces diverse pick-and-place demonstrations across many different simulated environments; MolmoBot-Data is a dataset of 1.7 million expert trajectories for item manipulation and pick-and-place tasks; and they train a number of VLA policies which can achieve strong performance on real-world pick-and-place tasks with zero real robot data. Rose Hendrix, Snehal Jauhri, Abhay Deshpande, and Maya Guru join us to give us the details. To learn more, watch RoboPapers episode 113 with Chris Paxton and Ruijie He! Abstract A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.7 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot (a Molmo2-based VLM with a flow-matching action head), MolmoBot-Pi0 (replicating the π₀ architecture for controlled comparison), and MolmoBot-SPOC (a lightweight policy for edge deployment). Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments, reaching 79.2% success rate on real-world tabletop pick-and-place, outperforming π₀.₅ at 39.2%. Learn More Project Page: https://allenai.github.io/MolmoBot/ ArXiV PDF: https://arxiv.org/pdf/2603.16861 Transcript Produced by ChatGPT and Riverside. Introducing MolmoBot Chris Paxton: Hey everybody, and welcome to another episode of RoboPapers. Today we’re excited to have the team behind the MolmoBot paper here to talk about their work. Would you like to introduce yourselves? Rose Hendrix: I’m Rose Hendrix. Until recently, I was a research scientist at AI2, and now I’m at a startup. Abhay Deshpande: I’m Abhay Deshpande. I conducted this work while I was a predoctoral researcher at AI2, and now I’m a PhD student at Princeton University. Maya Guru: I’m Maya Guru. I’m currently a predoctoral researcher at AI2, and I’m looking for PhD positions for this fall. Chris Paxton: Awesome. Would you like to get started and tell us a little about what you’ve been working on? Rose, maybe you can share your screen? Abhay Deshpande: I’ll pull it up. Rose Hendrix: Thanks, Abhay. On a previous episode of RoboPapers, the MolmoSpaces team came on to talk about their work, which is closely related to this. MolmoSpaces is about scaling synthetic environments and evaluation. MolmoBot is about cashing out that potential: scaling synthetic data generation, then training policies on that data to achieve zero-shot sim-to-real transfer for manipulation. MolmoBot is a data engine and a family of policies. We show that the policies transfer in large-scale simulation evaluations and in the real world, both for tabletop pick-and-place and for mobile manipulation on the RB-Y1 platform. That includes opening doors, drawers, and cabinets, as well as object manipulation and pick-and-place. Generating synthetic robot data at scale Abhay Deshpande: As Rose mentioned, we have this large framework, MolmoSpaces, for synthetic data and large-scale simulation. Within that, we built MolmoBot-Engine, a data engine for generating extremely diverse, large-scale datasets. The basic idea is to roll out heuristic demonstrators in these environments, then apply very aggressive randomization: camera extrinsics and intrinsics, textures, lighting, colors, and all sorts of other things. You keep rolling that out and scaling with compute, and you get high-throughput, large-scale data. For a concrete example, you sample an arbitrary scene from MolmoSpaces. There are more than 100,000 scenes available. Then you sample a workspace, choose an object for a pick-and-place operation, place a receptacle for the object, and put the robot in the workspace. From there, we randomize lighting, the robot’s initial configuration, inertial and frictional parameters, and other aspects of the scene. Then we roll out a scripted demonstrator. Since we have access to privileged information in simulation, a fairly simple heuristic policy can perform well under a wide variety of conditions. Of course, there are so many things we could randomize: textures, lighting, robot configurations, camera poses, task instructions, templates, and much more. How do we decide which ones matter? What we found is that a good philosophy is basically: yes. If you can randomize it, you generally should. That extreme diversity goes a long way toward making policies robust in the real world. We generate several different modalities. By default, we have two external cameras and a wrist camera. Since we’re using the DROID setup, we replicate those cameras in simulation: ZED 2 cameras for the external views and a ZED Mini for the wrist camera. We also have a randomized GoPro view. The GoPro has a very wide field of view and significant fisheye distortion. We apply post-processing to warp the rectified image so it looks like what the real GoPro would produce. In addition to RGB and depth, we save the camera intrinsics, since we’re randomizing them. We also save task-relevant object keypoints, which lets us compute visibility heuristics or do image-space reasoning downstream. For actions, we save all of the action representations rather than prescribing just one. That way, people using MolmoBot-Data can support a wide variety of downstream applications. We also save proprioception, joint positions and velocities, end-effector poses and twists, and other task-relevant privileged state: the demonstrator’s current phase, contact forces and points, grasps, door states, and so on. Finally, we label everything with rewards, success indicators, scene metadata, task prompts, and language referring expressions. Chris Paxton: One question: all these grasps and door states you’re talking about—where do those come from? Are those heuristics? Abhay Deshpande: Generally, they’re derived from the scripted heuristics or read directly from simulation. A door state might just be a joint angle, which we can access directly. The same goes for contact points and forces. Others, like rewards and success, are based on predicates: conditions that need to evaluate to true for us to say the task succeeded. Rose Hendrix: This was also motivated by the fact that knowing what the demonstrator is trying to do can be a useful training signal. That’s a strength of synthetic data. If you’re collecting human teleoperation demonstrations, it isn’t necessarily possible to extract that kind of nuanced information about what the operator intended at any given moment. RJ He: Was all the data scripted, or were there other sources? Abhay Deshpande: It was all scripted. We found that fairly simple heuristic policies go a long way. We have around 1.7 million demonstrations, and they’re all generated with scripted policies. By randomizing enough dimensions, you get surprisingly far. You can see in the data distribution that the scripted motions sometimes look a bit robotic—which, of course, is fitting for a robot. But we didn’t encounter major downstream problems, and the policies performed well in the real world. It might become more of an issue when co-training with dat… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
