RoboPapers

Chris Paxton and Michael Cho

Chris Paxton & Michael Cho geek out over robotic papers with paper authors. robopapers.substack.com

  1. 4 hr ago

    Ep#92: Human Universal Grasping

    How can we learn robot grasping from egocentric human video alone? General-purpose dexterous manipulation learning will require a lot of data, and yet robot data is hard to find at scale. Better leveraging human data, then, will be key to general-purpose dexterous manipulation. Kevin Wu, Irmak Guzey, and Dandan Shan join us to talk about HUG: Human Unversal Grasping. They collect a dataset of 1M frames (27.8 hours) of egocentric human grasping data, and use it to train a new flow-matching algorithm to predict hand pose. They can then predict human hand poses and retarget them to robot hands, allowing for a massive improvement on a wide variety of zero-shot robot grasping tasks in everyday scenes. To learn more, watch Episode 92 of RoboPapers with Michael Cho and Chris Paxton today! Abstract Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: this https URL Learn More Project Page: https://grasping.io/ ArXiV: https://arxiv.org/abs/2606.17054 Github: https://github.com/KevinyWu/hug This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

  2. 2 days ago

    Ep#91: RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

    Memory is one of the most important problems in robotics. Long horizon memory is key for a variety of robot manipulation problems. However, there exist no good benchmarks for understanding progress in how well generalist robot policies can understand language. Yinpei Dai and Yuejiang Liu made RoboMME as a solution: it’s a large benchmark which shows 16 different robot tasks, like counting objects or mastering timing. They show 14 different memory-augmented generalist policies across these different benchmarks. It’s an incredibly thorough and interesting result, aimed at driving forward this core robotic capability. To learn more, watch Episode 91 of RoboPapers with Michael Cho and Chris Paxton! Abstract Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits their systematic understanding, comparison, and progress measurement. To address these challenges, we introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates temporal, spatial, object, and procedural memory. We further develop a suite of 14 memory-augmented VLA variants built on the π0.5 backbone to systematically explore different memory representations across multiple integration strategies. Experimental results show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#91: RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

  3. 15 Jul

    Ep#90: From Capable Controllers to Deployable Humanoid Systems

    We want humanoid robots to be able to perform complex, long-horizon tasks in the real world — putting away the groceries or cleaning a room, for example. This requires diverse loco-manipulation skills, which can be easily parameterized to handle object affordances and interact with the world around it safely. In HANDOFF, Lizhi Yang proposes a 10-D learned whole body controller, which can be converted to whole body actions, and which can be used by a VLM-driven agentic planner to perform complex, multi-step manipulation actions in the real world. We then discuss how it’s possible to deploy such controllers in the real world, how to make them safe around people via controlled barrier functions and safety functions. This enables humanoids which can move safely through dynamic, crowded environments in the real world. Learn more in Episode 90 of RoboPapers, with Michael Cho and Chris Paxton. Abstract For a humanoid robot to be deployed in the real world, the choice of command space (i.e., the interface between task planning and whole-body control) is crucial. Existing whole-body controllers typically demand dense kinematic or spatial references that planners struggle to synthesize from task semantics. We instead propose a compact, explicit interface that is intuitive, general, modular, and expressive enough for diverse loco-manipulation skills. To this end, we introduce HANDOFF, a single humanoid whole-body controller that follows this interface and is distilled via multi-teacher KL distillation under a context-conditioned gating scheme into a mixture-of-experts student from three complementary specialists: whole-body motion tracking with safety-filtered data, locomotion, and fall-recovery. On the Unitree G1, HANDOFF matches state-of-the-art velocity tracking and offers one of the largest robust manipulation workspaces. We further demonstrate hardware feasibility through multiple natural-language-driven task roll-outs, powered by a VLM-driven agentic planner with no task-specific data or controller fine-tuning. Learn More Project page for HANDOFF: https://lzyang2000.github.io/HANDOFF/ Code: https://github.com/lzyang2000/HANDOFF ArXiV: https://arxiv.org/abs/2606.06493 Safe-SAGE on ArXiV: https://arxiv.org/abs/2603.05497 SHIELD on ArXiV: https://arxiv.org/abs/2505.11494 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#90: From Capable Controllers to Deployable Humanoid Systems

  4. 8 Jul

    Ep#89: Contact Grounded Policy

    Contact-rich manipulation is still very challenging for robotics. Problems like opening a jar, or in-hand reorientation of an object, require making repeated contact with different parts of a robot’s hand, and this is hard to do with pure vision. Instead, research is moving towards using tactile sensors in combination with visual policies. But what’s the best way to learn how to handle multi-point contact? Zhengtong Xu and Yeping Wang tell us about their new work Contact-Grounded Policy (CGP). CGP predicts future robot state and tactile feedback, and predicts this into actions for a compliant robot controller so that a four- or five-finger robot hand can perform complex tasks involving precise manipulation, delicate grasping, and tool use. To learn more, watch Episode #89 of RoboPapers, with Chris Paxton and Jiafei Duan. Abstract Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion-policy baselines. Learn More Project page: https://contact-grounded-policy.github.io/ ArXiV: https://arxiv.org/abs/2603.05687 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#89: Contact Grounded Policy

  5. 1 Jul

    Ep#88: DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation

    Human skin plays an important role in how we interact with the world and robustly manipulate objects. It’s not just important when we can’t see things with out eyes, but when we want to pick up something heavy, or apply a very specific amount of force. So, it makes sense to want to give robots skin. Enter DexSkin: a soft, deformable electronic skin which can be applied across different surfaces and used to cover robot hands or fingers. Suzannah Wistreich and Baiyu Shi talk to us about their work building DexSkin, showing how it’s useful for policy learning, including online reinforcement learning, and how it' can be calibrated and policies transferred across sensors. They also open sourced their code and methods for building the sensors. To learn more, watch Episode #88 of RoboPapers now, hosted by Chris Paxton and Jiafei Duan! Abstract Human skin provides a rich tactile sensing stream, localizing intentional and unintentional contact events over a large and contoured region. Replicating these tactile sensing capabilities for dexterous robotic manipulation systems remains a longstanding challenge. In this work, we take a step towards this goal by introducing DexSkin. DexSkin is a soft, conformable capacitive electronic skin that enables sensitive, localized, and calibratable tactile sensing, and can be tailored to varying geometries. We demonstrate its efficacy for learning downstream robotic manipulation by sensorizing a pair of parallel jaw gripper fingers, providing tactile coverage across almost the entire finger surfaces. We empirically evaluate DexSkin's capabilities in learning challenging manipulation tasks that require sensing coverage across the entire surface of the fingers, such as reorienting objects in hand and wrapping elastic bands around boxes, in a learning-from-demonstration framework. We then show that, critically for data-driven approaches, DexSkin can be calibrated to enable model transfer across sensor instances, and demonstrate its applicability to online reinforcement learning on real robots. Our results highlight DexSkin's suitability and practicality for learning real-world, contact-rich manipulation. Please see our project webpage for videos and visualizations: this https URL. Learn More ArXiV: https://arxiv.org/abs/2509.18830 Project Page: https://dex-skin.github.io/ Github: https://github.com/sdwistreich/dexskin Datasets: https://huggingface.co/datasets/swistreich/dexskin This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#88: DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation

  6. 18 Jun

    Ep#87: MolmoAct 2: An open foundation for robots that work in the real world

    There are few truly open models in the world, including both weights and data. However, these models are crucial for research and development of new systems — they help us learn which data is important and help develop new capabilities for deploying robots in the real world. MolmoAct2 provides a foundation for open research into robotics. It is associated with its own open dataset, an open-data action tokenizer, and a reasoning variant which predicts depth tokens. And people have actually been using it across the community, running experiments in their own labs or homes. Haoquan Fang and Jiafei Duan tell us more. Watch Episode 87 of RoboPapers, with Michael Cho and Chris Paxton, now! Abstract Vision-Language-Action (VLA) models aim to provide a single generalist controller for robots, but today’s systems fall short for real-world deployment. Frontier models are closed; open-weight alternatives are tied to expensive hardware; reasoning-augmented policies pay prohibitive latency for their grounding; and fine-tuned success rates remain below the threshold for dependable use. We present MolmoAct2, a fully open action reasoning model built for practical deployment, advancing its predecessor, MolmoAct along five axes. (1) MolmoAct2 is built on top of our new Molmo2-ER, a VLM backbone specialized for spatial and embodied reasoning, trained on a 3.3M-sample corpus with a specialize-then-rehearse recipe. (2) We release three new robot datasets spanning low-to-medium cost platforms: MolmoAct2-BimanualYAM Dataset, 720 hours of teleoperated bimanual trajectories that constitute the largest open bimanual dataset to date; MolmoAct2-DROID Dataset, a quality-filtered Franka subset of DROID; and MolmoAct2-SO100/101 Dataset, a quality-filtered SO-100/101 subset. (3) We train and release MolmoAct2-FAST Tokenizer, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments. (4) We design a new VLA architecture to graft the discrete-token VLM into the flow-matching continuous-action expert via per-layer key-value (KV) conditioning. (5) we propose MolmoAct2-Think, an adaptive-depth reasoning variant that re-predicts depth tokens only for scene regions that change between timesteps, retaining geometric grounding at a fraction of prior latency. In the most extensive empirical study of any open VLA to date, spanning 7 simulation and real-world benchmarks, MolmoAct2 outperforms strong baselines including π0.5, while Molmo2-ER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks. We release model weights, training code, and complete training data. Learn More Project page: https://allenai.org/blog/molmoact2 Code: https://github.com/allenai/molmoact2 ArXiV: https://arxiv.org/pdf/2605.02881v1 And check out our episode on the original MolmoAct: This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#87: MolmoAct 2: An open foundation for robots that work in the real world

  7. 12 Jun

    Ep#86: RISE: Self-Improving Robot Policy with Compositional World Model

    Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more! Abstract Despite the sustained scaling on model capacity and data acquisition, Vision-Language-Action (VLA) models remain brittle in contact-rich and dynamic manipulation tasks, where minor execution deviations can compound into failures. While reinforcement learning (RL) offers a principled path to robustness, on-policy RL in the physical world is constrained by safety risk, hardware cost, and environment reset. To bridge this gap, we present RISE, a scalable framework of robotic reinforcement learning via imagination. At its core is a Compositional World Model that (i) predicts multi-view future via a controllable dynamics model, and (ii) evaluates imagined outcomes with a progress value model, producing informative advantages for the policy improvement. Such compositional design allows state and value to be tailored by best-suited yet distinct architectures and objectives. These components are integrated into a closed-loop self-improving pipeline that continuously generates imaginary rollouts, estimates advantages, and updates the policy in imaginary space without costly physical interaction. Across three challenging real-world tasks, RISE yields significant improvement over prior art, with more than +35% absolute performance increase in dynamic brick sorting, +45% for backpack packing, and +35% for box closing, respectively. Learn More Project Page: https://opendrivelab.com/RISE/ ArXiV: https://arxiv.org/abs/2602.11075 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#86: RISE: Self-Improving Robot Policy with Compositional World Model

  8. 4 Jun

    Ep#85: Tutor Intelligence

    Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz Khan, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more! Learn More Blog post: https://tutorintelligence.com/blog/building-a-100-robot-data-factory-toward-factory-ready-ai This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

    Ep#85: Tutor Intelligence
See All (92)

About

Chris Paxton & Michael Cho geek out over robotic papers with paper authors. robopapers.substack.com

Information

  • Creator
    Chris Paxton and Michael Cho
  • Years Active
    2025 - 2026
  • Episodes
    92
  • Rating
    Clean
  • Copyright
    © Chris Paxton
  • Show Website
    RoboPapers

You Might Also Like