Overcoming Data Bottlenecks in Robotics Projects

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Robotics data is expensive and slow to collect. A lot of videos are available online, but not readily usable by robotics because of lack of action labels. AMPLIFY solves this problem by learning Actionless Motion Priors that unlock better sample efficiency, generalization, and scaling for robot learning. Our key insight is to factor the problem into two stages: The "what": Predict the visual dynamics required to accomplish a task The "how": Map predicted motions to low-level actions This decoupling enables remarkable generalizability: our policy can perform tasks where we have NO action data, only videos. We outperform SOTA BC baselines on this by 27x 🤯 AMPLIFY is composed of three stages: 1. Motion Tokenization: We track dense keypoint grids through videos and compress their trajectories into discrete motion tokens. 2. Forward Dynamics: Given an image and task description (e.g., "open the box"), we autoregressively predict a sequence of motion tokens representing how keypoints should move over the next second or so. This model can train on ANY text-labeled video data - robot demonstrations, human videos, YouTube videos. 3. Inverse Dynamics: We decode predicted motion tokens into robot actions. This module learns the robot-specific mapping from desired motions to actions. This part can train on ANY robot interaction data - not just expert demonstrations (think off-task data, play data, or even random actions). So, does it actually work? Few-shot learning: Given just 2 action-annotated demos per task, AMPLIFY nearly doubles SOTA few-shot performance on LIBERO. This is possible because our Actionless Motion Priors provide a strong inductive bias that dramatically reduces the amount of robot data needed to train a policy. Cross-embodiment learning: We train the forward dynamics model on both human and robot videos, but the inverse model sees only robot actions. Result: 1.4× average improvement on real-world tasks. Our system successfully transfers motion information from human demonstrations to robot execution. And now my favorite result: AMPLIFY enables zero-shot task generalization. We train on LIBERO-90 tasks and evaluate on tasks where we’ve seen no actions, only pixels. While our best baseline achieves ~2% success, AMPLIFY reaches a 60% average success rate, outperforming SOTA behavior cloning baselines by 27x. This is a new way to train VLAs for robotics which dont always start with large scale teleoperation. Instead of collecting millions of robot demonstrations, we just need to teach robots how to read the language of motion. Then, every video becomes training data. led by Jeremy Collins & Loránd Cheng in collaboration with Kunal Aneja, Albert Wilcox, Benjamin Joffe at College of Computing at Georgia Tech Check out our paper and project page for more details: 📄 Paper: https://lnkd.in/eZif-mB7 🌐 Website: https://lnkd.in/ezXhzWGQ

One of the biggest constraints in robot learning is data. Rhoda AI released work on their Direct Video-Action (DVA) Model that breaks through this bottleneck. Instead of predicting robot actions directly, the model imagines what the future should look like as video, then translates that into motion, basically treating robot control as real-time video gen. To do this, Rhoda pre-trains a causal video model on internet-scale video to learn physics, object behavior, and material properties. Today's VLAs do most of this learning through robot demonstrations, which are expensive to teach via teleoperation. From there, Rhoda post-trains on a thin layer of task-specific data. For example, just 11 hours of robot data for a complex, bimanual decanting task. Most VLAs operate on only a few frames of context, which makes long-horizon tasks difficult without hand-engineered scaffolding. The DVA retains hundreds of frames of visual history, so it can track where it is in a multi-step task end-to-end. Cool to see competing ideas in robot learning, and excited for what this means with regards to data efficiency and production reliability for end-to-end models.

I was listening to one of my favorite podcasts last week, Unsupervised Learning by Redpoint Ventures. They had Karol Hausman and Danny Driess (Research Scientist) from Physical Intelligence. Around the 33 minute mark of the podcast they mentioned the need for a tool or infrastructure to help them understand what is in their dataset, particularly given the massive amount of multimodal, time-series data that robotics generates. They outlined what they'd want in such a tool: - Decide what data to collect - Build machinery around understanding the collected data - Understand the quality of the data collected so far - Perform quality assurance at scale - Execute language annotations correctly at scale - Determine how much more data is needed for the model - Identify the optimal strategy for data collection - Provide a bird's-eye view understanding of the entire dataset I was excited by that, cuz, well, I work at FiftyOne and we have a tool that does just that... For understanding what's in your dataset, FiftyOne lets you visually explore massive datasets interactively. When they talked about needing a "bird's-eye view," that's literally what our embedding visualizations provide - you can see your entire dataset in embedding space, revealing clusters, gaps, and outliers. The QA at scale problem? FiftyOne has built-in queries to find labeling mistakes and inconsistent patterns across millions of samples. And for data collection strategy, it shows where your dataset has gaps and where models struggle - no more training for weeks to "get a signal." So I went to Physical Intelligence's Hugging Face org and found their "aloha_pen_uncap" dataset. I parsed it into FiftyOne format to see how well our tool would work with their data. In the process, I implemented a data loader for LeRobot format datasets, which means the entire robotics community can now load their datasets in FiftyOne and get all these benefits. The loader handles the multimodal nature of robotics data, parsing camera views, robot states, and actions. What became clear when I loaded their dataset: - You can visually browse task executions and see patterns in successful vs failed attempts - Embedding visualizations shows clusters of similar robot behaviors - Quality issues like poor lighting or occlusions become immediately apparent It's all open source, and all you need to do to get started is `pip install fiftyone` to see what your data looks like in FiftyOne. The tool mentioned in the podcast already exists, and it's open source!

Closing the sim-to-real gap in humanoid robotics requires massive simulation throughput and high-fidelity physics validation. WPP recently detailed their engineering pipeline, showing how they reduced reinforcement learning cycle times for complex humanoid locomotion from 24 hours down to less than 60 minutes. The hardware architecture relies on Google Cloud’s new G4 VMs (powered by NVIDIA RTX PRO 6000 Blackwell GPUs) running NVIDIA Isaac Sim, integrated closely with DeepMind’s MuJoCo physics engine. The mechanics: The team mapped raw human mocap data (over 200 degrees of freedom) down to a constrained 29-DOF OpenUSD digital twin. By leveraging a P2P GPU topology to bypass central processing bottlenecks, the infrastructure executed over 3 billion simulations in under an hour. The virtual environment continuously introduced physical micro-variances—simulated pushes, shifting floor friction, and momentum changes—to train the model against the chaos of the real world. The resulting reinforcement learning model was condensed into a highly efficient ONNX policy and deployed directly to the physical robot. This edge policy processes live IMU and joint telemetry to output immediate, stabilized motor commands. Reaching this scale of simulation volume is the precise engineering mechanism that allows control policies to handle unstructured physical deployment. To support the research, Unitree has open-sourced the underlying RL code on GitHub. Blog post : https://lnkd.in/g4-gWzTP #Robotics #PhysicalAI #ReinforcementLearning #MuJoCo #GoogleCloud #IsaacSim #Engineering

Most robotics teams use simulation too late. The usual playbook is: Collect real-world data. Train the model. Find the failures. Then use simulation to fill the gaps. That works, but you end up spending millions and collecting lots of repeated data that are "empty calories". The better approach is to simulate first. Before collecting data, define the scenarios your system needs to handle: Different environments. Lighting conditions. Object types. Weather. Edge cases. Failure modes. That scenario grid becomes your test plan. Now every real-world data collection run has a purpose. You can see what you captured, what is missing, where the model performs well, and where it breaks. Simulation should not replace real-world data. It should guide it. Fail fast in sim, verify in real. Use simulation to define what data matters. Use real-world data to calibrate the simulator. Use both to continuously improve reliability. The best robotics teams will not be the ones that collect the most data. They will be the ones that know what data to collect that drives the highest change in performance and reliability.

🚀 RoboCade: Gamifying Robot Data Collection is out on arXiv — and I’m thrilled to share this collaborative work with the community! One of the biggest bottlenecks in robotics today is scaling human demonstration data for imitation learning. Traditional collection is costly, tedious, and limited to experts with access to hardware. So we asked: 👉 Can we make robot data collection accessible, engaging, and scalable — even for non-experts? That’s where RoboCade comes in: 🎮 A gamified remote teleoperation platform that transforms robot demo collection into an interactive game-like experience. 👥 Designed to engage general users — with visual feedback, progress bars, badges, leaderboards, and more — while still generating useful data for downstream policy training. Key results: ✔️ Remote players collected data that, when co-trained with traditional demos, boosted policy success on real tasks (+16 – 56%). ✔️ In user studies, beginners found RoboCade significantly more enjoyable and motivating than standard interfaces (+24%). ✔️ We also propose principles for gamified task design so the collected data actually helps with real manipulation challenges. Why this matters: 🔹 Broadening participation in robotics research beyond labs and experts 🔹 Intrinsic motivation rather than paying for data labeling 🔹 A scalable crowd-sourced pipeline for future robot learning systems Huge thanks to Suvir Mirchandani, Mia Tang, Jubayer Ibn Hamid, Michael Cho, and Dorsa Sadigh for the collaboration. 🔧🤝 Read the full paper on arXiv — and check out our demo videos at https://lnkd.in/gjyE6A5S #Robotics #ImitationLearning #HumanAI #Crowdsourcing #Gamification #MachineLearning

The robotics community has a name for it now: the 100,000-year data gap. You can't scrape robot training data the way you scrape text. It has to be built. And the two options most teams have — teleoperation and hand-authored simulation — are either too expensive to scale or too synthetic to trust at deployment. Here's the part that kept me up at night: Every time a robot hesitates, clips something, or triggers a safety stop in the real world, that's ground-truth data. It's the exact edge case your sim never generated. It has trajectory, context, spatial geometry, failure signature. And in the current workflow, it gets reset and discarded. The failure repeats. The training set stays thin. The sim-to-real gap stays wide. We built Reconstructiv to close that loop. When an incident happens on a real fleet, we detect it, capture the logs and video automatically, and reconstruct the event as a 3D scene — semantically labeled and simulation-ready. The edge case that just happened becomes a training asset before anyone opens a rosbag. Real-world incidents are the most valuable data in robotics. We built the pipeline to stop throwing them away. First look 👇 https://lnkd.in/gZd-M9qB If your team is building VLA or Diffusion Policy models and fighting the data pipeline problem, I'd genuinely love to talk. #PhysicalAI #Robotics #RoboticsML #SimToReal #TrainingData

One of the core challenges in robotics is making robots generalize across tasks, environments, sensors, and data sources. Traditionally, robot learning has been siloed: one model per task, per dataset, per modality. This leads to expensive retraining and poor adaptability. A recent paper from MIT CSAIL introduces PoCo (Policy Composition from and for Heterogeneous Robot Learning), a diffusion-based framework that directly tackles this bottleneck. 🔹The idea: Instead of training a single “one-size-fits-all” model, PoCo allows us to train multiple policies (e.g., task-specific, domain-specific, behavior-constrained) separately, then compose them at inference time. The policies are built as diffusion models over action trajectories, which makes their combination mathematically flexible. This could be a foundational step towards true generalist robots, capable of adapting to diverse and unpredictable environments. Full paper: https://lnkd.in/eim6Se4r