Challenges of Robotic Manipulation

MIT just showed a clever new way for robots to grip heavy and delicate objects — without crushing or slipping them. This has been a long-standing problem in robotics. To lift something heavy, a robot usually has to grip hard. But gripping hard concentrates force at a few contact points, which is exactly how fragile objects break. The challenge is contradictory: - A gripper needs to be flexible enough to snake into clutter and wrap around objects. - But it also needs to be strong enough to hold weight once the object is lifted. Most designs pick one and suffer on the other. - Stiff grippers hold weight but create damaging pressure points. - Very soft grippers are gentle, but they buckle, slide, or fail when the load gets heavy. MIT’s new approach, called loop closure grasping, sidesteps that tradeoff entirely. Here’s the idea: - The gripper starts as an open loop, with a free tip that can move around obstacles and form a wrap. - Once positioned, the tip locks back onto the base, turning the shape into a closed loop around the object. - In this closed state, the load is carried mainly by tension, like a sling — not by squeezing. Because the force is distributed around the loop, the gripper can stay very soft and compliant while still holding heavy objects without high-pressure contact points. The prototype uses inflatable “vine” beams that grow from the tip, then a clamp and winch to close the loop, tighten it, and finally deflate for a gentle hold. The result? Awkward, previously hard tasks become easy — like lifting a 6.8 kg kettlebell from a cluttered bin or pulling objects from three meters away. A really elegant example of how rethinking the mechanics can unlock entirely new robot capabilities.

Humanoid Dexterity: Why Our Robot Achieved 100% Success on Fruit Picking but 0% on Block Stacking. Some learnings after training vision-language-action models for humanoid manipulation: 📍 The Camera Dependency Problem: Models trained on fixed viewpoints fail catastrophically with 30° camera shifts. It’s not poor generalization—it’s optical illusion at the neural network level. 🤖 The Embodiment Gap: Using Apple Vision Pro for teleoperation, our operators needed 12 attempts to pick up an apple. Why? No depth perception. No force feedback. If humans struggle with these constraints, imagine what we’re asking AI to do. ⚡ Inference Bottleneck: VLMs run at ~5Hz. Smooth robot control needs 20Hz minimum. The dual-architecture approach (System 1 for fast control, System 2 for reasoning) helps but introduces trajectory discontinuities. The difference between our fruit-picking success and block-stacking failure is revealing: ✅ Fruit picking: Single object, large grasp tolerance, terminal success state ❌ Block stacking: Sequential precision tasks, force-sensitive placement, cumulative error propagation Between “great idea” and “working robot”: ➡️ 3 weeks converting Unitree teleop data to LeRobot format ➡️ Custom bridges between NVIDIA’s own tools (IsaacLab doesn’t natively interface with GR00T) ➡️ Force Feedback Vacuum: Without tactile sensing, grasping becomes binary (success/failure) rather than continuous adjustment. ➡️ Depth Blindness: Current RGB-only models lack the stereo vision humans take for granted. Adding RGB-D could be transformative. ➡️ And least we forget - the sim2real gap: COSMOS + IsaacSim could potentially generate 20-100x training data, but sim2real transfer remains challenging. Every viral humanoid demo represents hundreds of failed attempts and carefully controlled conditions. This isn’t fraud—it’s the difference between possibility and reliability. The path from demo to deployment isn’t just long—it’s filled with fundamental challenges that simply throwing compute at will not solve. Nuaced, interconnected techniques are the name of the game….And that’s exactly why it’s worth doing. #Robotics #VLA #PhysicalAI #DeepLearning #Humanoids #EngineeringReality #TechLeadership #RoboticsEngineering #AI #FoundationModels

Presenting FEELTHEFORCE (FTF): a robot learning system that models human tactile behavior to learn force-sensitive manipulation. Using a tactile glove to measure contact forces and a vision-based model to estimate hand pose, they train a closed-loop policy that continuously predicts the forces needed for manipulation. This policy is re-targeted to a Franka Panda robot with tactile gripper sensors using shared visual and action representa- tions. At execution, a PD controller modulates gripper closure to track predicted forces -enabling precise, force-aware control. This approach grounds robust low- level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks. #research: https://lnkd.in/dXxX7Enw #github: https://lnkd.in/dQVuYTDJ #authors: Ademi Adeniji, Zhuoran (Jolia) Chen, Vincent Liu, Venkatesh Pattabiraman, Raunaq Bhirangi, Pieter Abbeel, Lerrel Pinto, Siddhant Haldar New York University, University of California, Berkeley, NYU Shanghai Controlling fine-grained forces during manipulation remains a core challenge in robotics. While robot policies learned from robot-collected data or simulation show promise, they struggle to generalize across the diverse range of real-world interactions. Learning directly from humans offers a scalable solution, enabling demonstrators to perform skills in their natural embodiment and in everyday environments. However, visual demonstrations alone lack the information needed to infer precise contact forces.

I’m super excited to release a multi-year project we have been cooking at NVIDIA Robotics. Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new embodiment and/or scene, we'll have to recollect the dataset in this paradigm for the best perf. Key Idea: Since grasping is a well-defined task in physics simulation - why can’t we just scale synthetic data generation and train a GenAI model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. Introducing… 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, antipodal pinch, and underactuated pinch grippers ✅ Generalization to both partial and complete 3D point clouds ✅ Generalization to both single-objects and cluttered scenes  ✅ Modular design relies on other robotics packages and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct any mistakes of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et. al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse We're also releasing: 🔹 Simulation-based grasp data generation workflows 🔹 Standardized formats and gripper definitions 🔹 Full training infrastructure 📄 arXiv: https://lnkd.in/gaYmcfz4 🌐 Website: https://lnkd.in/gGiKRCMX 💻 Code: https://lnkd.in/gYR77bEh A huge thank you to everyone involved in this journey — excited to hear the feedback from the community! Joint work with Clemens Eppner, Balakumar Sundaralingam, Yu-Wei Chao, Mark T. Carlson, Jun Yamada and other collaborators. Many thanks to Yichao Pan, Shri Sundaram, Spencer Huang, Buck Babich, Amit Goel for product management and feedback. #robotics #grasping #physicalAI #simtoreal

I started 2026 in Edinburgh, Scotland, in a room full of people trying to teach machines how to touch the world. No humanoid theatrics. No glossy demos. Just fingers, sensors, and a quiet, uncomfortable truth: We still haven’t taught robots how to use their hands like we do. The 7th UK Robot Manipulation Workshop didn’t pretend we’re “almost there.” It focused on the real bottleneck for service robots, construction robots, and humanoids: grasping, adjusting, feeling, and recovering in messy human spaces. We’ve taught robots to see. We’ve taught them to move. We’ve even taught them to plan. But manipulation is still the hard part. In factories, we redesigned the world to fit the robot. In homes, hospitals, and public spaces, that luxury disappears. So, for my first Six Degrees of Robotics articles of 2026, may we hope this year is remembered not as the year robots got smarter, but the year we finally admitted what was holding them back. Not their brains. Their fingers. Read it here: https://lnkd.in/gmfCxmEy

To the untrained eye, it’s just boxes on a pallet. But when you zoom in, you realize how hard this actually is... Each box comes in a different size and weight. They arrive in random order. Some need to be placed at the bottom, and crushable boxes are placed at the top. You can’t collide with any of the surrounding geometry or the pallet. You have to maintain stability, respect reach limits, avoid robot singularities - and still move fast enough to keep up with the line. A simple-looking task that’s an absolute nightmare with traditional rule-based automation. At Jacobi Robotics, we use AI-powered decision-making and real-time motion planning to tackle exactly these kinds of challenges. No offline planning. The robot sees the box, figures out where it needs to go, and decides how to get there - on the fly. 👉 What’s the most deceptively hard robotic task you’ve come across in the field?

OpenAI published this robot hand paper in 2018. 8 years later, scaling dexterity is still one of robotics’ hardest problems. In this paper, OpenAI trained a Shadow Dexterous Hand to reorient objects inside the hand. Not just pick up an object. But rotate it, adjust it, and keep control without dropping it. It showed that dexterous in-hand manipulation could be learned in simulation with reinforcement learning, then transferred to a real robotic hand. Since 2018, robotics has changed a lot: • Foundation models became real • Dexterous hands became cheaper • Tactile sensing improved • Simulation improved • Humanoids became mainstream again But the hard part has not disappeared. The challenge is still what happens after the grasp: • The object slips • Contact changes • Pressure shifts • Small errors accumulate • The hand must adjust and recover This is why dexterous manipulation is still difficult to scale in the real world. But it is still a problem worth solving. Because even when grippers work well in a structured environments, robotic hands matter when interaction becomes complex, uncertain, and contact-rich. I explored this question in a longer essay: Why do robots still need hands? Link in the comments. —- https://lnkd.in/geWVqgXZ Post it here again in case you missed) - Leo 磊 Su

Forget backflips. Watch the glass. This is Figure's latest demo running Helix 02—a fully autonomous humanoid loading a dishwasher. No teleoperation, no speed-ups, just raw feedback loops handling fragile glass. This represents a massive shift in how we define robotic capability: The "hard skills" (lifting heavy boxes) are solved. The "soft touch" was the bottleneck. Here is why this specific motion matters: ✔️ Pixels-to-Torque Control In the past, robots followed rigid coordinates. If a glass was 1mm off, it shattered. Helix 02 connects camera pixels directly to motor torque. It doesn't just "see" the glass; it learns the physics of fragility and modulates force in real-time. ✔️ Whole-Body Intelligence Watch the hips. The robot isn't just moving an arm; it stabilizes its entire frame to support the hand's precision. This is "System 0" at work—a unified neural network managing balance and manipulation simultaneously, replacing 100,000+ lines of hard code. ✔️ The "Messy" Reality Industrial robots need structured assembly lines. Domestic robots need to handle the chaos of a sink. By mastering the dishwasher—a task with high variability and high consequences for failure—we are moving from "automation" (repeating a task) to "autonomy" (adapting to a task).

I’m pleased to share our latest paper, “ChicGrasp: Imitation-Learning-Based Customized Dual-Jaw Gripper Control for Manipulation of Delicate, Irregular Bio-Products,” now published in Advanced Robotics Research. 🔗 https://lnkd.in/ggfFQBMx In this work, we benchmarked a long-standing challenge in poultry processing: automating the grasp-and-rehang of slippery, deformable chicken carcasses, a task that continues to rely heavily on human labor despite decades of automation. Our approach combines task-specific hardware design with imitation learning. Evaluated on real broiler carcasses, the system achieved an 80.71% grasp-and-rehang success rate, including on previously unseen birds and under multiple disturbance conditions. To support reproducibility and future research, all CAD files and code have been released publicly. I’m grateful to the outstanding work of Amirreza Davar and our collaborators Wan Shou Yu She, who made this study possible, and we sincerely appreciate the funding support from USDA NIFA! #Robotics #ImitationLearning #AgriculturalRobotics #FoodAutomation #RobotManipulation

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho and Chris Paxton! Abstract: Leveraging human motion data to impart robots with versatile manipulation skills has emerged as a promising paradigm in robotic manipulation. Nevertheless, translating multi-source human hand motions into feasible robot behaviors remains challenging, particularly for robots equipped with multi-fingered dexterous hands characterized by complex, high-dimensional action spaces. Moreover, existing approaches often struggle to produce policies capable of adapting to diverse environmental conditions. In this paper, we introduce HERMES, a human-to-robot learning framework for mobile bimanual dexterous manipulation. First, HERMES formulates a unified reinforcement learning approach capable of seamlessly transforming heterogeneous human hand motions from multiple sources into physically plausible robotic behaviors. Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth image-based sim2real transfer method for improved generalization to real-world scenarios. Furthermore, to enable autonomous operation in varied and unstructured environments, we augment the navigation foundation model with a closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise alignment of visual goals and effectively bridging autonomous navigation and dexterous manipulation. Extensive experimental results demonstrate that HERMES consistently exhibits generalizable behaviors across diverse, in-the-wild scenarios, successfully performing numerous complex mobile bimanual dexterous manipulation tasks Project Page: https://lnkd.in/e-aEbQzn ArXiV: https://lnkd.in/eemU6Pwa Watch/listen: Youtube: https://lnkd.in/erzbkYjz Substack: https://lnkd.in/e3ea76Q8