Control Strategies for Contact-Rich Robotics Systems

💥Introducing FACTR 2: learning external force sensing on commodity robot arms without needing dedicated sensors. We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies. FACTR 2 consists of: 1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors. 2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training. FIRST is motivated by a simple observation: policy failures usually do not happen in free space. They happen near contact, during brief pre-contact alignment and contact-rich interaction where fine corrections matter most. This work was done at Carnegie Mellon University with co-leads Steven Oh and Tony Tao, as well as Philip Han, Kenny Shaw, Satoshi Funabashi, Russ Salakhutdinov, and Deepak Pathak. Website: https://lnkd.in/gUrbhDjb Paper: https://lnkd.in/ggBYMTxZ

#Safety is crucial in human-robot interaction, especially for #mobile #robots. Without safety, #certification is impossible, and real-world applications are unfeasible. To address this, alongside our work on machine learning (which, despite their huge potential, are not yet certifiable), we use advanced #passivity and #powerbased control strategies to ensure optimal performance and safety. Recently, together with Theodora Kastritsi, we proposed a control strategy that decouples desired #dynamics from unintentional motion. This ensures changes in one direction do not affect the other. In the unintentional space, admittance parameters remain constant, while in the intended motion direction, inertia and damping gains adjust to provide compliance to the human user. We designed these variable terms to ensure a consistent response and perceived behavior, guaranteeing #strict #passivity under human force input for stable manipulation. In this video you can observe how smooth and robust the behavior of the proposed controller is in various trajectories and in comparison to advanced baseline controllers. Also, here is a link to our (open access) work: https://lnkd.in/dGfi7mJX

🦿 Can legged robots learn to control force and position… without force sensors? [📍 bookmark paper for later] This new work introduces a unified policy that enables legged robots to handle loco-manipulation tasks by learning both force and position control without using force sensors. It estimates contact forces from motion history and adapts in real time. Why this matters ✅ Jointly learns force and position control in one policy ✅ Works without force sensors by estimating forces from past states ✅ Handles complex tasks like force tracking and compliant behaviors ✅ Boosts imitation learning success by ~39.5% in contact-rich tasks Learn more 📄 Paper: https://lnkd.in/d2VnU4uE 📂 Project: https://lnkd.in/die5gyRA This brings us one step closer to agile, adaptable legged robots that can walk, push, and manipulate; All through a single, sensor-free policy.

Robotic loco-manipulation requires coordinated control of both contact force and position, yet many visuomotor policies treat them separately. We propose a unified control policy for legged robots that jointly models force and position — learned without force sensors. By simulating diverse interactions, the policy estimates contact forces from past states and compensates via position and velocity adjustments. Such a policy enables a wide range of manipulation behaviors under varying combinations of force and position inputs, including position tracking, force application, force tracking, and compliant robot behaviors. Additionally, we demonstrate that the learned policy enhances trajectory-based imitation learning pipelines by incorporating essential contact information through its force estimation module, achieving approximately ~39.5% higher success rates across four challenging contact-rich manipulation tasks compared to position-control policies. Experiments on both a quadruped and a humanoid demonstrate the robustness and versatility of our method across diverse scenarios.

New work on robot learning & teleop with force: FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learning Paper + Code: https://lnkd.in/eqfvtRRE Force information is crucial for contact-rich tasks, but behavior cloning policies tend to ignore robot force input if added naively, as policies tend to overfit to vision input. Key Idea: FACTR is a behavior cloning training curriculum that corrupts vision input with decreasing intensity throughout training. This helps the policy to properly attend to force data. Our policies perform and generalize better: ·  46% improvement in success rate for unseen objects ·  Exhibit emergent recovery behavior, unobserved in baselines We also add force-feedback to low-cost leader-follower teleop system, which is especially helpful for collecting data for contact-rich tasks: ·  64.7% higher task completion rate ·  37.4% reduced completion time ·  83.3% improvement in the subjective ease of use With Jason Liu, Yulong Li, Kenneth Shaw, Tony Tao, Deepak Pathak

TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models Arxiv: https://lnkd.in/eu9BkcM8 Project: https://lnkd.in/ein7jC4F How can robots sense and adapt to subtle contact dynamics—like distinguishing a failed plug-in from a successful one—when most VLA models only see pixels and text? TA-VLA explores how to integrate torque signals into Vision-Language-Action models, systematically analyzing when, where, and how to embed them. The result: torque-aware VLAs achieve higher success, robustness, and generalization on contact-rich tasks. 🔁 At a Glance 💡 Goal: Bridge the gap between perception-driven VLAs and force-aware manipulation by embedding torque signals. ⚙️ Approach: Decoder-side torque integration: Outperforms encoder integration for proprioceptive alignment. Torque history (single-token): Summarizes dynamics without disrupting decoder input patterns. Action–torque joint prediction: Anticipates physical consequences via unified diffusion modeling. 📈 Impact (Key Metrics) 🧪 Contact-rich tasks (Button, Charger, USB, Socket, Door Handle) π0 baseline: 0–16/20 SR π0 + torque obs + torque obj: 15–19/20 SR across tasks. 📊 Regular tasks (Bottle, Pouring, Stacking, Push, Drawer) Baseline strong, but torque-aware still improves (e.g., 19/20 vs. 17/20 in Bottle Pick-and-Place). 🤖 Cross-Model Generalization RDT baseline: 1/20 (Charger Plug) → 15/20 with torque integration. ⚡ Cross-Embodiment Generalizes from one robot arm (ALOHA) to another (ROKAE SR) without retraining—successfully inserting EV charging connectors. 🔬 Experiments 🦾 Robots: Dual-arm ALOHA + ROKAE SR. 📐 Inputs: RGB (multi-view), text, joint angles, torque from motor currents. 🎯 Tasks: 10 total (5 contact-rich, 5 regular). 🛠 How to Implement 1️⃣ Encode torque → single token → decoder alongside joint angles. 2️⃣ Aggregate torque history (past 2s) → compressed via MLP. 3️⃣ Train with joint action–torque diffusion loss for anticipatory learning. 📦 Deployment Benefits ✅ Sensorless force awareness (from joint currents). ✅ Robust retry behavior (detects failed attempts, retries until success). ✅ Generalizes across tasks, models, and embodiments. ✅ Lightweight integration—training/inference efficiency preserved. Takeaway TA-VLA shows that touch matters as much as vision and language. By embedding torque into VLAs, robots gain a sixth sense for contact, unlocking more reliable manipulation in the wild. Follow me to know more about AI, ML and Robotics!

🔧 Precise manipulation meets generalization -- with just 32 demos and 45 minutes of interaction. Robots are getting better at learning from large-scale data -- just like we’ve seen in vision and language. But when it comes to precise tasks like inserting plugs, swiping cards, putting keys in locks or plugging USBs, scale alone isn’t enough. These contact-rich tasks demand millimeter-level accuracy, and collecting diverse, high-quality data is difficult. This leads to an unwanted tradeoff: generalization vs precision. We introduce VisuoTactile Local (ViTaL) policies -- a framework that leverages the complementary strengths of vision and touch to achieve generalizable, precise control for contact-rich manipulation. Our framework has two components: 🧠 Global policy (e.g., a pretrained VLM) handles coarse semantic localization. ✋ Local ViTaL policy takes over for the last-mile of precise, contact-rich execution. 💥 With just 32 demos per task and 45 min of real-world RL, ViTaL achieves >90% success on 4 contact-rich tasks -- inserting plugs, swiping cards, putting keys in locks and plugging USBs -- in cluttered, unseen environments. ViTaL policies can be trained in the lab and deployed in kitchens, homes and offices without any retraining! So how do you train a ViTaL policy? Two simple steps: 1️⃣ Behavior Cloning with semantic augmentations for robust visual generalization. This policy excels at reaching, but fails about ~50% of the time at the contact-rich portion of the task. 2️⃣ Visuotactile Residual RL effectively leverages tactile feedback for offset-based refinement, while maintaining the generalizability of the behavior cloning phase. 🔑 Key insights: 1️⃣ Tactile sensing is critical -- removing it drops performance by ~40%. 2️⃣ Egocentric vision offers consistent spatial context tied to the robot’s frame, enabling deployment on new robots. 3️⃣ Semantic augmentations improve generalization under scene and spatial variations. 4️⃣ Residual RL with strong visual encoders can boost task performance while preserving robustness. This work would not be possible without Zifan Zhao's relentless pursuit of precise policies that actually generalize, Siddhant Haldar's invaluable insights on policy learning and residual RL, and consistent feedback from Jinda Cui and Lerrel Pinto. For more details and videos: https://lnkd.in/eXP5xTht Check out our paper for a comprehensive ablation study: https://lnkd.in/e-wARCsB Open-source code: https://lnkd.in/ezTYsD4Q

🚨Paper Alert 🚨 ➡️Paper Title: Touch begins where vision ends: Generalizable policies for contact-rich manipulation 🌟Few pointers from the paper 🎯Data-driven approaches struggle with precise manipulation; imitation learning requires many hard-to-obtain demonstrations, while reinforcement learning yields brittle, non-generalizable policies. 🎯Authors of this paper introduced “VisuoTactile Local (ViTaL)” policy learning, a framework that solves fine-grained manipulation tasks by decomposing them into two phases: 🧵a reaching phase, where a vision-language model (VLM) enables scene-level reasoning to localize the object of interest,  🧵and a local interaction phase, where a reusable, scene-agnostic ViTaL policy performs contact-rich manipulation using egocentric vision and tactile sensing. 🎯This approach is motivated by the observation that while scene context varies, the low-level interaction remains consistent across task instances. 🎯By training local policies once in a canonical setting, they can generalize via a localize-then-execute strategy. 🎯ViTaL achieves around 90% success on contact-rich tasks in unseen environments and is robust to distractors. 🎯ViTaL's effectiveness stems from three key insights: (1) foundation models for segmentation enable training robust visual encoders via behavior cloning; 2) These encoders improve the generalizability of policies learned using residual RL; and (3) Tactile sensing significantly boosts performance in contact-rich tasks. 🎯Ablation studies validate each of these insights, and they demonstrated that ViTaL integrates well with high-level VLMs, enabling robust, reusable low-level skills. 🏢Organization: New York University Shanghai, New York University, Honda Research 🧙Paper Authors: Zifan Zhao, Siddhant Haldar, Jinda Cui, Lerrel Pinto, Raunaq Bhirangi 📝 Read the Full Paper here: https://lnkd.in/gkqFaMh7 🗂️ Project Page: https://lnkd.in/gDaF7Uvi 🧑💻 Code: https://lnkd.in/gRFfuMTR 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.