Advances in Neural Networks for Robot Tracking

What can half of GPT-1 do? We trained a 42M transformer called SONIC to control the body of a humanoid robot. It takes a remarkable amount of subconscious processing for us humans to squat, turn, crawl, sprint. SONIC captures this "System 1" - the fast, reactive whole-body intelligence - in a single model that translates any motion command into stable, natural motor signals. And it's all open-source!! The key insight: motion tracking is the one, true scalable task for whole body control. Instead of hand-engineering rewards for every new skill, we use dense, frame-by-frame supervision from human mocap data. The data itself encodes the reward function: "configure your limbs in any human-like position while maintaining balance". We scaled humanoid motion RL to an unprecedented scale: 100M+ mocap frames and 500,000+ parallel robots across 128 GPUs. NVIDIA Isaac Lab allows us to accelerate physics at 10,000x faster tick, giving robots many years of virtual experience in only hours of wall clock time. After 3 days of training, the neural net transfers zero-shot to the real G1 robot with no finetuning. 100% success rate across 50 diverse real-world motion sequences. One SONIC policy supports all of the following: - VR whole-body teleoperation - Human video. Just point a webcam to live stream motions. - Text prompts. "Walk sideways", "dance like a monkey", "kick your left foot", etc. - Music audio. The robot dances to the beat, adapting to tempo and rhythm. - VLA foundation models. We plugged in GR00T N1.5 and achieved 95% success on mobile tasks. We open-source the code and model checkpoints!! Check it out today: - Website: https://lnkd.in/gjNW_Y9g - Code and weights: https://lnkd.in/g8AnBUne - Whitepaper: https://lnkd.in/gVCaPFHw

Nature just published our article on Bee-Nav, a strikingly efficient robot navigation strategy inspired by honeybee learning flights 🐝->🚁. The strategy enables even small drones to travel for hundreds of meters and then successfully return home, while using only a tiny 42-kB neural network 🧠. A link to the open access article is included in the comments below. Most autonomously navigating robots need extensive computing 💻 to construct highly detailed 3D maps 🗺. In contrast, #honeybees 🐝 have been solving the problem of navigation with tiny brains for already millions of years. In Bee-Nav we leverage biological knowledge about honeybee navigation, while filling in some of the blanks. Specifically, just like honeybees, our robot first performs a short learning flight near its home. During this flight, the robot gathers panoramic images of its environment. It then trains a tiny neural network to form a #view #memory that maps the images to the direction and distance home: 🖼️-->🧠-->🧭,📏. The targets for learning come from the robot’s own, noisy #path #integration: keeping track of where it is by integrating its movement speed and direction over time 🐝 💨🧭. After the learning flight, the robot can immediately fly far away to perform its tasks, while doing path integration. When it decides to return home, it will come back in a straight line to the supposed home position. Upon arrival, there will be an offset to the home location due to path integration drift 🚁<--->🏠. However, as long as the robot ends up in the learned homing area, it can use its neural network to cancel the drift and come home 🎯. We show with simulation experiments that the learned homing area can be very small compared to the total flight area °↔⚪ (~4% without and ~0.25% with a compass 🧭). Moreover, using path integration for learning the view memory is no problem for coming home. However, it does lead to winding paths within the learned homing area 〰️🏡. In robotic experiments, Bee-Nav enabled a small drone to navigate over hundreds of meters in various environments with tiny neural networks. Congratulations to the first author, Dequan Ou, with his first scientific article 🤯👏  and all other co-authors Jesse Hagenaars, Maciek Jankowski, Michiel Firlefyn, Christophe De Wagter (all at TU Delft | Aerospace Engineering), Florian Muijres (from Wageningen University & Research), and Jacqueline Degen (from Carl von Ossietzky University of Oldenburg) – it was an exhilarating ride 😅 I think it is worth emphasizing that much of the work for the article has been done in MSc thesis projects 🎓, while another larger part has been funded by NWO (Dutch Research Council) in the context of my NWO VICI grant on neuromorphic learning for advanced insect-inspired AI (NL-AI²). 

Underwater robots will either have to continue relying on expensive, typically bulky, and often power-hungry sensors such as optical gyroscopes or 3D sonars, or they will exploit novel approaches that, at the extreme, allow long-term navigation even in the lack of perceptual data and more broadly facilitate extremely low-light vision-driven odometry. To that end, we present "DeepVL," a novel method on Dynamics- and Inertial Measurements-based Deep Velocity Learning for Underwater Odometry. The work - led by Mohit Singh - presents a learned model to predict the robot-centric velocity of an underwater robot through dynamics-aware proprioception. The method exploits a recurrent neural network using as inputs inertial cues, motor commands, and battery voltage readings alongside the hidden state of the previous time-step to output robust velocity estimates and their associated uncertainty. An ensemble of networks is utilized to enhance the velocity and uncertainty predictions. Fusing the network's outputs into an Extended Kalman Filter, alongside inertial predictions and barometer updates, the method enables long-term underwater odometry without further exteroception. Furthermore, when integrated into visual-inertial odometry, the method assists in enhanced estimation resilience when dealing with an order of magnitude fewer total features tracked (as few as 1) as compared to conventional visual-inertial systems. Tested onboard an underwater robot deployed both in a laboratory pool and the Trondheim Fjord, the method takes less than 5ms for inference either on the CPU or the GPU of an NVIDIA Orin AGX and demonstrates less than 4% relative position error in novel trajectories during complete visual blackout, and approximately \SI{2}{\percent} relative error when a maximum of 2 visual features from a monocular camera are available. https://lnkd.in/dJfREQwP #robotics #autonomy #underwater #vision #odometry #slam #navigation #ntnu #maritime #rcn

Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states? Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖 Quickly and dynamically moving around and in-between obstacles (motion planning) is a crucial skill for robots to manipulate the world around us. Traditional methods (sampling, optimization or search) can be slow and/or require strong assumptions to deploy in the real world. Instead of solving each new motion planning problem from scratch, we distill knowledge across millions of problems into a generalist neural network policy.  Our Approach: 1) large-scale procedural scene generation 2) multi-modal sequence modeling 3) test-time optimization for safe deployment Data Generation involves: 1) Sampling programmatic assets (shelves, microwaves, cubbys, etc.) 2) Adding in realistic objects from Objaverse 3) Generating data at scale using a motion planner expert (AIT*) - 1M demos! We distill all of this data into a single, generalist policy Neural policies can hallucinate just like ChatGPT - this might not be safe to deploy! Our solution: Using the robot SDF, optimize for paths that have the least intersection of the robot with the scene. This technique improves deployment time success rate by 30-50%! Across 64 real-world motion planning problems, Neural MP drastically outperforms prior work, beating out SOTA sampling-based planners by 23%, trajectory optimizers by 17% and learning-based planners by 79%, achieving an overall success rate of 95.83% Neural MP extends directly to unstructured, in-the-wild scenes! From defrosting meat in the freezer and doing the dishes to tidying the cabinet and drying the plates, Neural MP does it all! Neural MP generalizes gracefully to OOD scenarios as well. The sword in the first video is double the size of any in-hand object in the training set! Meanwhile the model has never seen anything like the bookcase during training time, but it's still able to safely and accurately place books inside it. Since, we train a closed-loop policy, Neural MP can perform dynamic obstacle avoidance as well! First, Jim tries to attack the robot with a sword, but it has excellent dodging skills. Then, he adds obstacles dynamically while the robot moves and it’s still able to safely reach its goal. This work is the culmination of a year-long effort at Carnegie Mellon University with co-lead Jiahui(Jim) Yang as well as Russell Mendonca, Youssef Khaky, Russ Salakhutdinov, and Deepak Pathak The model and hardware deployment code is open-sourced and on Huggingface!  Run Neural MP on your robot today, check out the following: Web: https://lnkd.in/emGhSV8k Paper: https://lnkd.in/eGUmaXKh Code: https://lnkd.in/e6QehB7R News: https://lnkd.in/enFWRvft

Articulated-Body Dynamics Network: Dynamics-Grounded Prior for Robot Learning Arxiv: https://lnkd.in/ePQe8ZuF Project: [Link not provided] 🔁 At a Glance 💡 Goal: Incorporate the dynamics structure of articulated robots into control policies to improve learning efficiency. ⚙️ Approach: - Inertia propagation: Adapted from the Articulated Body Algorithm, propagating inertial quantities. - Learnable parameters: Replace physical quantities with learnable ones. - Graph neural network: Embeds dynamic propagation physics into policy architecture. - Bottom-up message passing: Mimics forward dynamics accumulation. 📈 Impact (Key Results) 🧪 Sample efficiency & generalization: Outperforms baselines across diverse robots & tasks. - Validation on real robots shows robust sim-to-real transfer. 🔄 Robustness to dynamics shifts: Maintains performance with increased mass & different terrains. - Visualizations show learnt link representations capture meaningful physical relationships. 🤖 Model extensions & efficiency: - Compatible with model-based RL & dynamics prediction. - Computationally efficient inference suitable for real-time control. 🔬 Experiments 🧪 Benchmarks: Genesis, SAPIEN, MuJoCo, ManiSkill. 🎯 Tasks: Locomotion, velocity tracking, standing. 🦾 Setup: Sim-to-real on Unitree G1 & Go2, NVIDIA RTX 4090 hardware. 📐 Inputs: Proprioception, velocity commands, foot contacts, images (future work). 🛠 How to Implement 1️⃣ Extract robot kinematic tree. 2️⃣ Encode observations into link features. 3️⃣ Perform dynamics-inspired bottom-up message passing. 4️⃣ Decode actions from link representations. 5️⃣ Train with PPO & orthogonality regularization. 📦 Deployment Benefits ✅ Improved sample efficiency & robustness. ✅ Real-time inference on onboard hardware. ✅ Enhanced generalization to dynamics variations. ✅ Compatible with sim-to-real transfer pipelines. 📣 Takeaway This physics-grounded GNN architecture provides an effective inductive bias for articulated robot control. It captures inertial propagation, boosting learning speed, robustness, and transferability. Advances in physics-informed policies open new horizons for efficient, adaptable robot behaviors. Follow me to know more about AI, ML and Robotics!