Plug-and-Play VLA Solutions for Robotics Integration

🚀𝐖𝐞 𝐚𝐫𝐞 𝐢𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐢𝐧𝐠 𝐒𝐦𝐨𝐥𝐕𝐋𝐀-𝟒𝟓𝟎𝐌, 𝐚𝐧 𝐨𝐩𝐞𝐧-𝐬𝐨𝐮𝐫𝐜𝐞 𝐕𝐢𝐬𝐢𝐨𝐧-𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞-𝐀𝐜𝐭𝐢𝐨𝐧 𝐦𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐫𝐨𝐛𝐨𝐭𝐢𝐜𝐬! SmolVLA achieves best-in-class performance and inference speed, and the best part? It’s trained entirely on open-source datasets from the 🤖 LeRobot project hosted on the Hugging Face Hub. 🔍 Why is SmolVLA so good? Turns out that pretraining on a large, diverse and noisy collection of real-world community robotics data leads to better generalization and control. We saw a 26% boost in task success rate simply from adding community dataset pretraining! ⚡ Why is SmolVLA so fast? 1. We halved the size of SmolVLM and extract intermediate representations 2. Introduced interleaved cross- and self-attention layers in the action expert 3. Enabled asynchronous inference so the robot acts and reacts simultaneously 💡 Unlike most academic datasets, these community-contributed datasets are naturally diverse: ✅ Multiple robots, camera angles, and manipulation tasks ✅ Real-world messiness and complexity ✅ Crowd-sourced and community-cleaned using Qwen2.5-VL for high-quality task descriptions 🌍 SmolVLA is a step toward making robotics research more affordable, reproducible, and collaborative. 📖 Want to dive deeper? Check out our blog post & start using it today: https://lnkd.in/e3Gmy8gT Huge thanks to the team who made this possible: @Mustafa Shukor Francesco Capuano Remi Cadene, and the entire Lerobot team, amazing HF team Andrés Marafioti Merve Noyan Aritra Roy Gosthipaty Pedro Cuenca Loubna Ben Allal, Thomas Wolf  and to the amazing contributors to the LeRobot community: Ville Kuosmanen, Alexandre Chapin, Marina Barannikov, and more!

💥 A 450M model just beat bigger VLAs on real robot tasks, and… 100% open source! [📍 bookmark for later] Came across SmolVLA, a new vision-language-action model for robotics that’s compact, fast, and trained entirely on open community datasets from LeRobot via Hugging Face. What stood out to me is how it matches or outperforms much larger models like ACT using noisy, real-world community data instead of giant private datasets. Why it’s worth a look ✅ 26% performance boost from pretraining on open-source data ✅ Runs on consumer hardware, even a MacBook ✅ 30% faster responses with async inference and smart architecture tweaks ✅ Strong results across Meta-World, LIBERO, SO100, and SO101 ✅ Fully open source: weights, code, training pipeline, eval stack They also introduced smart efficiency tricks like using fewer visual tokens, pulling outputs from mid-layer, and separating perception from action to make it all run fast. Useful links 📘 Blog: https://lnkd.in/dnZSHdqU 📦 Model: https://lnkd.in/dUZMzTDN 📄 Paper: arxiv.org/abs/2506.01844 SmolVLA is a strong case for what can happen when the robotics community shares data and builds in the open. Definitely worth keeping an eye on.

Excited to share VLA-0: Building State-of-the-Art VLAs with Zero Modification! 🚀 What's the right architecture for Vision-Language-Action (VLA) models? The field has explored custom action heads (π₀), discrete tokens (OpenVLA), and novel designs (OpenVLA-OFT). We tried the simplest approach: a VLM with ZERO modifications. Just ask it to predict actions as text. The results surprised us: 📊 LIBERO Benchmark: • 94.7% success rate - #1 among all methods without large-scale pretraining • Outperforms π₀.5-KI, OpenVLA-OFT, SmolVLA (all without pretraining) • Even beats models WITH large-scale pretraining: π₀ (94.2%), GR00T-N1 (93.9%), MolmoAct (86.8%) 🤖 Real-World Results: • Tested on SO-100 robotic arm • +12.5 percentage points over SmolVLA • Despite SmolVLA's large-scale SO-100 pretraining advantage The key insight? With the right training recipe (action decoding, ensemble prediction, masked augmentation), you don't need to modify the VLM at all. Paper, code, and full results in comments below! ⬇️ #MachineLearning #Robotics #AI #VisionLanguageAction #Research #NVIDIA #DeepLearning #ComputerVision

Deploying VLAs on edge devices? 🤔 OpenVLA needs 7B parameters and an RTX 4090. EdgeVLA needs an A100. Most VLA research optimizes for benchmark accuracy on datacenter GPUs — then calls it "efficient." Researchers from 𝐂𝐥𝐚𝐫𝐤 𝐀𝐭𝐥𝐚𝐧𝐭𝐚 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 and 𝐒𝐢𝐞𝐦𝐞𝐧𝐬 asked a different question: what's the smallest model that can do closed-loop robotic control on production edge hardware.Their answer? 𝐋𝐢𝐭𝐞𝐕𝐋𝐀-𝐄𝐝𝐠𝐞 — 256M parameters. Fully on-device. On a Jetson AGX Orin. 𝐓𝐡𝐞 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞: ➡️ 𝐒𝐦𝐨𝐥𝐕𝐋𝐌-𝟐𝟓𝟔𝐌 backbone — LoRA fine-tuned on image-to-action data in FP32, then compressed to 4-bit GGUF post-training ➡️ 𝐥𝐥𝐚𝐦𝐚.𝐜𝐩𝐩 CUDA backend — all 42 transformer layers offloaded to the Orin GPU ➡️ 𝐑𝐎𝐒 𝟐 integration — modular perception, reasoning, and actuation with deterministic safety overrides 𝐑𝐞𝐬𝐮𝐥𝐭𝐬: ➡️ 150.5ms mean latency — 6.6 Hz closed-loop control (~220% improvement over previous LiteVLA baselines) ➡️ σ = 0.125ms jitter — deterministic enough for stable ROS 2 control frequencies ➡️ Runs on a 40W edge module — no cloud, no API calls, no desktop GPU At 150ms the robot corrects its trajectory mid-motion — reacting to changes within a single human attention window. That's the threshold between "predict-then-execute" and real-time visual servoing. The paper offers a practical deployment path for VLAs on hardware you can actually put on a robot. https://lnkd.in/dRsZu2Af

100% task success rate on a VLA model deployed in C++ on edge hardware. No Python. No cloud. No model-specific glue code. Deploying embodied AI models on real robots has been a mess. Every vision-language-action model ships with its own Python stack, its own backend assumptions, its own I/O wiring. Move to a different robot or a different edge device and you're rewriting integration code from scratch. Existing inference runtimes weren't built for this - they assume request-response serving, not the tight closed-loop control that a physical robot demands. The team analyzed representative VLA and world-action models, found a shared execution path, and organized it into five modular layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters. One abstraction layer handles heterogeneous hardware, different robots, and simulators. Embodied.cpp is a portable C++ runtime which is purpose-built for embodied constraints - multi-rate execution so vision and action run at different frequencies, latency-first batch-1 inference instead of throughput-optimized batching, and extensible I/O beyond fixed token formats. On HY-VLA, closed-loop control hit a 100.0% task success rate. On pi0.5, 91.0%. A preliminary world-action model benchmark cut block memory from 312.2 MiB down to 88.1 MiB - a 3.5x reduction. The bottleneck in embodied AI isn't model quality. It's getting models to actually run on the robot. That bottleneck just got a proper runtime. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

Google DeepMind has launched an on-device version of its Gemini Robotics AI, allowing robots to operate without an internet connection. This smaller, more efficient vision-language-action (VLA) model retains many of the dexterous capabilities of the original, enabling robots to generalize tasks and respond to commands with minimal training (50–100 demonstrations). Though not as powerful as the cloud-enabled flagship, the offline model is surprisingly capable and ideal for low-connectivity or high-security environments. It has been adapted to various robots, including Apptronik’s Apollo humanoid and Franka’s bi-arm robot. Google is also releasing an SDK to let developers evaluate and fine-tune the model — a first for its VLA tech. Initially, access is limited to trusted testers. Read more: https://lnkd.in/dFs5Sw5P

Ant Group Releases LingBot-VLA, A Vision Language Action Foundation Model For Real World Robot Manipulation Ant Group releases LingBot VLA, a vision language action foundation model trained on about 20,000 hours of real world dual arm teleoperation data from 9 robot embodiments, designed for strong cross morphology and cross task generalization. The model combines a Qwen2.5 VL backbone, a Flow Matching based action expert, and depth aware spatial perception via LingBot Depth distillation, so robots can reason more accurately about 3D structure. On the GM 100 benchmark across 3 platforms LingBot VLA with depth reaches about 17.30 percent average Success Rate and 35.41 percent Progress Score, outperforming π0.5, GR00T N1.6, and WALL OSS under a shared protocol, while simulation tests show similar gains under domain randomization. The open source toolkit provides an efficient post training stack that reaches about 261 samples per second per GPU on 8 GPUs, delivering 1.5 to 2.8 times higher throughput than existing open VLA frameworks..... Full analysis: https://lnkd.in/gmj8MwJt Paper: https://lnkd.in/gXmzqkXy Model weight: https://lnkd.in/gw6rYbQp Repo: https://lnkd.in/gq9YdsT6 Project: https://lnkd.in/gMztVbM5 Ant Group #robotics #robots #opensource

Robotics models are increasingly bulky and difficult to run directly on robots. With Remi Cadene and the team at LeRobot and Hugging Face, we’re changing that. Today, we're introducing SmolVLA, a sub-500M VLA designed for efficient training and inference. We present to the robotics and open-source community three main contributions: 1️⃣ SmolVLA is *small*. We use: - A small pretrained VLMs - Fewer visual tokens - A layer-skipping mechanism in the VLM - Interleaving self-/cross-attention layers in the action expert ➡️ This allows us to train on a single consumer-grade GPU, and run SmolVLA efficiently, even on CPUs 💥 2️⃣ We ditch proprietary megadatasets, using community-contributed datasets - SmolVLA is trained end-to-end on <30k episodes - All datasets are publicly available, community-contributed and available on Hugging Face Hub - Training this SmolVLA is a community effort, resulting on training on ~10x less data than SOTA VLAs ➡️ Across real-world and simulations, we match much larger models trained on 10x more data 💥 3️⃣ SmolVLA is deployed asynchronously, for greater adaptability - Robots shouldn’t lag (let that sink in) - With SmolVLA, we present an asynchronous inference stack decoupling action execution from planning. ➡️ This results in fast, smooth, and resource-efficient control in the real world (2x throughput!) With this, we hope to push open-source robotics research. We're releasing everything, from data, to training and inference recipes---the Hugging Face way 🤗 SmolVLA has been proudly brought to life by the SmolVLA team, Mustafa Shukor, Dana Aubakirova and yours truly 😊, standing on the shoulders of the entire LeRobot team, and with the guidance of Remi Cadene and Thomas Wolf. Interested? Check out our technical report (🔗 link in the first comment!) Thank you to the whole team behind this project & amazing co-authors Pepijn Kooijmans Michel Aractingi Adil Zouitine Martino Russi Caroline Pascal Andrés Marafioti Thomas Wolf

Gemini VLAs can now run on-device! Google DeepMind just released 'Gemini Robotics On-Device', an optimized version of its VLA that can run locally on a robot. This builds on their March release of Gemini VLA, which demonstrated how a single VLA can control multiple embodiments (from ALOHA arms to a humanoid) using vision, language, and motion. Now, the lightweight version of the same model can run locally and in real-time. The demonstrations were performed with a Raspberry Pi 4 Coral Edge TPU (they still make those?! 😅). This is much needed. In the real world, network connections are flaky, and we need model inference on a device. It's natural to sacrifice performance when we optimize for local inference, but Gemini On-Device seems to be solid! On their 9-task generalization benchmark: - Gemini On-Device beats every prior on-device model - Performed nearly as well as cloud-inferred Gemini, despite drastically lower compute DeepMind also released the Safari SDK, which enables easy prompting, control, and evaluation of VLA models on both real and simulated hardware. Two things I found particularly useful were the unified API across embodiments and the out-of-the-box benchmarking/logging, which are usually annoying to build from scratch. Finally, they released a MuJoCo simulator designed for the Gemini family of VLAs. It's awesome to see a research lab consider the practical (and dare I say, production) use cases that robotics engineers have.