Autonomous Vehicle Engineering

One of the key challenges of autonomous driving is scalably handling the complexity of driving scenarios, where traffic rules, city environments, and vehicles/pedestrians can interact in a myriad of possible ways. It’s not tractable to create hand-crafted rules that handle every case, so instead we rely on the power of “next frame prediction” in a compact world representation. Here the world representation is semantic segmentation, which captures the essence of what’s happening around a vehicle, and can be stably computed in real-time using Helm.ai’s production grade perception stack. One example of a set of complex scenarios is an intersection with traffic lights, which presents a large number of possibilities that an autonomous vehicle must navigate safely. To tackle this challenge, we added traffic light segmentation and traffic light state to our world model representation, and trained a foundation model to predict what might happen next based on an input sequence of observed segmentations. Our foundation model learned in a fully unsupervised way from real driving data the relationship between traffic light state and what the vehicles/agents on the road should do in various contexts. The result is an ability to forecast a wide variety of scenarios of interaction between traffic lights, intersection geometry, vehicles, and pedestrians that are consistent with potential real world scenarios, including predicting the paths of the ego vehicle and the other agents. In our latest demo, our intent and path prediction models predict 9 seconds into the future using 3 seconds of observed driving data, at 5 frames per second. This prediction capability includes learned human-like driving behaviors, such as intersection navigation, interaction with green and red lights, yielding to oncoming traffic before turning, and keeping a safe distance to other vehicles. Our foundation models are able to predict these future behaviors and plan safe paths by scalable learning from real driving data, without any hand crafted rules nor traditional simulators. Stay tuned for upcoming updates as we continue to expand our unified approach to ADAS through L4 autonomous driving by enriching the world model representation and scaling up our predictive DNNs. #helmai #generativeai #selfdrivingcars #artificialintelligence #ai #autonomousdriving #adas #computervision 

When AI meets the physical world, the rules of engineering change completely. In autonomous driving, tech complexity is never isolated—an architecture tweak at the top of the software stack can completely blow up your hardware costs and manufacturing timelines at the bottom. The AV Product stack is built like a pyramid: from the physical Vehicle Platform and Sensor hardware up through Middleware, Data Pipelines, AI Models, and finally, the consumer Service layer. Today, we are seeing a fascinating, bifurcated evolution across these tiers. - On the software side, we are transitioning from traditional modular architectures (AV 1.0) to End-to-End networks (AV 2.0) and cognitive Reasoning Stacks (AV 3.0). - On the hardware side, the industry remains split between Camera-Only minimalism and Lidar-Dominant maximalism. The core insight? There is a strict Law of Conservation of Complexity. If you opt for a low-cost, Camera-Only hardware suite to accelerate your build time, you don't eliminate complexity—you merely push it up into your Data Pipelines and AI Models, demanding massive compute and world-class developer talent to infer what physics could have provided. Conversely, chasing an AV 3.0 reasoning stack dramatically enhances edge-case handling, but hits you with an exponential increase in training costs and validation iteration times. True success in Physical AI requires rigorous, cross-layer trade-off analysis. You cannot optimize the model without optimizing the silicon, and you cannot optimize the sensor suite without understanding the data pipeline burden it creates. Winning the autonomous race requires mastering this delicate balance between digital intelligence and physical cost. What's your take? Is software-defined hardware the winning playbook, or is heavy sensor redundancy non-negotiable for scale? In my new Substack series "Bits vs. Atoms: When AI Meets the Physical World at Scale," I dive deep into managing the technical complexity and hidden trade-offs of scaling autonomous vehicles. Throughout this series, I break down the engineering friction points across the six layers of the AI Product Pyramid. While these insights are framed around autonomous driving, the core principles apply universally to robotics, automation, and any physical AI product navigating the real world. Read the full post for free and welcome to subscribe for future readings: https://lnkd.in/g59yEcrG #PhysicalAI #AutonomousVehicles #ArtificialIntelligence #DeepLearning #Robotics #TechStrategy #AutomotiveTech

🚗 What really controls a modern car?  Not the steering wheel. Not the brake pedal. 👉 Software. Signals. Decisions. Most people still think vehicles are mechanical systems. But in reality, every critical action is now electronic, networked, and software-driven. From throttle to braking to steering — we’ve quietly transitioned into a world of Drive-by-Wire systems .  I created a deep-dive engineering presentation on: ⚙️ Throttle, Brake, Steer, Shift & Suspension-by-Wire 🧠 Sensor → ECU → Actuator control loops with real-time feedback 🏗️ Distributed vs Domain vs Zonal E/E architectures 🛡️ Functional Safety (ISO 26262, ASIL, redundancy, fail-operational design) 🔐 Cybersecurity (Secure Boot, SecOC, IDS, UNECE R155/R156) 🌐 CAN FD, FlexRay, Automotive Ethernet in real-time control 🤖 AI, OTA, and the transition to autonomous vehicles Drive-by-Wire is not just an innovation. 👉 It is the foundation of software-defined vehicles . No Drive-by-Wire → No Autonomy. No Autonomy → No future mobility.   The real shift is this: We are no longer designing *mechanical systems with electronics*. We are designing software systems that move machines. #Automotive #EmbeddedSystems #DriveByWire #AUTOSAR #FunctionalSafety #Cybersecurity #SoftwareDefinedVehicle #AutonomousDriving #ECU #Engineering #AI

🚗 Excited to Share: Automotive Engineering Smart Handbook 2026 I’m glad to share a comprehensive reference guide I’ve been working on for the automotive engineering community. This handbook is designed as a practical, system-level companion for engineers, architects, and safety professionals working across automotive electronics and software. 🔍 What’s Inside? It brings together key domains that modern automotive engineers deal with daily: ECU architecture and in-vehicle networks (CAN, Ethernet, TSN) AUTOSAR Classic & Adaptive (architecture, BSW, RTE, methodology) ADAS and autonomous driving fundamentals Vehicle dynamics and control systems ISO 26262 functional safety (ASIL, HARA, safety architecture) EV powertrain, BMS, inverter control Cybersecurity (ISO 21434, TARA) Diagnostics (UDS, DoIP) and communication stacks ⚖️ NDA & Data Integrity For industry peers, an important note: This handbook is fully based on publicly available standards (ISO, AUTOSAR, SAE, IEEE) It does NOT include any OEM-specific, customer, or project-confidential data No proprietary code, calibration, or internal architecture has been used It is a personal, educational reference, not representing any employer or client 💡 Why This Handbook? In automotive, knowledge is often scattered across domains and projects. This effort is to: ✔ Connect electronics, software, safety, and vehicle dynamics ✔ Help engineers move from component-level to system-level thinking ✔ Provide a structured reference for both learning and quick revision 🙌 Who Is It For? Automotive software engineers (AUTOSAR, Embedded, ADAS) System architects and technical leaders Functional safety engineers Students and professionals entering the automotive domain 🔁 Open to Feedback This is Version 1.0. I’d love to hear your thoughts, suggestions, or areas where we can go deeper. #AutomotiveEngineering #AUTOSAR #EmbeddedSystems #ADAS #FunctionalSafety #EV #BMS #CyberSecurity #Learning #EngineeringLeadership

On the heels of the Alpamayo announcement — NVIDIA’s fully open ecosystem for accelerating the development of reasoning-based autonomous vehicles — I’m excited to share our latest advances in researching reasoning-based Physical AI models. Starting with Latent‑CoT‑Drive (LCDrive), a novel approach that learns to reason in a *latent* action-aligned space for end-to-end driving decision-making. Traditional vision-language-action models rely on natural language for chain-of-thought reasoning — but is language the best medium for encoding driving decisions? In our paper, we explore this question and introduce a latent representation that integrates both action proposals and predictions of future outcomes, enabling richer reasoning and improved performance. 🔍 Key Contributions - Latent reasoning for driving: LCDrive rethinks reasoning in vision–language–action (VLA) models using latent chain-of-thought tokens aligned with driving actions and a latent world model. - Effective training framework: Combines latent CoT cold-start, world model training, and closed-loop reinforcement learning, tailored for latent reasoning models. - Empirical gains: Shows faster inference and higher driving quality compared to non-reasoning and text-reasoning baselines. This work shows that latent reasoning provides a compelling representation for reasoning-based VLA models. 📄 Full paper here: https://lnkd.in/ejsZnwkw #AutonomousVehicles #AutonomousDriving #PhysicalAI #ReasoningAI #Alpamayo NVIDIA AI NVIDIA DRIVE

163 kW. 48 kWh. 1,412 kg. The biggest inefficiency in electric vehicle design is not chemistry. It is human psychology. These are the EPA-confirmed specifications of the Tesla Cybercab, published yesterday.   Whether Tesla's FSD delivers on its promise is a different conversation. What these numbers reveal is more fundamental.   When a vehicle operates autonomously, the requirement logic changes entirely. Range anxiety is no longer a factor. The average passenger car driver in Europe covers around 30 km per day. Yet the industry has spent a decade engineering 500+ km ranges because customers demand it, not because they use it, except in a minority of cases.   Autonomous operation allows engineers to dimension by data rather than by perception. Real utilization, real duty cycles and real economics count. The result you see in the numbers with a battery of 48 kWh instead of 75-100 kWh, 163 kW power instead of 300+. A kerb weight that makes the Cybercab the lightest electric vehicle currently on the US market. Less material, less energy in production, lower cost per unit and per kilometre operated.   It is a systems engineering argument that only becomes possible when human perception is removed from the requirements specification.   That is precisely why autonomous driving and electrification are not parallel trends. They reinforce each other structurally. One enables you to right-size the other.   The race is open. The engineering case is not. https://lnkd.in/e7D5fBWf

🚗 AUTOSAR vs SDV vs AIDV: Understanding the Evolution of Automotive Software Architecture The automotive industry is undergoing one of the biggest transformations in its history. From traditional ECU-based systems to software-centric vehicles and now AI-powered mobility, understanding the relationship between AUTOSAR, Software-Defined Vehicles (SDV), and AI-Defined Vehicles (AIDV) is becoming essential for automotive engineers. 🔹 AUTOSAR: The Foundation AUTOSAR (AUTomotive Open System ARchitecture) established a standardized software architecture that enabled: ✅ Reusability of software components ✅ Hardware abstraction ✅ Interoperability across suppliers and OEMs ✅ Functional safety and reliability For years, AUTOSAR has been the backbone of automotive software development, helping manage increasingly complex vehicle electronics. 🔹 SDV: The Software Revolution The Software-Defined Vehicle shifts the focus from hardware-centric development to software-centric innovation. Key characteristics: ✔ Centralized and zonal architectures ✔ Decoupling software from hardware ✔ Over-the-Air (OTA) updates ✔ Service-oriented architectures ✔ Faster feature deployment and continuous improvement In many cases, AUTOSAR remains a critical enabler within SDV architectures, particularly through Adaptive AUTOSAR and middleware solutions. 🔹 AIDV: The Next Evolution AI-Defined Vehicles take SDV one step further by placing Artificial Intelligence at the core of vehicle functionality. Key characteristics: 🤖 AI-driven perception and decision-making 🤖 Continuous learning from vehicle data 🤖 Personalized user experiences 🤖 Predictive maintenance and intelligent diagnostics 🤖 Advanced autonomous driving capabilities In AIDV, software is no longer just defining vehicle functions—AI becomes the primary driver of how the vehicle perceives, learns, adapts, and evolves. 📈 The Evolution Path AUTOSAR ➜ SDV ➜ AIDV 🔹 AUTOSAR standardizes software architecture. 🔹 SDV makes vehicles software-centric and continuously updatable. 🔹 AIDV transforms vehicles into intelligent, adaptive systems powered by AI. 💡 Key Takeaway AUTOSAR, SDV, and AIDV should not be viewed as competing concepts. Instead, they represent different stages of automotive software evolution: ✅ AUTOSAR provides the foundation ✅ SDV provides the flexibility ✅ AIDV provides the intelligence The future vehicle will likely combine all three: a standardized architecture, software-defined functionality, and AI-driven intelligence. What do you think will be the biggest challenge in the transition from SDV to AIDV: safety, validation, compute power, data management, or regulations? #Automotive #AUTOSAR #SDV #SoftwareDefinedVehicle #AIDV #AI #ArtificialIntelligence #AutomotiveEngineering #ADAS #AutonomousDriving #EmbeddedSystems #VehicleArchitecture #DigitalTransformation #FutureMobility #AutomotiveSoftware #Testing #QualityAssurance #OTAUpdates #Cybersecurity #Innovation

Imagine a vehicle that can "see" the world around it with precision and detail surpassing human capability. Autonomous vehicles achieve this feat thanks to an orchestra of sophisticated sensors, including cameras, LiDAR, radar, and ultrasonic sensors. These sensors capture a torrent of data, from images and videos to light pulses and radio waves, painting a detailed picture of the environment. But data alone isn't enough. This is where the magic of deep learning and computer vision comes in. The first step towards autonomous navigation is environment perception. Autonomous vehicles rely on a multitude of sensors to capture information about the world around them, including: 1) Cameras: Capture images and videos of the surroundings, essential for detecting objects like vehicles, pedestrians, traffic signs, and lane markings. 2) LiDAR (Light Detection and Ranging): Emits laser light pulses and measures the return time to create precise 3D maps of the environment, including the distance and depth of objects. 3) Radar: Detects moving objects using radio waves, proving useful in low-visibility conditions like rain or fog. 4) Ultrasonic Sensors: Measure the distance to nearby objects, primarily used for low-speed collision avoidance. 5) Planning and Decision-Making: The Brain of Autonomous Vehicles. Once the environment has been perceived, autonomous vehicles need to make intelligent decisions to navigate safely and efficiently. This is where route planning and decision-making come into play. Machine Learning algorithms play a critical role in these tasks: A) Route Planning: Determine the optimal route to reach the destination, considering factors like traffic, traffic regulations, and road conditions. B) Predicting the Behavior of Other Users: Anticipate the actions of other vehicles, pedestrians, and cyclists to avoid collisions and dangerous maneuvers. C) Real-Time Decision Making: Adapt the driving plan to unexpected events, such as sudden braking, lane changes, or pedestrians crossing the street. D) Continuous Learning: Improving with Experience E) Adapt to New Situations: Learn from experience and adjust their behavior based on different driving conditions, climates, and environments. F) Update Maps and Models: Incorporate new information about the surroundings, such as changes in road infrastructure or new traffic signs. G) Personalize the Driving Experience: Tailor the driving style to user preferences, prioritizing safety, efficiency, or comfort. Autonomous vehicles, powered by Machine Learning, have the potential to revolutionize the way we move. They offer the promise of safer, more efficient, and sustainable mobility, reducing traffic accidents, congestion, and emissions. However, the technological and regulatory challenges are significant. Continued research and development, along with an ethical and responsible approach, are essential to ensure autonomous vehicles become a reality that benefits all of society.

Autonomous vehicle technology represents a paradigm shift in societal mobility, offering the potential to eliminate human error and provide critical independence to the elderly and mobility-impaired populations. Despite significant progress, the "long-tail" of rare, high-risk edge cases remains a primary barrier to safe, full-scale deployment. In a new paper (https://lnkd.in/eKZp6tpa) with my colleagues (Dr. Edward Griffor, Munawar Hasan, Mahima Arora, Honglan Jin, Pavel Piliptchak, Thoshitha Gamage) we approach this problem by evaluating perception performance using predictive sensitivity quantification based on an ensemble of models, capturing model disagreement and inference variability across multiple models, under adverse driving scenarios in both simulated environments and real-world conditions. We propose a notional architecture for assessing perception performance that comprehends multiple input sources and extensible AI architecture that provides detection and classification as outputs along with predictive sensitivity and post-processing. Diminished lighting conditions, e.g., resulting from the presence of fog and low sun altitude, are seen to have the greatest impact on the performance of the perception models. Additionally, adversarial road conditions such as occlusions of roadway objects increase perception sensitivity and model performance drops when faced with a combination of adversarial road conditions and inclement weather conditions. Also, it is demonstrated that the greater the distance to a roadway object, the greater the impact on perception performance, hence diminished perception robustness. This work is only the first step in a bigger effort at the National Cybersecurity Center of Excellence (https://lnkd.in/eYjDs5tM) where we continue the work by developing a public dataset spanning multiple modalities (camera, LiDAR, radar) and a testbed with difficult-to-handle and adversarial road/traffic conditions with the goal of improving autonomous vehicles and accelerating their safe deployment. We are open to collaboration, join us at NCCoE.

TOPS & TFLOPS: Spec Sheet Numbers Don’t Drive Autonomous Vehicles In the race toward scalable autonomy, many engineering teams fall into a familiar trap: selecting compute hardware based on advertised TFLOPS or TOPS numbers. As I am working in autonomy, I’ve learned that these metrics while useful for benchmarking are often misleading when it comes to building real-world, production-grade autonomous systems. At a basic level, TFLOPS and TOPS measure raw computational throughput. High TFLOPS or TOPS ratings make for impressive marketing, but raw silicon performance doesn’t equate to operational performance in autonomous vehicles. In autonomy, the real measure of compute hardware isn’t just theoretical peak throughput it’s how consistently, efficiently, and reliably that compute can drive perception and control in real-time, under real-world constraints. In our journey developing autonomous platforms, we’ve evaluated processors that delivered 200+ TOPS, yet underperformed against systems with half the theoretical capability. Why? Because system-level bottlenecks memory bandwidth, thermal throttling, I/O constraints, and poorly optimized inference pipelines negate theoretical compute advantages. In many cases, power-efficient processors optimized for AI workloads outperform higher-rated chips because they’re designed around the actual needs of end-to-end autonomy stacks. At the system level, what matters most is sustained inference throughput, real-time decision latency, and software optimization not peak TFLOPS/TOPS. Choosing the right autonomy hardware requires evaluating your models, understanding your control loop timing, and benchmarking end-to-end latency, not relying on spec-sheet numbers. In autonomy, decisions happen in milliseconds not marketing cycles. My advice to technology leaders: select compute platforms based on real-world performance, not theoretical peaks. It’s not the chip with the highest TOPS that wins it’s the system that moves safely and reliably in the real world. #AutonomousVehicles #AIHardware #EdgeAI #RealWorldPerformance #ScalableAutonomy #AutonomousSystems #LeadershipInTech #SystemDesign #NextGenAutonomy #InnovationStrategy