The physical AI stack

The layers of a physical AI system

A physical AI system turns real-world observations into useful actions. Four layers work in sequence: perception captures the environment, models decide, actuation executes, and compute runs the whole loop under latency constraints. A fifth dimension, data, governs the quality of all the others: without demonstrations or simulation, no model learns anything.

Unlike a classic software stack, these layers are tightly coupled. Sensor choices determine what the model can learn, actuator control rates impose latency budgets on compute, and training data must match the robot's embodiment. You cannot evaluate any single brick in isolation: performance comes from the coherence of the whole chain.

Sensors and perception

Vision dominates. Cheap RGB cameras provide most of the signal, often complemented by depth cameras or LiDAR for scene geometry. Touch is progressing fast: high-resolution tactile sensors such as GelSight are becoming essential for fine manipulation, where vision alone cannot see what happens under the fingers. Proprioception (joint encoders, IMUs, torque sensors) tells the robot about the state of its own body.

The 2026 trend is sensor frugality. Mistral AI's Robostral Navigate navigates with a single RGB camera and no depth sensor, reaching 76.6% success on the R2R-CE benchmark in unseen environments, ahead of multi-sensor systems. Fewer sensors mean lower cost, less calibration and more robustness in deployment.

Models: VLAs, world models and policies

Three model families coexist. Vision-language-action models (VLAs) translate pixels and instructions directly into actions: NVIDIA's Isaac GR00T N1.6, Google DeepMind's Gemini Robotics 1.5 or Physical Intelligence's π0.5. World models learn the dynamics of the environment and are used to generate synthetic data, evaluate behaviours or plan (NVIDIA Cosmos, Genie 3). Finally, reinforcement-learned policies, usually trained in simulation, handle locomotion and reflex-level motions.

These families are converging: GR00T N1.6 embeds the Cosmos Reason model as its slow-thinking brain, and recent architectures stack a planning module on top of a fast control module. We cover these models in depth in the dedicated VLA and world model pillars.

Actuation: motors, hands, locomotion

Electric actuators have displaced hydraulics on nearly every recent platform: easier to maintain, quieter, and simpler to control precisely. Quasi-direct-drive architectures, popularised by legged robotics, provide the torque transparency needed to handle unplanned contact.

The hand remains the hardest frontier. Packing some twenty degrees of freedom together with tactile sensing, useful force and industrial durability involves mechanical trade-offs no one has solved yet at a reasonable cost. Many applications stick to two-finger grippers, which are perfectly adequate for logistics.

Locomotion, by contrast, is largely solved: policies trained with reinforcement learning in simulation have made walking robust, including on rough terrain, and entry-level platform prices are falling fast, driven in particular by Chinese manufacturers such as Unitree.

Onboard or cloud compute

A robot's control loop runs between 50 Hz and 1 kHz: routing it through the cloud is simply not an option. Control model inference therefore runs onboard, typically on modules from the NVIDIA Jetson family. Figure, for instance, runs its Helix model entirely onboard, on low-power GPUs.

Slow reasoning (interpreting an instruction, planning a task sequence) tolerates more latency and can run remotely. But three forces push towards fully onboard operation: safety (the robot must stay safe if the network drops), confidentiality of shop-floor data, and certification. The dominant compromise in 2026: train in the cloud or datacenter, run inference onboard, and stream operational data back asynchronously to retrain the models.

The data bottleneck

There is no internet of actions. Unlike text or images, robot trajectories do not pre-exist anywhere: they have to be produced. Three sources share the load. Teleoperation yields high-quality demonstrations but is expensive and scales poorly. Simulation produces near-unlimited volumes, at the price of a reality gap that must then be closed. Egocentric human video is emerging as the third path: work published in 2026 shows that 30 minutes of human video per task can outperform an equal collection time of teleoperation by 41%.

The dominant strategy combines all three: large-scale pretraining on human video and synthetic data, then targeted teleoperation fine-tuning on critical tasks. This bottleneck, more than model architectures, is what sets the pace of progress in the field today. Our simulators comparator details licences, physics engines and ROS support for these tools.

Keep reading

• Go back to the fundamentals of simulation and data: our world models and sim2real pillar.
• Dig into the models layer with our guide to VLA models.
• Compare onboard compute chips and platforms in our chips comparator.