Multimodality & Action Interfaces¶
What this page solves
How models see images and hear sound, and how they go from describing to acting: how vision-language models (VLMs) connect to robot actions (VLAs). This is where foundation models meet embodied intelligence. Introduction required, VLA adaptation elective.
How far to go¶
| Level | Learn | Done when |
|---|---|---|
| Orientation | Image–text alignment, VLM, VLA inputs and outputs | You can state the difference between a VLM and a VLA |
| Foundation | Run a VLM, probe spatial and counting ability | F3 done with at least three error types recorded |
| Advanced | Action representations, robot data adaptation, closed-loop evaluation | One route of embodied E4 done |
Core knowledge¶
Contrastive image–text alignment (CLIP): an image encoder and a text encoder map matching pairs into one space, pulling matches together and pushing mismatches apart. Result: zero-shot classification and retrieval, and the vision encoder of many VLMs.
VLM structure: a vision encoder (ViT) turns the image into patch vectors, a projection maps them into the language model's embedding space, and they enter the LLM as "image tokens" alongside text.
VLM limits: good at description and common-sense QA, often wrong on exact counting, fine spatial relations, metric quantities (distance, size) and 3D reasoning. Verify spatial outputs separately before using them on a robot.
From VLM to VLA: add an action output and train on robot data (image + instruction + action). Actions can be discretised into tokens generated like words (RT-2, OpenVLA) or produced by a continuous head (diffusion or flow matching, e.g. π0) as smooth action chunks.
Action representations: joint angles or end-effector pose, absolute or delta, single step or chunk. The choice decides cross-robot data reuse and control smoothness.
Cross-embodiment and data: robots differ in joints, camera placement and control rate. Datasets like Open X-Embodiment train across bodies, but adapting to your robot still needs its own data.
Two ways to connect a robot: an LLM chooses skills and parameters for existing controllers (interpretable, easy to verify), or a VLA outputs actions directly (more generalisation potential, harder to debug).
Debate · discrete action tokens vs. continuous action generation¶
Discrete: treat actions as vocabulary tokens and reuse language-model training and inference unchanged. Continuous: discretisation loses precision; high-frequency fine manipulation needs smooth continuous chunks. High-performing VLAs are moving to continuous heads; discrete is the simplest to implement.
Curated resources¶
-
VLM practice · Image-text-to-text
Hugging Face's official image–text QA task guide, used in F3.
-
VLA practice · SmolVLA
A small VLA in the LeRobot ecosystem, from data to fine-tuning and evaluation.
Follow-along experiments¶
Self-check¶
How does CLIP differ from a VLM?
CLIP only maps images and text into one space for matching and cannot generate text; a VLM feeds image features into a language model and answers open questions about the image.
Why can't a VLM's "the cup is left of the plate" drive a grasp directly?
The description is qualitative, may be wrong and gives no metric pose; grasping needs millimetre-level 3D position and orientation from, or verified by, geometric perception.
Why do VLAs often predict action chunks instead of single steps?
Large-model inference is slow; single-step prediction cannot keep up with control rates, while a chunk covers several control cycles per inference and is more coherent.
What does using an LLM as a skill selector buy you?
Actions run on verified controllers, so they stay safe and controllable; the language model only interprets instructions and composes skills, and failures are easy to locate.