Task-Aware Modality Selection
for Multimodal VLA in Robotic Manipulation

The importance of sensory information varies with the task. The human cognitive system handles this through active perception: it modulates the weight of each sensory channel according to task context and goal, suppressing irrelevant signals rather than processing everything uniformly. When turning off an overheated electronic device, a person relies on visual localization while exploring, shifts attention to thermal sensing when assessing risk, and prioritizes tactile feedback while grasping.

Static fusion in current multimodal VLAs has two failure modes. First, redundancy and interference — projecting irrelevant data into high-dimensional feature spaces hinders representation learning and slows convergence. Second, reduced robustness — at certain phases, irrelevant modalities act as noise that dilutes meaningful information. We therefore route sensory streams dynamically instead of fusing them uniformly.

We build on the pretrained $\pi_{0.5}$ policy, which consists of a PaliGemma vision-language backbone and a separate action expert. All RGB, thermal and tactile observations are processed by the shared SigLIP vision encoder, with thermal and tactile inputs converted into three-channel image representations. A Camera Router is inserted between the SigLIP encoder and the PaliGemma transformer to estimate the task relevance of the auxiliary views and gate their features before multimodal fusion.

During task adaptation, the SigLIP encoder and the original parameters of the Gemma component and action expert remain frozen. Only their LoRA adapters ($r=16$, $\alpha=16$), the Camera Router, the learnable gating parameter $\boldsymbol{\gamma}$ and the fusion projection layer are optimized. Training uses AdamW with batch size 8 for 50K steps.

The router takes the wrist camera image and the task prompt as inputs. Attention pooling over the prompt tokens gives the text feature $h_{\mathrm{prompt}}$; mean pooling over the wrist camera tokens gives the visual feature $h_{\mathrm{wrist}}$. The wrist camera serves as a fixed anchor view that is always active during execution, providing stable proximal information for close-range robot-object interaction.

With $M$ auxiliary streams to route (here external RGB, thermal and tactile, so $M=3$), the router input is $\mathbf{x} = [h_{\mathrm{prompt}};\, h_{\mathrm{wrist}}]$ and it predicts a categorical distribution over them:

For each auxiliary stream $m \in \{\mathrm{external},\, \mathrm{thermal},\, \mathrm{tactile}\}$, the scalar weight $w_m$ is applied together with a stream-specific learnable channel-wise scaling $\boldsymbol{\gamma}_m \in \mathbb{R}^{D}$:

where $F_m$ are the original feature tokens of stream $m$ and $F'_m$ the gated feature. The scaling parameter mitigates abrupt changes during gating and stabilizes training. The wrist camera feature bypasses routing, is concatenated along the channel dimension with the gated auxiliary features, and the result is projected back to the embedding dimension. The fused feature $F_{\mathrm{fused}}$ is what the pretrained VLM sees — visual information selectively emphasized by task context.

The two objectives are optimized jointly, so the router learns from both explicit routing supervision and the action prediction objective:

Certain modalities are not needed throughout a task; they matter only at sparse moments involving contact or decision-making. This induces an imbalance in the ground-truth routing labels whose severity varies across tasks. To complement minority-class learning under such imbalance, the router is trained with Focal Loss ($\alpha_{\text{focal}}=0.25$, $\beta=2.0$):

Manually annotating task stages or required sensory data for every frame of a large-scale dataset is a scalability bottleneck. We instead use Qwen3-VL to recognize task phases in the demonstration videos and emit discrete modality-relevance labels that supervise the router, which then predicts continuous routing weights at execution time.

A single frame is ambiguous — a static grasping pose is hard to place between the onset of Transporting and Placing. We therefore use a History-Aware Prompting strategy that pairs the current visual information with past action history. Input videos are segmented into chunks of $T=30$ frames, and $N_s=10$ frames are uniformly sampled per chunk as VLM input. The prompt is composed of:

1. Role — the assigned persona defining the agent's behavioral context.
2. Task Description — the overall objective of the task.
3. Workflow Definition — the sequence of actions during task execution.
4. Gripper Action — the scalar value representing the gripper's opening width.
5. Action History — a summary of the previous three chunks.

Given the sampled frames and this context, the VLM infers the task phase from the robot's motion, gripper state, visible objects and consistency with the predefined workflow, and assigns one categorical routing label from $\{\text{external RGB},\,\text{thermal},\,\text{tactile}\}$ to each 30-frame chunk, propagated to all frames in that chunk.

Against independent human annotations over 150 demonstration episodes, frame-level agreement was 82.4%, with most discrepancies at task-phase boundaries rather than in modality selection. Segmental F1 scores — F1@10: 96.3%, F1@25: 93.8%, F1@50: 79.0% — indicate that the temporal structure and modality choices produced by the VLM closely track human annotation.

50 expert demonstrations were collected per task. Each sample contains State and Action (7-dimensional vectors: six joint angles plus gripper state, and six target joint angles plus gripper command), Image, and a natural-language Task prompt. Images were collected at 15 Hz from wrist and external RGB cameras ($640\times480$), a thermal camera ($256\times192$) and a tactile sensor ($640\times480$).

All three tasks require thermal or tactile information beyond RGB, and in each of them the auxiliary modality is informative only during specific phases — which is what makes them a test of selective routing rather than of fusion capacity. Evaluation trials were evenly divided between two predefined object locations about 30 cm apart, with the object position further perturbed by roughly 5 cm at each reset.

The RGB-only baseline (w/o Multimodal) failed on every task. Uniform-Weight Gating reached an average success rate of 10.00%, whereas the proposed Dynamic Router with wrist-camera and prompt inputs reached 83.33%.

Among router inputs, Wrist Cam + Prompt performs best. External Cam + Prompt is competitive at 73.33%, but 10 points lower — the physical distance and viewpoint of the fixed external view constrain how well it captures fine-grained contact or the precise timing of manipulation, whereas the wrist camera gives an ego-centric close-range view that makes object state changes and task phases easier to distinguish. Wrist Cam + Proprio is substantially worse, and on Task 1 it even falls below the baseline — a negative transfer consistent with shortcut learning, where the router latches onto proprioceptive cues correlated with task phase instead of learning robust visual features.

Binary success rates do not capture how far a policy gets before failing, so we also report Mean Task Progress:

where $k_i$ is the number of consecutively completed subtasks before the first failure in rollout $i$, $N_t$ the total number of subtasks in task $t$, and $M$ the number of rollouts. The baseline ranges from 27.1% to 53.8% MTP and completes the full task in only 3 of 30 rollouts. The proposed method ranges from 91.4% to 97.5%.

The failures that remain for the proposed method are mostly low-level manipulation errors — objects dropped during transport, unsuccessful grasps — rather than visible errors in modality selection.

We proposed a task-aware sensory routing framework for robust multimodal robotic manipulation. The Dynamic Router uses wrist camera observations and task prompts to estimate modality importance in real time and attenuate task-irrelevant sensory inputs, and a VLM-based auto-labeling pipeline supplies router supervision without manual frame-level annotation or additional data collection. The resulting VLA model performs comparably to its human-labeled counterpart, and explicit sensory routing improves task success in scenarios requiring fine-grained reasoning over visual, tactile and thermal cues.

The current mechanism selects the single most informative auxiliary modality. Extending it to allow concurrent activation of complementary sensors — using stereo streams for depth estimation, for instance — is a promising direction for broader multimodal environments, and decoupled training with pre-trained routers may improve scalability. The evaluation is also limited to a single VLA backbone; assessing the routing mechanism across architectures and pretraining strategies remains important for establishing its general applicability.