MulPlanLM: multimodal robotic task planning with vision-language models and physical feedback

Traditional industrial automation depends on rule-based scripts or domain-specific programming languages, so every setup or reconfiguration demands skilled engineers and long development cycles. LLMs can translate natural language instructions into executable code, control scripts, or robot commands, which shifts task instruction away from expert-driven workflows and lets non-experts interact with robots.

But executing a command in the physical world requires grounding it in sensory data. Most existing VLM-based approaches reason over RGB images alone and never explicitly represent non-visible physical state: the contents of an opaque container, the weight of an object, the contact forces during manipulation. Frameworks that do add tactile or audio signals typically treat them as auxiliary channels rather than integrating them into the reasoning process itself.

MulPlanLM addresses this gap. It combines spatial understanding from RGB images with force-derived physical feedback from a wrist-mounted F/T sensor, and processes both through a hierarchical three-agent architecture — Extractor, Planner, and Sub-Planner — so that high-level planning and low-level code generation are both grounded in visual and physical context.

Figure 1: (a) The user interface, through which a natural-language Task Goal is submitted and a real-time camera feed is monitored. (b) The system architecture: three LLM-based agents with a clear separation of concerns. The Extractor analyzes the environment from RGB images and end-effector force measurements; the Planner designs a high-level task plan from that analysis; the Sub-Planner translates the plan into executable robot code, with a retry mechanism that regenerates failed code up to three times.

Figure 2: Data flow from multimodal perception to action execution. Each modality — the natural-language goal, RGB-based visual detections, and force/torque signals — is processed and structured before being integrated into the Extractor input space, then passed through the Planner and Sub-Planner reasoning pipeline.

Figure 3: Input and output structure of each agent. All three receive the same task goal, and the prompts stay consistent regardless of the task. (a) The Extractor analyzes the environment using vision and force sensor data. (b) The Planner generates a high-level task plan from the task goal and environmental information. (c) The Sub-Planner converts that plan into executable robot control code.

The Extractor perceives and interprets the environment, emitting structured environmental information from sensor inputs and user instructions. Its multimodal input is the set $X_t = \{l, I_t, \bar{F}, \mathcal{G}_{ext}\}$:

• $l$ — the natural-language Task Goal provided by the user.
• $I_t$ — the RGB image annotated with object detection results, yielding an object set $O_t = \{(b_i, c_i)\}_{i=1}^{N}$, with bounding box $b_i$ and class ID $c_i$ superimposed on the raw image.
• $\bar{F}$ — force feedback from the wrist-mounted 6-DoF F/T sensor on the UR5e. The raw measurement $F_t = [F_x, F_y, F_z]^T \in \mathbb{R}^3$ is collected at 100 Hz for 1 second after grasping, and the averaged vector $\bar{F}$ serves as the final input.
• $\mathcal{G}_{ext}$ — guidelines specifying how to interpret multimodal correlations and handle missing sensor data.

The averaged force vector is serialized into structured numerical text tokens and appended to the VLM's textual input. From these inputs the Extractor infers semantic context about the target object and its surroundings without task-specific prior knowledge, and emits structured JSON in two segments: Visible Object Information (object ID, shape, size, and the estimated mass $\hat{m}$ derived from $\bar{F}$) and Visible Environment Information (element types, appearance, spatial locations). Because the output is structured text rather than a model-specific representation, the framework is not tied to any one VLM architecture, and the reasoning process stays interpretable.

The Planner generates a high-level natural-language task plan from the Extractor's analysis. Prior work often generates entire plans or control code in a single end-to-end pass, which can produce execution errors by omitting a verifiable intermediate reasoning step or failing to incorporate environmental feedback. MulPlanLM instead introduces an explicit intermediate planning layer: a human-readable plan that can be verified and modified, and into which contextual information is integrated, before any code is generated.

It takes three inputs — the user's Task Goal; output guidelines comprising an Action Description of basic robot capabilities, Output Examples demonstrating possible action combinations, and High-Level Task Constraints guiding step-by-step composition; and the Extractor's environmental analysis. Formally, given the structured environment representation $\mathcal{E}_t$, it produces the plan $P_t = \mathrm{Planner}(\mathcal{E}_t)$.

The Sub-Planner interprets the natural-language plan and converts it into executable Python control code composed of predefined API-level actions. It takes five inputs: the Task Goal; output guidelines (Action Description, Output Examples, and Low-Level Task Constraints describing motion limits and axis-based rules); the Extractor output; the Planner output; and the object position list from the RGB-D camera.

The action space $\mathcal{A}$ is a finite set of primitives, restricted to four API calls:

• move_robot_to_tcp(target_pose)
• get_robot_tcp()
• gripper(command)
• check_visual_condition()

Generated code is statically verified before execution. Code containing unauthorized commands or invalid formatting is regenerated up to three times before the trial is aborted, and even valid API calls are rejected if they violate workspace boundaries or safety constraints. This dual validation at the API and controller levels keeps execution safe and bounded.

Figure 4: Overall experimental setup. Object positions were obtained from a RealSense D435 camera, while images from a RealSense L515 camera were used as input to the Extractor.

The platform consists of a Universal Robots UR5e arm with a built-in F/T sensor, an OnRobot 2FG7 parallel gripper, and two Intel RealSense RGB-D cameras (L515 and D435). An object detector provides bounding boxes for object detection and position estimation.

Detection was reliable in this setup, so the performance differences analyzed below are more likely attributable to multimodal reasoning, force-based state interpretation, and code-level conditional execution than to detection failures.

Three LLM configurations were evaluated: OpenAI GPT-4V + GPT-4.1 (GPT-4V as the visual information extractor, GPT-4.1 for the planner and sub-planner), Google Gemini 2.5 Pro, and Meta LLaMA 4 Maverick.

15 tasks were designed against two criteria: modality coverage and discriminability across input conditions, and progressive reasoning complexity. Each task is scored as a binary success or failure on completion of a predefined mission, with five repeated trials per model and input condition.

The tasks are split evenly into three categories of five. Vision-based tasks turn on object shape and size (Task 1), color (Tasks 3–4), and other distinguishing features (Tasks 2, 5). Force-based tasks reason about object weight (Tasks 6–10) — in Task 8, for instance, the target moves out of the camera's field of view when pushed, which removes visual comparison entirely and strictly requires force-based state reasoning. Multimodal tasks (Tasks 11–15) need both at once: reading text on paper while weighing an object, comparing weights of visually identified blocks, or selecting the lightest of several filled containers.

Complexity increases across the set. Some tasks are single-attribute selection; Task 8 requires detecting weight changes at intermediate steps; Tasks 9 and 10 require distinct choices under specific weight conditions; and the multimodal tasks require conditional action planning over synthesized multimodal context. Execution times estimated from the recorded videos follow the same gradient — roughly 15 s for vision-based pick-and-place, 30–40 s for force-based tasks, and about 1 min for multimodal tasks. These durations come from video observation rather than instrumented logging and are therefore approximate.

Every model performs well whenever visual information is present and collapses without it. GPT-4V + GPT-4.1 reaches 84% with vision and force against 76% vision-only, but falls to 12% on force alone. Gemini 2.5 Pro and LLaMA 4 follow the same pattern — 80% in both the multimodal and vision-only settings, dropping to 4% and 12% respectively on force alone. Vision is the primary cue for this group, and adding force provides limited additional benefit.

Here the ordering inverts. Vision-only success is low across all three models, since the state of stacked or occluded objects cannot be inferred visually. Force alone solves some tasks by combining positional and force cues, but without visual context it still trails the multimodal condition. Task 10 is the hardest in this group — the weight difference between objects is minimal, which makes force cues alone difficult to discriminate.

Gemini 2.5 Pro achieves the highest average here. GPT-4V + GPT-4.1 is slightly lower overall but consistent across tasks, and LLaMA 4 is likewise stable in force-dominant settings. Force alone enables moderate performance; combining it with vision improves it further.

Figure 5: Success and failure cases for Only Force, Only Vision, and the proposed method given the same task goal. Because the box is actually empty, it should be moved onto the paper labeled "Empty."

These tasks are decision problems that need vision and force jointly — appearance alone is insufficient, and force alone cannot supply a complete interpretation. Tasks 11 and 12 score relatively high across all models: they analyze force information against visual cues written on paper, so the two modalities interact without conflict and their complementary effect shows clearly.

For every model the best result comes from using both modalities. Performance drops sharply under vision-only or force-only conditions because essential cues are missing — force-only success in this group is near zero (0–4%). Both modalities are necessary for reliable performance here.

Figure 6: Success rates (%) for overall task performance (Tasks 1–15) across GPT-4V + GPT-4.1, Gemini 2.5 Pro, and LLaMA 4 under three input modalities: Vision + Force, Vision only, and Force only. Error bars represent standard error across trials.

Averaged over all 15 tasks, the multimodal condition is both the highest and the most consistent. Gemini 2.5 Pro shows the highest overall consistency across tasks, GPT-4V + GPT-4.1 performs well where visual interpretation dominates, and LLaMA 4 stays relatively stable where force-based reasoning dominates.

This matters for deployment. Vision-only approaches work when clear visual cues are available but fail under internal-state inference or occlusion — conditions that are common in practical robotic environments. Force data acts as the complementary source that covers those cases.

Breaking the results down by category exposes the framework's scope and its systematic limits. Vision-based tasks average 80.0% (±9.3) and force-based tasks 72.0% (±9.6); both involve mostly single-modality reasoning and threshold-based physical judgments, where performance is relatively stable. Tasks requiring genuine integration of vision and force average 69.3% (±12.4) — lower, though still a clear advantage over the corresponding unimodal baselines (41.3% vision-only, 2.7% force-only).

Degradation concentrates in tasks that require (1) intermediate state updates during execution, (2) deep conditional branching, or (3) sequential comparative reasoning across multiple objects. Failures in these cases typically came from incomplete condition updates at the code level rather than from misreading the force signals — the framework is stable on single-step perceptual or threshold-based reasoning, while explicit intermediate state tracking and hierarchical procedural reasoning would benefit from further refinement.

Integrating visual information with force-derived physical feedback improves LLM-based task planning specifically when task-relevant physical states are not directly observable from vision. Vision supplies symbolic and spatial grounding for identifying and localizing candidate objects; force supplies the additional evidence that resolves hidden properties such as weight or internal contents. The improvement therefore comes from modality complementarity rather than redundant sensing.

The framework does not attempt to solve low-level contact-rich manipulation, continuous force regulation, or force-control policy learning. Its contribution is enabling force-based state reasoning within a symbolic task-planning framework, not addressing contact dynamics directly.

Model-dependent performance differences further suggest that reasoning over force-derived numerical signals may be partially separable from visual scene understanding. These observations are specific to this framework and task setup, and should not be read as a general benchmark of model-level performance.

Failure analysis points at downstream reasoning and planning rather than object-level perception. Smaller weight differences and visual occlusion did appear to increase difficulty, but they were not independently controlled as a predefined difficulty axis. Future work will parameterize task difficulty by systematically varying occlusion level, lighting condition, contact ambiguity, weight-gap granularity, and conditional branching complexity. Task completion time also warrants separate treatment: timing statistics were not systematically recorded across all trials, so future evaluations will measure VLM/LLM inference time, planning time, code generation time, physical execution time, and recovery procedures independently.

MulPlanLM is an LLM-based robotic task-planning framework that integrates visual information and force-derived physical feedback for physically grounded decision-making. It improves task execution where vision or force alone provides incomplete information, particularly when hidden physical states such as object weight or internal contents must be considered. Incorporating explicit state tracking, post-condition verification, and closed-loop replanning is expected to further improve performance on tasks involving conditional reasoning and sequential decision-making.