Beyond Appearance Shifts: Task-Semantic Action Calibration for VLA Models
Abstract
Vision-language-action (VLA) models have achieved strong performance in embodied manipulation, but still struggle to distinguish between changes that should preserve behavior and those that require different actions. We identify two complementary failure modes. Under task-preserving changes, where task semantics remain unchanged but scene appearance varies (e.g., style, illumination, clutter, or paraphrasing), policies often exhibit unnecessary action drift. Conversely, under semantic-breaking changes, where key task semantics such as the target object or constraint are altered, policies frequently continue following the original trajectory instead of adapting to the new task. To address these failures, we propose BAS-VLA, a task-semantic action calibration framework built on top of a frozen base VLA. BAS-VLA adopts a breaking-centered calibration core as its default path, complemented by a selective evidence-gated preserving auxiliary that corrects appearance-induced drift when task-preserving evidence is strong. Across four target-object swap tasks, BAS-VLA improves changed-task success from 2.3% to 68.5% while maintaining strong clean and semantics-preserving performance and suppressing outdated behavior. On validated style-preserving shifts, it further improves success from 42% to 70% without degrading clean performance. These results demonstrate the importance of calibrating VLA policies to remain stable when task semantics are preserved and adapt when task semantics change.
Overview
BAS-VLA frames reliability as an action-level calibration problem. Task-preserving interventions should remain in the same action basin, while semantic-breaking interventions should push the policy away from the stale clean trajectory.
- Task-preserving changes: style, illumination, clutter, viewpoint, or wording changes that keep task semantics fixed.
- Semantic-breaking changes: edits to target objects or constraints that require a different behavior.
- Frozen carrier: the base VLA remains fixed; BAS-VLA attaches a lightweight calibration layer at the action output.
Figure 1. BAS-VLA overview. Part (A) summarizes the two intervention types studied in this work: task-preserving changes and semantic-breaking changes. Part (B) shows BAS-VLA as an action-calibration layer over a frozen base VLA. Part (C) illustrates the task-aware visual filtering probe, from RGB input to text-conditioned grounding, SAM 2 mask refinement, merged task-relevant masks, and the resulting probe image.
Motivation
Strong VLA backbones can still fail in opposite ways. Under appearance-only changes, they may drift from an otherwise correct behavior. Under true semantic changes, they may keep executing the original task. BAS-VLA separates these cases by making the calibration target explicit at the action level.
Figure 2. Task-preserving motivation on a single Black-Bowl-to-Plate instruction. The three panels show a clean success, a failure under a darkened illumination shift, and a failure under a semantics-preserving noise perturbation.
Method
Breaking-Centered Core
The default BAS-VLA path adds a small residual calibrator to the frozen base action. Matched clean, preserving-control, and semantic-break triplets train the residual to keep clean/control actions close while pushing breaking edits outside the clean action basin.
Evidence-Gated Preserving Auxiliary
The preserving auxiliary is intentionally selective. It activates only when early rollout phase, frozen visual-feature discrepancy, semantic consistency, and action disagreement jointly indicate nuisance-induced drift rather than a true task change.
Figure 3. Dark-scene preserving example. The frozen carrier misreads the target under illumination change; the evidence-gated probe path activates and produces a corrected action.
Results
The tables below reproduce the corresponding paper tables in full rather than using shortened web summaries.
Table 1. Motivation evidence for reliability gaps before BAS-VLA. Original denotes native carrier performance on the unmodified task condition, and Perturbed denotes performance under the corresponding preserving or task-changed intervention.
Table 3. Main task-preserving style-shift results on the Bowl-on-Ramekin style-shift setting. Each method uses 200 episodes over four seeds; Δstyle is measured relative to the baseline style-shift row.
Table 4. Main semantic-breaking results on matched target-object triplets. Each row uses 200 clean, 200 control, and 200 break episodes; lower is better in Break.
Table 5. Matched-carrier external strategy comparison on Milk-Swap. Each strategy uses 200 clean, 200 control, and 200 break episodes; lower is better in Break.
Table 6. Real-robot changed-task completion. Every case uses 80 trials per method, for 640 changed-condition rollouts in total. T1/T2 include the initial 17 trials per method.
Table 7. Changed-task outcome decomposition. New, Old, and Other/timeout are mutually exclusive; easy swaps aggregate Milk/OJ/BBQ/Ketchup.
Table 8. External-protocol success without benchmark-specific tuning. LIBERO-Plus: four categories × ten cases × 120 rollouts per method. LIBERO-PRO: ten cases × 120 rollouts per method.
Initial real-robot clean/control and behavioral diagnostics
Table 11. Real-robot semantic-adaptation results on two deployment tasks. Each method-task pair uses 16 clean, 17 control, and 17 break episodes. Here Break reports success on the changed instruction, and the separation score averages old-target suppression and new-target first-commit.
Image Gallery
The gallery below presents selected figures from the paper.
Mean steps.
Median steps.
190-step saturation.
Break-clean by family.
Break-expert by family.
Control invariance by family.
Citation
@inproceedings{liu2026basvla,
title = {Beyond Appearance Shifts: Task-Semantic Action Calibration for VLA Models},
author = {Liu, Shuaijun and You, Feiyang and Wu, Chengyu and Hao, Shuyang and Zhang, Chenglong and Cai, Jingyao and Chen, Xingwei and Sun, Li and Su, Ningxin},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://arxiv.org/abs/2609.23650}
}