EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

Real-Robot Evaluation

We fine-tune VLAs on 50 teleoperated demonstrations per task and evaluate 10 rollouts per task on a Franka Research 3 arm with a Robotiq 2F-85 gripper. The same mid-trained checkpoint used in our simulation experiments (selected with simulation VLA data only) increases the overall real-world success rate from 54% to 72% on the InternVL3.5-1B backbone. Below are rollouts from our evaluation runs: for each task, the VLA fine-tuned from the off-the-shelf VLM (left) and from the VLM after EmbodiedMidtrain (right).

Videos play at 3× real time.

Sim-selected data transfers to the real world. The proximity estimator only ever sees simulation VLA data (CALVIN, Bridge, LIBERO), yet the resulting mid-trained VLM improves four of five real-robot tasks.

Better grasping and placement. In the qualitative examples, the baseline grasps too close to an edge, approaches at insufficient height, or loses the object during transport, while the mid-trained model grasps, transports, and places successfully.

+18 points overall. Success over 50 evaluation rollouts rises from 54% to 72%, with the largest gain (+30) on the cube-to-cup task.

Abstract

The Data Distribution Gap between VLMs and VLAs

Since most VLAs inherit their representations from VLM pretraining, the quality of this initialization is fundamentally shaped by the data the VLM was trained on. We represent VLM and VLA data in a shared feature space (last hidden states of InternVL3.5) and quantify the distance between each pair of datasets with Maximum Mean Discrepancy (MMD):

where $k(x,y) = \exp\!\big(-\|h(x) - h(y)\|_2^2 \,/\, 2\sigma^2\big)$ is a Gaussian RBF kernel with bandwidth $\sigma$ set via the median heuristic.

Key findings

Data Engine for EmbodiedMidtrain

We construct a candidate pool spanning general VLM data (LAION-400M, CC-12M with BLIP captions, LLaVA-Instruct-665k, VCR) and embodied-oriented VLM data (RefSpatial, EmbSpatial-Bench, Robo2VLM, RoboPoint), and select at the sample level to account for variation within each source.

Proximity-based data selection

Given a selection budget $K$ and a scoring function $s$, we select the subset of the candidate pool $\mathcal{D}_{\mathrm{VLM}}$ with the highest aggregate proximity to the target VLA corpus $\mathcal{D}_{\mathrm{VLA}}$:

Framing sample scoring as domain-membership classification, the population-optimal score under equal domain priors is monotonically increasing in the density ratio between the two domains:

A lightweight learnable proximity estimator

We instantiate the estimator as a linear scoring function $f$ on the frozen VLM’s last-token, last-hidden-state representation $\phi(x)$, followed by a sigmoid: $\; s(x) = \sigma\big(f(\phi(x))\big)$. It is trained with VLA samples as positives and candidate VLM samples as negatives under binary cross-entropy:

Scoring the full pool and retaining the top $K{=}1.28$M samples produces EmbodiedMidtrain-1M, the corpus used to mid-train the VLM before VLA fine-tuning. The pipeline is lightweight, effective, and requires no architectural changes to the underlying VLM or VLA.

Main Results

Following the VLM4VLA evaluation protocol, we fine-tune each VLM into a VLA with an identical two-branch MLP action head and robot data, and evaluate on CALVIN ABC-D (avg. consecutive tasks completed), SimplerEnv-Bridge and LIBERO-10 (success rates). With proximity-based mid-training, our models achieve consistent and substantial improvements across all three benchmarks while being fine-tuned on the same robot action data.

Main results on CALVIN ABC-D, SimplerEnv-Bridge, and LIBERO-10. Full comparisons with expert VLA baselines and off-the-shelf VLM baselines, along with per-subtask CALVIN breakdowns, are in the paper.

The gains transfer

Analysis

Mid-training provides a better initialization, not a head start

The mid-trained model leads from the earliest fine-tuning checkpoints, and the gap widens rather than narrows over training, indicating that proximity-based mid-training yields a backbone better suited to VLA learning, not a transient warm start.

What does the proximity estimator prefer?

Proximity scores across the eight candidate VLM datasets

High score Spatial grounding and reasoning

Q: You are standing at the point marked by the coordinate point at point (0.878, 0.780). Which object is directly in front of you? A: The white matte truck at lower right.

Q: Locate a point on the yellow metallic crane at upper right. A: [(0.976, 0.244)]

Low score Text-only VQA on a book cover

Q: Who wrote this book?
A: Charles P. McKeague.

Q: What is the title of this book?
A: Trigonometry.

The estimator assigns clear dataset-level preferences (RefSpatial highest, VCR lowest) while also performing fine-grained sample-level selection within every source. High-scoring samples exercise spatial grounding and reasoning; low-scoring ones involve text-centric content with little relevance to embodied tasks. This suggests the estimator learns to distinguish visual reasoning patterns that transfer to embodied control from those that do not, without requiring manual domain expertise. The selected mixture also retains the diversity of the VLM data (uniformity 1.93, closely matching the general VLM pool at 1.96), shifting the distribution toward the VLA domain without collapsing it.

Ablations

Random sampling from the candidate pool consistently underperforms the learned estimator, showing that the gains come from identifying the aligned subset, not merely from additional mid-training. Hand-crafted proximity metrics capture VLA alignment only partially; the learned estimator is the most robust and transferable signal.

Ablation of selection strategies for mid-training the InternVL3.5-1B backbone.

BibTeX

@misc{du2026embodiedmidtrainbridginggapvisionlanguage,
      title={EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training},
      author={Yiyang Du and Zhanqiu Guo and Haochen Zhang and Annabella Macaluso and Xin Ye and Liu Ren and Yonatan Bisk and Chenyan Xiong},
      year={2026},
      eprint={2604.20012},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.20012},
}