Unbundling Multimodal Vision Encoding

§2Setup

What a vision encoder bundles

Most multimodal models acquire visual understanding from a vision encoder pretrained for a separate objective, such as image recognition or image–text alignment. Recent recipes discard parts of that setup. Kimi K3 trains its encoder from scratch with the multimodal backbone, and encoder-free models feed image patches straight into the backbone. Each recipe implies a different answer to what the encoder contributes, if anything at all.

One controlled protocol

Published models differ in data, architecture, training stages, and compute, so comparing them cannot isolate any one choice. Controlled studies of pretrained encoders hold the multimodal stage fixed, but they vary which prior an encoder imports rather than whether it needs one. They also treat encoder pretraining as free, though SigLIP 2, for example, trains on 40B image–text samples. We instead run every training stage ourselves, from random initialization.

Every recipe pretrains on the same text and image–text pairs, with images at 224×224. Avg. VQA averages four category means over 16 Cambrian benchmarks, after instruction tuning on Cambrian-7M with 448×448 images. Avg. Text does the same over 12 language benchmarks.

§3Is separate pretraining needed?→

Separate pretraining is optional

Under a fixed lifetime budget, encoder pretraining competes with multimodal training for compute. We pretrain the same encoder with four objectives (contrastive MetaCLIP, self-supervised WebSSL, and captioning with Cap and CapPa) on the same images, update it during multimodal training, and compare it with the encoder trained from scratch with the backbone.

At 1.7B and 4B, MetaCLIP trails the scratch encoder at every budget, by 1.8 Avg. VQA points at the largest 4B budget.

§4Is vision capacity needed?→

Dedicated vision capacity is key

Encoder-free models drop the encoder and map each image patch directly into the backbone with a linear embedding. Removing the encoder also removes its parameters and changes how raw patches enter the backbone, so a gap could reflect fewer parameters, a weak patch embedding, or missing dedicated capacity.

Across budgets, encoder-free trails by 4.1–6.4 Avg. VQA points at 1.7B and 2.8–6.8 at 4B. To test whether the gap is just the encoder's extra parameters, we give the encoder-free backbone the encoder's 0.4B parameters as extra width, depth, or both.

A richer patch embedding doesn't close it either. Trained on the same 224×224 images, a two-layer projection inspired by Gemma 4 helps, scoring 40.5 Avg. VQA against 39.0 for the linear default, but stays well below the encoder's 44.4.

§5Must capacity precede fusion?→

Vision capacity can move in-fusion

An encoder's parameters are dedicated, used only by vision tokens, and pre-fusion, run before vision and text interact. To separate the two, we move the encoder's 0.4B-parameter budget in-fusion, into vision-only modules inside the shared backbone.

Holding the budget at 0.4B parameters, we vary where it goes and which tokens use it.

Placed in the first eight layers, the vision-only FFN branch matches the scratch encoder (45.0 vs. 44.4), while the other placements do no better than encoder-free. Early placement also helps most on OCR and chart questions (36.0 vs. 25.2 for encoder-free), which depend on fine visual detail. With the first eight layers as the default, in-fusion, which also adds vision-only attention projections, holds the same dedicated budget as the encoder on the other side of fusion.

§5.4Scale

The comparisons hold with scale

We also train the three recipes on a ladder of backbone sizes and token budgets, growing the vision module with the backbone, and fit one law per recipe in model size and training tokens. The figure shows the runs at 64 tokens per parameter.

Caption loss shows the same pattern: encoder-free has higher (worse) loss than the encoder at every size, while in-fusion matches it. On text loss and Avg. Text, the three recipes are comparable.

§6Beyond lifetime compute

The findings carry over, and in-fusion can be faster to serve

Lifetime compute isolates each design choice, but teams often build on pretrained components and care about inference speed. As shown above, a pretrained encoder doesn't help even when its pretraining is free. We also test adding vision to a pretrained language model, and measure time to first token.

Midtraining: adding vision to a pretrained language model

We continue our own 1.7B text-only model from 1T to 1.33T token-equivalents, adding each recipe at equal compute, and include mixture-of-transformers (MoT) for comparison.

MoT with vision-only copies of all 28 layers falls behind MoT on the first eight layers alone, despite 3.5× as many vision-only parameters, consistent with late placement not helping.

Time to first token

A separate encoder must finish each image before the backbone can start, which adds a serial step to inference. We measure time to first token for a 1.7B backbone on one H200 serving 16 concurrent requests, each with one 448-px image and 256 text tokens.

MoT, which copies whole backbone layers for vision tokens, is about as fast as in-fusion up to 0.4B vision-only parameters, but its added time grows faster with the budget, while staying below the encoder's. The encoder-free model is fastest, but at 1.7B and 2.6×10²¹ FLOPs it trails the encoder by 5.4 Avg. VQA points.

FAQ

Common questions

Citation

BibTeX

@article{brown2026unbundling,
  title   = {Unbundling Multimodal Vision Encoding},
  author  = {Brown, Ellis and Hoang, Chris and Qian, Shengyi and
             Li, Shang-Wen and Lewis, Mike and Ghazvininejad, Marjan and
             Zettlemoyer, Luke and Beyer, Lucas and Ren, Mengye and
             Verbeek, Jakob and Xie, Saining and Xu, Hu},
  journal = {arXiv preprint arXiv:2611.XXXXX},
  year    = {2026}
}