Shared dimensions: a new direction for going multimodal

No benchmarks of our own here. The numbers come from the paper and the model card, linked at the end; the judgments are ours and labelled.

Start with a scene

Suppose you want a system that can search everything inside a company.

Contracts are PDFs. Invoices are scans. Meetings are audio. Product photos are images.

Multimodal used to offer two paths. One was to run four models — text embeddings, image embeddings, speech-to-text, audio classification — with four vector spaces that do not talk to each other, plus an alignment layer on top. The other was to make one model big enough to swallow every modality.

In October 2026, Google's EmbeddingGemma 2 demonstrated a third path: share the semantic backbone, and isolate capacity for each modality in branches. Text-only runs at 271M; turn on vision when you need images, turn on audio when you need sound.

What is actually valuable here is not "one model handles four data types". It is that going multimodal does not require making the whole model fatter.

The old bill for scaling up

To see why the third path matters, look at what the old paths cost.

The first is multiplication. A dense model's bill is parameters times tokens. Want one more capability, and you scale everything up together.

The second is gradient conflict. The same weights must satisfy "this pixel is a cat" and "this word is a subject". The gradients pull in different directions — negative transfer, in multitask learning terms.

The third is input geometry. Text is discrete tokens, images are pixel grids, audio is waveforms. They do not even agree on what one unit is.

These three costs are not an argument that multimodal is impossible. They explain why the old path stops paying off past a certain size. The value of the new direction is a different way to pay.

The new direction: shared dimensions, isolated capacity

EmbeddingGemma 2 splits at the entrance and merges in semantics.

image/video/audio ──[per-modality encoders]──> soft tokens ──[one 24-layer transformer]──> vector
                        ↑ split                                    ↑ shared

Each modality encoder handles low-level features (pixels, waveforms) and turns them into "soft tokens" — continuous vectors that sit in the sequence like text tokens but need not correspond to any word. High-level meaning goes through one shared transformer.

The parameters split like this: a 271M trunk (text also runs here), a 167M vision encoder, and a 305M audio encoder.

Note that 305M: the audio encoder is bigger than the entire text trunk.

That is the point of the new direction. Capacity lives in branches, so it can be switched off:

On a Pixel 11 Pro at Q4_C, peak memory (including runtime compilation overhead): 191 MB for text, 447 MB for text plus images, 567 MB for everything.

And text quality does not pay for it. Its 271M text path scores 61.36 on multilingual MTEB; the previous generation at 308M scored 61.15. Fewer parameters, three extra modalities, no text regression.

That is the most promising part: scale can be opened selectively instead of expanded universally.

Two ways to split, two bills

"Share the trunk, isolate capacity" can be done at different places. Two current answers:

MoE: split in the middle of the model

Many experts sit inside a layer; each token visits a few, chosen by a router.

What it buys: capacity decouples from compute per token. Parameters can be plentiful while each token touches a small slice.

The cost is training difficulty: router collapse, uneven expert load, sparse gradients. MoE trades training difficulty for inference cost.

EmbeddingGemma 2: split at the entrance

Heterogeneous modalities go to designed encoders; homogeneous meaning goes to the shared trunk.

What it buys: capacity decouples from resident memory. You do not keep the whole model resident for the occasional video search.

These are not the same problem, and they are not mutually exclusive. One saves compute; the other saves memory.

Early fusion, late fusion: where alignment starts

If you know CLIP, this looks like "multimodal, again".

The difference is where the merge happens.

Late fusion (CLIP, SigLIP): two full encoders run independently, then a contrastive loss aligns them. The towers never exchange information.

Early fusion (EmbeddingGemma 2): encoders produce soft tokens that enter one transformer, where all modalities mix.

Late fusion lets each tower be strong on its own; the cost is that it cannot learn cross-modal interaction — text inside images, the original audio of a video, the relationship between a chart and its heading.

Early fusion costs something too: every modality must fit through one narrow channel. The model is 512-dimensional and 24 layers deep. (Our reading: deliberately narrow.)

Its bet: only a narrow semantic space forces a genuinely shared representation.

Three pieces of evidence for this direction

The paper ran ablations; three of them speak directly to the "shared trunk" idea.

Removing a text-only mechanism improves everything

Gemma's per-layer embeddings used an extra lookup table to inject token information into intermediate layers. EmbeddingGemma 2 drops the lookup table and keeps only the projection.

Overall goes from 55.07 to 55.46; image 54.57 → 54.67, video 47.13 → 47.82, visual documents 61.76 → 62.37. These ablations are intermediate training recipes, not the final model (which reaches 59.01 MMEB-v2 overall).

Removing it made everything better, text included. The paper's explanation: the lookup table carries "no direct signal" for non-text modalities, and costs memory.

Our reading: a shared space is achieved by removing things, not adding them. Any mechanism that lets one modality bypass the shared space drags the whole down.

The vision encoder cannot be frozen

Trainable: 52.27 overall, 50.10 image, 43.26 video. Frozen: 48.18, 44.35, 39.24 — a 4.1 point gap.

Many multimodal models freeze modality encoders to save cost and preserve prior ability. This result says otherwise: encoders have to be trained to adapt to the shared space.

For this direction that is both good news and a reminder — the shared trunk is not free; it asks every branch to converge somewhere, and that convergence can be trained.

The audio encoder is frozen, and audio merely ties

The paper is explicit: the audio encoder is identical to Gemma 4 E2B/E4B and frozen during training.

Audio is also where it is closest to its rival: MAEB Mean(Task) 49.87 versus Jina Omni Nano 50.14, a 0.27 gap. Mean(Type) is 48.65 versus 55.31 — not a tie. By contrast, image is 57.28 versus 55.43 and video 50.67 versus 35.87, both clearly ahead.

The authors leave one line: further training of the audio encoder "may also yield similar gains".

This is our speculation, not the paper's conclusion: unfreezing audio could open that gap. They did not do it, so we can only say it is possible — and that is exactly the part of this path that is not finished.

How to use it: selection is not a ranking

What matters is what your use case needs, not who is first.

Some numbers worth reading together:

(— means the paper reports no score for that cell; the paper uses ∗ for "not supported" and – for "score unavailable", which are not the same thing. All numbers from Table 6.)

Three things stand out.

First, versus the previous generation, text barely moved — but three modalities appeared. The real difference is not the score; it is that the extra capability did not eat the existing one.

Second, text-only it loses to specialists. Jina leads by 4.2 on multilingual and 2.7 on English; Qwen3 by 3.0. And 744M is the "everything loaded" number — text-only runs at 271M. The appendix gives the fair comparison: harrier-oss-v1-270m at 268M scores 66.5 and jina-embed-v5-text-nano at 212M scores 65.5, both higher. Both are text specialists.

That is not a flaw, it is a division of labour. Wanting one checkpoint that can grow into other modalities, and wanting the strongest text model, are two different requirements.

Third, multimodal it leads clearly. Video 50.67 versus Jina 30.73 and InterestFM 35.87 — a 15 to 20 point gap. Image 57.28 versus 55.43 is modest but consistent.

On audio: same model, native audio 69.54 versus 57.59 for ASR — about 12 points. What is lost in transcription is the non-linguistic part of sound. (The previous generation's ASR scored 50.95; comparing that to 69.54 gives 18.6, but that is a generational comparison, not a modality gap.)

What is not finished

To judge whether a new path is worth betting on, look at what it still lacks. Four things.

Text is not the strongest. The Pareto frontier the paper claims is a quality-versus-parameters efficiency frontier (beating models 2–12× larger), not a first place in every column.

Visual documents still have room. At a 280-token budget, VisDoc is 67.84 versus Jina Omni Nano's 71.34. Scans, tables and dense layouts depend on fine visual detail, and the paper notes that raising the token budget still improves VisDoc — this looks more like "not tuned yet" than a ceiling.

Quantization hurts audio most. Moving from bfloat16 to int4, text barely notices (61.36 → 60.84) while audio drops 5.6 points (69.54 → 63.92). The paper notes, however, that float and quantized runs use different evaluation harnesses, so the drop includes harness differences and cannot be blamed on quantization alone.

The same token budget moves three tasks in three directions. Images are trained at 280 tokens per frame, video at 140; raising inference from 280 to 560 improves visual documents sharply (63.38 → 67.62), saturates images, and makes video worse (48.51 → 46.52). The paper attributes this to excessive sequence length diluting attention and mean-pooled representations.

So the knob has to be tuned per task — not a defect, but the control this direction gives you.

Conclusion: a new direction for scaling up

Back to the scene. You no longer have to choose between "four models" and "one big model".

Shared dimensions, capacity isolated in branches: one semantic space, open the branch you need.

Two ways to split, two different bills:

MoE splits inside the model. Homogeneous experts, a learned router decides who works. It buys decoupling of capacity from compute per token.

EmbeddingGemma 2 splits at the entrance. Heterogeneous modalities, designed encoders decide who handles what. It buys decoupling of capacity from resident memory.

The shared lesson: splitting is not about getting bigger; it is about making some parts switchable off.

And the most counter-intuitive lesson of all — a shared space comes from removing specialised machinery, not from adding more modules.

We have not measured this model's deployment difficulty, tooling support or real latency, so this is not a recommendation to migrate. If you do, measure three things first: how big your text trunk actually is, whether the branches can be switched off, and what share of your queries are cross-modal. Sources: arXiv 2610.37364, Gemma docs. Related MoE work: arXiv 2610.37386 (channel selection, not a MoE survey).