MJev-Spatial: Candidate Classification and Refinement

Spatial answers can be hard to generate and parse reliably; MJev-Spatial turns options, numeric ranges, and image regions into candidates scored with pretrained attention projections, then refines the selected range or location.

Read the paper ↗

The short version

What it is

Candidate scoring Instead of writing an answer as text, the model chooses among explicit possibilities: answer options, numeric ranges, or image regions.
Coarse-to-fine refinement A broad first choice narrows the next set of possibilities; for locations, the model revisits a selected image crop to inspect it more closely.
Pretrained matching The model reuses its existing attention matching machinery to compare the question’s decision state with each candidate.

Why it matters

Different spatial answers need different representations, but they can share a decision interface. Refinement avoids asking one flat decision to distinguish every fine value or location, while crop re-encoding gives later pointing decisions more local visual context.

Before we get into the machinery, here is the basic idea: MJev-Spatial turns a spatial answer into a choice among explicit candidates. The candidates might be the options in a question, ranges of distances, or regions of an image. A broad choice can narrow what the model considers next, so it does not have to pick every fine-grained answer in one giant leap.

Answer format: Spatial questions can ask for a yes-or-no choice, a distance, or a point. If the model generates text, getting the meaning right is only part of the job; the answer also has to be expressed and parsed correctly.
Resolution: A number or location can need more precision than a short list of answer choices. A flat list of every possible fine answer is not an attractive way to represent that problem.
Visual detail: A point prediction has to locate something in the image, not just name a broad area. Once a broad area is selected, looking at that part more closely can help the next decision.

So what does it mean to make these tasks classification? The canonical example is a yes-or-no question like “Is the chair in front of the desk?” The candidates are simply “yes” and “no,” and the model selects one. The same idea generalizes: a number candidate can stand for a range, and a point candidate can stand for a region. The rule for choosing is shared; what the candidate means and how it becomes a final answer depend on the task.

The model scores candidates using its existing attention matching machinery. Intuitively, it builds a representation of the decision to make and compares it with a representation of each candidate. It is like checking which labeled card best matches the question—not a literal lookup table, because the candidate representations are contextualized by the surrounding image and text. The useful twist is that the matching uses pretrained query and key projections rather than adding a separate prediction head.

But can the candidate list stay small and still give a precise answer? For a distance, think of first choosing a neighborhood on a ruler, then choosing a smaller interval inside it. The first choice is coarse; the next one spends resolution only inside that selected range. The model decodes the final interval by its midpoint. Counts use a related two-choice structure, while distances and angles have separate codebooks and units. This is why “one scorer” does not mean one universal numerical scale.

Pointing adds a visual version of the same idea. The canonical example is finding a pillow in a full image: first choose a broad cell, then crop that selected area from the original image and look again, then refine once more. The original image stays available for context, while the crop gives the next choice more local detail. The crop comes from the original pixels rather than repeatedly shrinking an earlier crop, which avoids compounding resampling. The limit of the notebook analogy is that a crop does not erase the global view; both views inform the decision.

What if the question asks for several points rather than one? A single greedy path gives one location, so the decoder can instead expand multiple plausible branches. It combines the conditional probabilities along each path into a spatial map, then groups connected high-probability cells into point predictions. That gives a way to recover multiple locations without learning a separate stop decision, at the cost of more inference work. What remains is the practical tradeoff: how many branches and crop encodings are worth paying for on a given task?

Overall, I like the clean separation here: the scoring operation is reused, but the task controller still decides what candidates mean and what happens after a selection. That makes the interface coherent without pretending that a count, a distance, and a point are the same prediction problem.

Read the deep dive

Why People Wanted It

A spatial answer might be a supplied choice, a distance with units, or a location inside an image. Generating each as free-form text makes the output format part of the problem: a model must say the right thing and express it in a form the system can interpret. The real question was whether these different answers could share one decision rule without flattening their different structures.

There is a second tension. A single decision over every possible fine-grained number or pixel region would require an unwieldy candidate set. So the useful question is not only how to score answers, but how to make a coarse choice narrow the next decision while preserving visual detail where location matters.

The Core Move

Turn each answer possibility at the current stage into a candidate, score candidates by matching the question’s decision state against their contextualized representations, and let the selected candidate determine any next-stage candidates or crop.

For candidate $i$ at stage $t$ , the basic decision is $\hat{i}_t=\arg\max_i z_{t,i}$ . In words: choose the highest-scoring currently legal candidate; for multi-stage tasks, that choice defines what can be considered next.

Why One Candidate Rule Can Cover Different Answers

Now the version for readers who want more of the machinery: the unification is over the decision operation, not over the meaning or decoding of every answer type. A choice candidate is one supplied option. A numerical candidate is an interval or a special class such as zero or overflow. A point candidate is a region in normalized image coordinates. At each stage, only legal candidates enter the softmax; padded candidate slots are masked with $-\infty$ so they cannot win.

That distinction matters because the candidate controller carries task structure. The scorer returns a class, while a task-specific decoder maps the class back to an option, a value, or a location. Numerical quantities also keep separate codebooks and declared units: counts, meters, and degrees are not interchangeable merely because all are classified. What still needs explanation is how these distinct candidate descriptions get comparable scores without a new task-specific head.

Why Pretrained Query and Key Projections Can Score Candidates

Here is the part that gets technical: each candidate is followed by the same End marker, and its contextual state is passed through the pretrained key projection $W_K$ ; the final Decide state supplies a query through pretrained $W_Q$ . Candidate logits come from the similarities between that query and the candidate keys, retaining the model’s native normalization, rotary transformations, and head averaging.

In compact form, $z_i=\operatorname{sim}(W_Q h_{\mathrm{Decide}}, W_K h_{\mathrm{End},i})$ . In words, transform the decision state and each candidate state with the existing projections, then use their similarity as the candidate score. The architecture adds no prediction head or new parameters for this matching step. But a good score alone does not make an enormous fine-grained candidate set practical: how can the system spend resolution only where the current decision points?

Why Refinement Trades Candidate Count for Context

Now the deeper refinement mechanism: for numerical prediction, a coarse interval selects a smaller fine-bin set, and the final answer is decoded as that selected fine cell’s midpoint. Exact-zero and overflow classes can terminate after the coarse decision. Counts use two discrete choices—coarse group and within-group integer—while length and angle use their own codebooks.

Point prediction uses a different kind of refinement. Each of three decisions selects one of nine cells in the current rectangle; after the first two choices, the selected rectangle is cropped from the original image and re-encoded. The original view remains available for global cues, while each crop allocates visual tokens to the selected area. The final cell center is expressed in original-image coordinates. For several points, the decoder expands conditional branches into a spatial probability map, groups connected high-probability cells, and returns one representative per group. This is conditional tree expansion, not an additional learned stop decision. The tradeoff left open is how much branching and re-encoding a given task can afford.

Minimal Causal Code

This loop shows the classification-and-refinement control flow; score stands in for the model’s shared candidate scorer, and each task supplies the next legal candidates.

# candidates_for(stage, history) returns only legal candidates.
# score(question, history, candidates) returns one logit per candidate.
history = []
for stage in range(max_stages):
    candidates = candidates_for(stage, history)
    logits = score(question, history, candidates)
    selected = candidates[logits.argmax()]
    history.append(selected)
    if selected.is_terminal:
        break
answer = decode(history)

What You Gain, and What You Lose

You gain a common candidate-scoring interface for supplied options, numeric codebooks, and image regions; answer selection does not depend on generating and parsing answer tokens. Reusing pretrained query/key projections avoids adding a dedicated prediction head for this matching step. Coarse-to-fine decisions keep each stage’s candidate set focused, and point crops add local visual detail while retaining the original image context.

You do not get a universal fixed-cost prediction. Numerical and point answers can require multiple decisions; crop refinement requires additional image encodings, and expanding multiple-point branches costs more inference work. A wrong early choice can exclude the correct fine region, and the geometric resolution of a final cell is not a guarantee that the model selected the right path. Candidate design and decoding remain task-specific, even though the scorer is shared.

Quiz

What is shared across choice, numerical, and point decisions in MJev-Spatial?

One universal codebook that maps all answers to meters

A separate newly learned prediction head for every task

A free-form text generator followed by answer parsing

A candidate scorer that selects among the currently legal candidates

Quiz

How does point prediction gain finer spatial detail across its decisions?

It enlarges the previous crop repeatedly and discards the original view

It selects regions in sequence and re-encodes selected crops between decisions

It predicts every possible image coordinate in a single flat classification

It generates a coordinate string and parses it into pixels

Core Takeaway

The scorer stays shared; the candidate set and the path of refinement give each spatial answer its shape.