A Shared Coordinate-Token Interface for Visual Grounding

Why People Wanted It

A model can recognize what you mean and still leave you with an awkward answer: a category name in one format, a point in another, and a box in a third. The practical question is whether language-conditioned perception can return both what an image region means and where it is, using a representation that works across tasks. GroundingPI's answer is a shared output vocabulary: ordinary language labels sit beside structural markers and coordinate tokens.

The Core Move

Represent each grounded result as a semantic label paired with a delimited geometric payload, and generate the whole response autoregressively. The paper describes the token roles and coordinate encoding, but gives no compact standalone formula for this interface.

A category can be paired with a box, a referring expression with its target's box, or recognized text with the box around that text. Pointing uses the same structure with a point payload; an explicitly queried category with no match keeps its label and carries the ordinary text None.

Why Labels and Geometry Share One Output Vocabulary

Now the version for readers who want more of the machinery: the shared vocabulary has three distinct jobs, not one generic “grounding token.” A label or transcription is ordinary language text bracketed by <|object_ref_start|> and <|object_ref_end|>. A geometric payload is bracketed by <|box_start|> and <|box_end|>. Atomic coordinate tokens encode the geometry, while punctuation separates instances and entries.

The distinction matters: the wrapper says what kind of field follows, the label gives its semantic identity, and the payload gives its location. The same shape can bind OPEN to a text region or cup to one or more boxes. What changes across tasks is the query and the payload's geometry, not the basic label-plus-payload interface. How can different geometric answers fit the same wrapper?

Why Points and Boxes Fit the Same Payload

Here is the part that gets technical: a box is represented by four consecutive coordinate tokens in xyxy order, while a point uses two tokens in (x, y) order. Multiple instances under one label share a wrapper, with coordinate tuples separated by commas; distinct entries are separated by a comma and a space. The closing <|box_end|> ends a payload, while <|im_end|> ends the assistant response.

Coordinates are normalized relative to image width and height, quantized onto the intermediate integer grid from 0 through 1000, then encoded using the corresponding atomic coordinate token from <0> through <999>. On recovery, the quantized value is divided by 999 and multiplied by the relevant image dimension. Source xywh boxes are converted to xyxy before normalization; polygon annotations become their axis-aligned enclosing boxes. Invalid, out-of-range, inverted, or degenerate geometry is rejected rather than silently repaired. What does this shared structure buy when the task itself changes?

Why One Protocol Covers Different Grounding Queries

Now the deeper interface constraint: the query specifies both the task and its semantic targets, and the response associates each label, referring expression, or OCR transcription with geometric output. Category grounding and layout return boxes; pointing and GUI targeting return points; OCR binds recognized text to boxes; visual prompting supplies example boxes in the same coordinate vocabulary used for output. The protocol also preserves explicitly requested category strings, including an absent category represented with None.

This is a common output protocol, not a claim that every task has identical prompts or geometry. Its leverage is that the decoder can express different language-conditioned grounding requests as structured language generation. Which task-specific distinctions remain outside the shared representation?

Minimal Causal Code

This toy loop exposes the protocol's structure; it is not a tokenizer or a complete model implementation.

# A box entry: semantic label, then four coordinate tokens.
label = "cup"
boxes = [(10, 20, 30, 40), (50, 60, 70, 80)]

response = ["<|object_ref_start|>", label, "<|object_ref_end|>"]
response.append("<|box_start|>")
for i, box in enumerate(boxes):
    if i:
        response.append(",")
    response.extend(f"<{coord}>" for coord in box)
response.append("<|box_end|>")

What You Gain, and What You Lose

A shared protocol makes labels and locations explicit in one generated response, and lets box prediction, point prediction, OCR, layout, GUI targeting, referring expressions, and visual-example search reuse the same broad structure. It also makes output parsing depend on clear wrappers, coordinate conventions, and task prompts; a malformed or invalid geometric payload is not repaired silently. The representation does not erase task differences: a box is not a point, and the prompt still has to say what kind of grounding is wanted.

Quiz

In GroundingPI's shared output protocol, what distinguishes a point payload from a box payload?

A point is represented by the label alone; a box includes coordinates.

Points use a separate output wrapper from boxes.

A point uses ordinary text, while a box uses coordinate tokens.

A point has two coordinate tokens; a box has four in xyxy order.

$ check