GroundingPI’s Iterative Data Engine
Grounding supervision can be incomplete or contradictory; GroundingPI combines complementary teachers field by field, checks each task’s required evidence, and iteratively refines unresolved annotations.
Read the paper ↗What it is
- Field-by-field fusion Combine teachers’ annotations by separating what they say about an object from where they place it, so a disagreement in one part does not erase every useful part.
- Task-specific checks Check only the evidence a particular query requires, and require a completeness check when the query asks for every matching instance.
- Iterative refinement Use an expert’s predictions and focused new observations to resolve uncertain labels, then update labels that depend on the revised evidence.
Why it matters
Mixed annotations are not automatically reliable just because several systems agree. Separating fields, checking query-specific requirements, and revisiting unresolved cases lets the data engine preserve useful supervision while withholding incomplete answers.
Before getting into the machinery, here’s the short version: GroundingPI treats annotation as a collection of claims, not one indivisible label. It combines useful pieces from different teachers, checks whether the particular task has enough evidence, and sends unresolved cases back through an expert-and-observation loop.
-
Complementary evidence: Different teachers can contribute different useful fields, such as an object’s identity or its geometry. The canonical example is a category-grounding query where one prediction gets the box right but another is more reliable about which object it is. So rather than letting one disagreement invalidate everything, the engine keeps verified pieces and investigates the disputed field. That still does not mean agreeing teachers are automatically correct.
-
The query sets the bar: A box may be enough for one request and insufficient for another. The canonical example is “find every instance”: a few good boxes do not establish that the whole relevant region was checked. GroundingPI marks fields accepted, rejected, or unresolved, and only accepts an item when all fields required by that query are accepted. For an all-instances request, it also checks coverage. An empty prediction does not prove absence; absence is itself something to verify. What can be reused when a broader query is blocked by one unresolved region?
-
Evidence can revise the record: Accepted annotations train a grounding expert, whose predictions go back through the same fusion and checks. When identity, geometry, or coverage is still unclear, complementary teachers or a targeted local observation can help. Think of zooming in on the one part of a notebook page you could not read; the limit is that a crop only helps if its result can be related back to the original image. Crop predictions are mapped to original coordinates, and a needed local view is kept when the original training input lacked that detail. New evidence can add labels or change old ones, with dependent labels revised or withdrawn accordingly. Which unresolved field is worth asking the next observation to settle?
I like the discipline of this design: it does not confuse “some useful supervision exists” with “this entire query is answered.” The tradeoff is that requirements, status, coverage, and dependencies all need explicit bookkeeping.
Read the deep dive
Why People Wanted It
A grounding label is not just a box. Depending on the query, you may also need the right identity, a mask, text, attributes, or evidence that every relevant instance has been found. Different annotation systems can contribute useful but non-interchangeable pieces, and agreement alone does not prove a label is right. The real question is how to turn mixed predictions into supervision you can trust without throwing away useful partial evidence.
The Core Move
Fuse complementary annotations one field at a time, accept a query only when its required fields and coverage checks pass, then use an expert and targeted observations to repair what remains unresolved.
An annotation is treated as a collection of semantic and geometric fields. Each field is accepted, rejected, or unresolved; a supervision item is accepted only when all fields required by its query are accepted.
Why Field-Level Fusion Keeps Useful Evidence
Now the version for readers who want more of the machinery: candidates first have to refer to the same query scope and annotation granularity, with coordinates mapped to the original image. Fusion then handles semantic and geometric fields separately: retain verified information, but seek more evidence for the fields that disagree.
This separation matters because a correct location does not establish a correct identity, and a matching category does not establish correct geometry. Nor should every overlap be collapsed: object–part containment and word–line relations are preserved as relations rather than treated as duplicate instances. Field-level agreement is evidence to consider, not a guarantee of correctness.
Why Validation Depends on the Query
Here is the part that gets technical: validation is defined by what the task asks for, not by whichever fields a teacher happened to emit. For a query with required-field set , accept the item only if every required field is accepted:
In words, one missing or unresolved required field blocks that item from being accepted. Geometry, syntax, semantic correspondence, and consistency with available evidence are checked. If the query asks for all instances, it also needs coverage over the relevant region. An empty result is not proof that a target is absent: absence itself must be verified. An unresolved instance can block an all-target or counting query while leaving independently verified local supervision usable.
Why Expert Iteration Can Change Old Labels
Now for the deeper loop: accepted annotations train a unified grounding expert, and its predictions return through the same fusion and validation process. When identity, geometry, or coverage is still unresolved, complementary teachers or targeted local observations can provide evidence for that particular gap.
A crop’s predictions are mapped back to original-image coordinates. If the detail needed for supervision was not available in the original training input, the necessary local view is retained. New evidence can add an annotation or revise an existing one; revisions propagate to dependent tasks, and invalid labels are withdrawn until their dependencies are resolved. What remains difficult is deciding which unresolved field deserves the next observation.
Minimal Causal Code
A plain loop makes the acceptance gate visible. This sketches required-field validation; it does not implement teacher fusion or evidence collection.
def accept_item(fields, required_fields):
# fields maps a field name to "accepted", "rejected", or "unresolved"
for name in required_fields:
if fields.get(name) != "accepted":
return False
return True
for round_id in range(num_rounds):
candidates = collect_teacher_candidates(images, queries)
fused = fuse_fields(candidates)
for item in fused:
item.status = validate_fields(item)
item.accepted = accept_item(item.status, item.required_fields)
if item.requires_all_instances:
item.accepted = item.accepted and coverage_passes(item)
expert.train([item for item in fused if item.accepted])
unresolved = [item for item in fused if not item.accepted]
observations = request_targeted_evidence(unresolved, expert)
revise_and_propagate(fused, observations)
What You Gain, and What You Lose
You gain a way to preserve trustworthy pieces of annotations without pretending that one teacher or one all-or-nothing label is enough. Query-specific required fields and coverage checks make it possible to distinguish usable local supervision from an incomplete answer to an all-instances query. Iteration can also correct dependent labels, not only add new ones.
You pay for this with more bookkeeping: fields need explicit statuses, queries need their own requirements, and revisions must propagate to dependent tasks. Teacher agreement can still be wrong, and unresolved cases call for extra evidence rather than automatic acceptance. The engine is therefore a process for managing uncertainty, not a guarantee that every annotation is correct.
Trustworthy grounding supervision is built field by field: keep what is verified, demand evidence for what is not, and revisit labels when new evidence changes their dependencies.