VideoJev: Shared Candidate Scoring for Video Decisions

Why People Wanted It

A video question can ask for a word-sized answer—what was added?—or a time-sized answer—when did the event happen? A generative video-language model can express either answer as text, but these tasks need structured decisions, not necessarily an unconstrained response.

Both tasks ask the model to judge candidates against a video and a query. The real question was whether one scoring mechanism could make those judgments while leaving the final decision rule to the task.

The Core Move

Represent each possible answer or time interval as a candidate, score it against the video-and-query context with one shared head, and change how the scores are interpreted by task:

$s_i = 10\,u^\top v_i + b$

Here, $u$ is the normalized query-level decision representation and $v_i$ is the normalized representation for candidate $i$. The score is their scaled similarity plus a learned scalar bias. Written options compete for selection; temporal intervals receive membership judgments that are combined into a span.

Quiz

VideoJev uses the same candidate scorer for timestamp intervals and written answer options. What changes between these two tasks?

The shared head predicts event boundaries directly instead of scoring candidates.

The output rule: sigmoid-based interval membership and span decoding for grounding, versus softmax ranking for answer options.

The video encoder is replaced by a separate model for each task.

Temporal candidates are free-form generated answers, while text options are represented as timestamps.

$ check

Why Candidate Readouts Can Share a Scorer

Now the version for readers who want more of the machinery: sharing the head requires a common readout interface, not identical candidate contents.

The final MLP is removed from the causal language backbone, and a candidate scorer replaces the language-model output head. The input places the video first, followed by the task instruction, query, and candidate list. Each candidate ends with an <|option_end|> readout token; the list ends with a <|decide|> readout token.

Separate learned linear projections map the candidate and decision hidden states to 512 dimensions, then normalize each to unit length. Those normalized vectors are $v_i$ and $u$ in the shared scoring equation. The scale is fixed at 10, while the scalar bias is learned.

The causal layout makes the readouts asymmetric. A candidate readout can use the video, query, and preceding candidates. The final decision readout sees the full candidate list. These are contextual representations, not independently embedded answer strings.

Temporal candidates are timestamp texts identifying frame-pair intervals; answer candidates are ordinary text options. For answer selection, the output rule is:

$p_i = \operatorname{softmax}(s)_i$

The logits become probabilities over the supplied options, and the highest-scoring option is selected. But an event can occupy several intervals, so why force its temporal candidates to compete for a single winning slot?

Why Temporal Grounding Needs Membership and Coverage

Here is the part that gets technical: temporal labels depend on the mean timestamp of each frame pair, while coverage supervision also accounts for interval duration.

The backbone groups frames into pairs through temporal compression. Each pair defines a candidate interval. Internal interval boundaries lie halfway between the previous pair’s last frame and the next pair’s first frame; the outer boundaries are the start and duration of the video. Timestamp strings identify candidates, but labels and boundaries use unrounded times.

A candidate is positive when the mean of its two frame timestamps lies in the union of annotated event intervals. This mean need not be the midpoint of the candidate’s boundary-defined interval. Membership probabilities use:

$p_i = \sigma(s_i)$

Each candidate receives an independent sigmoid probability rather than a share of one option distribution. Binary cross-entropy supervises these local judgments over valid candidates.

Local membership alone does not directly reward covering the event as a whole. A duration-weighted soft intersection-over-union loss adds that pressure: its numerator rewards event coverage, and its denominator penalizes predicted coverage outside the event. Duration weighting accounts for unequal interval lengths. The soft-IoU term is omitted when no candidate’s mean frame timestamp is positive.

The two objectives therefore have different jobs: identify relevant pieces and reward event-level coverage. Even with both, the output remains a list of interval scores. How does that become one continuous prediction?

Why Contiguous-Span Decoding Matches the Output

Now for the structured-decoding detail: a valid prediction is a nonempty contiguous range, not an arbitrary collection of independently selected intervals.

The decoder adjusts logits using a probability threshold, then chooses the contiguous range with the largest sum of adjusted logits. The threshold is fixed at $0.25$, with smoothing window 1, in the reported experiments. This makes the decision depend on accumulated support across a range rather than simply retaining every interval above a cutoff.

Kadane’s algorithm finds the maximum-sum range in $O(K)$ time for $K$ candidate intervals. If every adjusted logit is negative, the decoder returns the strongest candidate’s interval. Otherwise, the selected range’s first left boundary and last right boundary define the prediction. Lightweight boundary postprocessing makes the result valid and nonempty.

The scorer supplies local evidence; the decoder imposes the output shape. What flexibility does that separation buy, and what does it constrain?

Minimal Causal Code

This plain PyTorch loop implements only contiguous-span selection. Its input is the decoder’s threshold-adjusted logits in temporal order; threshold adjustment, training losses, and boundary postprocessing are outside this snippet. The running state is deliberately visible.

import torch


def select_span(adjusted_logits, left, right):
    # One adjusted logit and boundary pair per valid interval.
    assert adjusted_logits.ndim == 1
    assert adjusted_logits.numel() > 0

    if torch.all(adjusted_logits < 0).item():
        i = int(torch.argmax(adjusted_logits).item())
        return left[i], right[i]

    running_sum = adjusted_logits.new_zeros(())
    best_sum = adjusted_logits.new_tensor(float("-inf"))
    start = 0
    best_start = best_end = 0

    for i in range(adjusted_logits.numel()):
        value = adjusted_logits[i]
        if running_sum.item() <= 0:
            # Start a new range rather than carry negative support.
            running_sum = value
            start = i
        else:
            # Extend the current contiguous range.
            running_sum = running_sum + value

        if running_sum.item() > best_sum.item():
            best_sum = running_sum
            best_start, best_end = start, i

    return left[best_start], right[best_end]

What You Gain, and What You Lose

A shared candidate scorer lets temporal intervals and written answer options use one interface over multimodal representations. It separates judging candidates from deciding what shape the answer must take. Neither task requires the head to generate an unconstrained textual answer.

The shared interface does not make the tasks identical. Temporal grounding needs interval labels, coverage supervision, and contiguous-span decoding. Multiple-choice answering needs supplied options and normalization across them.

Candidate construction also limits the decision space. Answer selection cannot return an answer outside the supplied list. Grounding initially selects boundaries from its interval partition, followed by boundary postprocessing; it is not unrestricted boundary generation. And a single contiguous prediction cannot directly express several disjoint spans, even though the membership labels can refer to a union of annotated intervals.