Reusable Video-Prefix Inference

Repeated video questions can repeat expensive video processing; VideoJev reuses one causal video prefix either through independent cache branches or a single joint pass over multiple queries.

Read the paper ↗

The short version

What it is

Cached inference Encode the video once, then answer each incoming question from its own branch of the saved video context.
Joint inference Place several questions after one video context and produce a separate decision for each during one causal pass.
Shared-prefix training Train with multiple question turns after a common video context, so the video work can be shared across questions.

Why it matters

Repeated questions do not require rebuilding the video context each time. The choice is whether to keep each question isolated in its own branch or process a batch of questions together, where later questions can see earlier question text.

Before getting into the mechanics, here is the short version: VideoJev can keep a video’s computed context around and reuse it for follow-up questions. If questions are handled one at a time, each gets an independent branch; if several are available together, they can be placed after one shared video prefix and read in one causal pass.

Repeated video work: The canonical example is asking several questions about the same clip. The video itself has not changed, so rebuilding its representation for every question repeats work that does not depend on the wording of the question. The useful boundary is simple: reuse the video-side context, but still process each question.
Independent answers: Think of the saved video context as a notebook copy. For each question, start from the same copy rather than carrying the previous question’s answer into the next one. That keeps the branches independent; the analogy stops at the fact that a model cache is internal activations, not a human-readable notebook.

So if each branch starts from the same saved context, what exactly does it still need to compute? In cached inference, the model processes that question and its readout on a forward pass, while the video prefix is prefetched only once. The saved prefix is shared, not the query’s computation.

Queries together: When several questions are available at once, put their turns after one video prefix and read a separate decision position for each. This is like keeping several questions in one ordered document: because processing is causal, a later question can see earlier question text and its candidates. The answer slots stay empty, so answers are not passed along as history. I like this distinction because “one pass” can sound like “independent,” but the later turns do have access to earlier query wording.

So the real question is not only whether computation is shared; it is what information the later turns can see. Joint inference shares the video prefix and runs all readouts together, while cached inference branches independently from that prefix. Training uses the concatenated query layout too, sharing video computation across questions. For outputs that need sequential token feedback, the described approach uses cached inference. One related task formulation uses direct candidate scoring, but the key idea here is reuse of the video prefix.

Read the deep dive

Why People Wanted It

If you ask several questions about one video, why should the model rebuild the video context each time? The expensive part—the video representation—does not depend on which question comes next. The real question is how to reuse that prefix without accidentally letting one question’s answer become another question’s evidence.

The Core Move

Compute the video prefix once, then either branch independent queries from its cache or append multiple queries after it and read each query’s decision separately. In both cases, causal order makes the video prefix available to later text without adding answer history.

Why Cached Queries Stay Independent

Now the version for readers who want more of the machinery: causal attention lets each query suffix read the prefetched video prefix, while a fresh branch prevents it from reading other queries or their generated tokens. Write the prefix cache as $C_V$ and a query suffix as $Q_i$ ; the model’s query-specific readout is $f(C_V, Q_i)$ . The cache is computed once, but each branch still runs a forward over its own query and readout tokens. This is reuse of video computation, not reuse of the query computation.

Why Joint Queries Share One Causal Pass

Here is the part that gets technical: place query turns in order after the video prefix, with a separate empty answer/readout position for each turn. The causal mask gives turn $i$ access to the video and preceding query text, but not future turns. At readout positions, the model produces each query’s score from the same forward pass. No reference or predicted answer is appended, so later queries may see earlier query wording and candidates, but not an earlier answer. The shared prefix removes repeated video processing; it does not make query turns mutually isolated.

Why Training Uses the Same Layout

For the deeper training connection: concatenating question turns after one video prefix lets training share the video computation across questions using the same arrangement used at joint inference. The supervision remains attached to each question’s own readout. This makes the training layout align with the joint path, while cached inference remains useful when queries arrive sequentially or when outputs need token-by-token feedback.

Minimal Causal Code

# Conceptual loop: cache the video once, then branch each query independently.
video_cache = model.prefill(video_tokens)

for query_tokens in queries:
    branch_cache = copy_cache(video_cache)
    logits = model.forward(query_tokens, cache=branch_cache)
    scores = read_query_scores(logits)
    save(scores)

The joint path instead concatenates all query turns after the video prefix and reads the separate answer positions from one causal forward.

What You Gain, and What You Lose

Cached inference avoids re-encoding and prefilling the video for every query, but it still performs a query-suffix forward per question. Its branches preserve independence. Joint inference produces all readouts in one causal forward, but later questions can condition on earlier query text and candidate lists. Neither mode inserts answers as shared history. For outputs that require sequential token feedback, the described inference protocol uses cached branches rather than the joint readout path.

Quiz

What is the key distinction between cached inference and joint inference for multiple questions about one video?

Cached inference shares earlier answers between queries; joint inference prevents later queries from seeing earlier query text.

Cached inference recomputes the video for every query; joint inference stores each answer in the prefix for later questions.

Cached inference branches each query independently from a shared video cache; joint inference places multiple queries after one video prefix and reads their decisions in one causal pass.

Core Takeaway

Reuse the video prefix, but choose whether questions should branch independently or share one causal pass.