Vela: User Behavior as Supervision

Why People Wanted It

A user history contains evidence about preferences and habits, but simply handing that history to a model does not tell it what to learn or how to score its answer. The practical question is how to turn naturally occurring behavior into tasks with an observable target, without making every useful question depend on a single exact answer. Vela’s move is to divide a history into what the model may see and what remains hidden, then build both prediction tasks and evidence-grounded interpretation tasks around that split.

The Core Move

Treat a chronological behavior history as both context and supervision: earlier events are visible, later events are withheld, and a task combines questions scored either against those later events or against a rubric grounded in the visible record.

Let $H=(e_1,\ldots,e_T)$ be the event sequence and let $t$ be the cutoff. The model receives $H_{<t}=(e_1,\ldots,e_{t-1})$ and answers a task $Q=(q_1,\ldots,q_k)$; the future $H_{\ge t}=(e_t,\ldots,e_T)$ supplies targets for predictive questions. Predictive answers use future behavior as reference, while rubric answers are judged against task-specific criteria.

Why the Two Families Need Different Scores

Now the version for readers who want more of the machinery: the split creates two distinct supervision contracts, not merely two styles of prompt. Predictive questions have an answer source in the hidden future. Rubric questions instead have criteria tied to the visible history or to whether an explanation is supported by that history.

For Objective questions, the prompt includes candidate items, and selected candidates are matched against the future-derived answer set. Open-prediction removes the candidate list; a scorer extracts predicted items, maps semantic equivalents to reference items, and computes set-F1. The candidate list makes the target space explicit, while free-form prediction permits a wider response but requires normalization before scoring.

For Rubric questions, each answer is assessed against criteria with scores in $[0,1]$. If there are $m$ criteria, the question score is $s(a_i)=\frac{1}{m}\sum_{j=1}^{m}c_j$. In words, the score is the equal-weight average of the criterion scores, rather than an exact-match test against one canonical sentence. A task score can then average question scores: $S(Q)=\frac{1}{k}\sum_{i=1}^{k}s(a_i)$. In words, each question contributes equally to the composite task score.

Why Attribution Must Not Give Away Prediction

Here is the part that gets technical: Attribution questions use a future event as something to explain, whereas Open-prediction questions ask the model to infer future events. If both questions expose the same event in one task, the explanation prompt has already leaked the prediction target.

The task synthesis rule therefore avoids selecting both a Predictive template and an Attribution template for the same future event. This preserves the distinction between inferring an outcome from visible evidence and explaining an outcome that has been supplied. Objective and Open-prediction are the two Predictive types; Open and Attribution are the two Rubric types. Open summarizes attributes, interests, or language style from visible history; Attribution explains a supplied future behavior using that visible history.

Why Templates Follow the Available Evidence

Now for the deeper construction constraint: templates are eligible only when their required behavior categories occur in the relevant segment. Histories are divided into segments while keeping each behavior record intact; within a segment, an earlier portion becomes visible and a later portion becomes hidden. Predictive questions are constructed by predefined rules, while Open and Attribution questions use a composer to produce a concrete question and rubric from the permitted evidence.

This means the task is not just a question pasted onto a log. Its type determines which evidence may be used to construct it and how an answer can be judged: future behavior supplies predictive references, while visible evidence grounds summaries and explanations. A separate module covers difficulty filtering and trajectory construction.

Minimal Causal Code

This deliberately plain loop shows the information boundary. The scorer receives future events only for predictive targets; rubric evaluation uses criteria built from the allowed evidence.

for history in histories:
    cutoff = choose_cutoff(history)
    visible = history[:cutoff]
    future = history[cutoff:]

    task = make_questions(visible, future)
    answers = model(task, visible)

    scores = []
    for question, answer in zip(task, answers):
        if question.kind in ("objective", "open_prediction"):
            score = compare_with_future(answer, question, future)
        else:
            score = judge_against_rubric(answer, question.rubric, visible)
        scores.append(score)

    task_score = sum(scores) / len(scores)

What You Gain, and What You Lose

The gain is a supervision scheme that uses the same behavior history for two complementary jobs: predicting observed outcomes and interpreting visible evidence. Predictions have concrete future-derived references; summaries and explanations can be scored without pretending there is only one correct wording.

The tradeoff is that rubric scoring depends on criteria and a scorer, while open prediction needs semantic extraction and matching. The visible/hidden boundary also matters: show the future too early, and prediction stops measuring inference. Finally, answers are only as informative as the available histories and eligible templates; this turns behavior into supervision, not a guarantee that every task is useful.