Joint Text-and-Image Autoregressive Modeling from Scratch
Separate language and image pipelines make text conditioning depend on pretrained components; Halcyon trains tokenizers and a decoder-only model from scratch to predict both modalities in one sequence.
Read the paper ↗What it is
- One shared sequence Halcyon places text and discrete image pieces in one vocabulary and trains a decoder to predict whichever piece comes next.
- Order sets the task Put a caption before image pieces to generate from text, or put image pieces before a caption to describe an image.
- Balanced prediction Training scores text and image predictions separately, then gives the image score a smaller weight so it does not dominate.
Why it matters
The model learns caption understanding and image generation together from scratch, without relying on a separately pretrained language model. A common sequence format supports both directions, while token ordering and loss weighting determine which behavior is trained.
Before getting into the mechanics, here is the short version: Halcyon trains one decoder to continue sequences containing either text pieces or image pieces. The same model can learn to make an image from a caption or write a caption after seeing image tokens, without starting from a separately pretrained language model.
- Language-model dependence: A pretrained text model can help interpret captions, but its text-only training may be much larger than the paired image-caption supervision used for generation. Halcyon tests whether the model can learn the needed text behavior from paired captions instead.
- A common interface: Separate tokenizers still produce different kinds of tokens, but one decoder needs a common way to consume and predict them. The key move is to put both kinds in a shared vocabulary and let sequence order tell the model what it is doing.
- More than one direction: The same paired example can teach generation or description. That is useful, but it means the training sequence has to make the direction unambiguous.
The canonical concrete example is one 256-by-256 image and its caption. The image tokenizer turns the image into a grid of discrete pieces; the text tokenizer turns the caption into text pieces. Think of the sequence as a notebook with typed entries: some entries are words, some are image codes, and markers say where an image starts and stops. The analogy breaks because these are learned token IDs, not human-readable notes.
Why does one model know which kind of prediction to make? The sequence itself answers. If the caption comes first and the image marker follows, the decoder is being asked to continue by generating image tokens. If image tokens come first and caption tokens follow, it is asked to describe the image. In other words, changing the order changes the conditioning context; the model still does next-token prediction.
The image tokenizer compresses spatially: each image dimension is reduced by a factor of 8, then the resulting grid is flattened in raster order. For the 256-by-256 example, that gives 1,024 image tokens. Text and image token IDs occupy separate ranges within one joint vocabulary, with special markers for image boundaries, sequence endings, and padding.
So the real question was never just how to fit two modalities into one decoder — it was how to stop one kind of token from swamping the other during learning. Halcyon sums text-token prediction loss and image-token prediction loss, then gives the image part a smaller weight. In mechanism terms, this is like turning down one stream's contribution to the shared training signal; it does not make image predictions less important at inference. The image loss weight is 0.5.
For image generation without a caption, training sometimes removes the caption from text-to-image examples. At generation time, classifier-free guidance combines predictions made with and without the caption, moving the unconditional prediction toward the caption-conditioned one. This is a steering knob, not a guarantee that every prompt is followed perfectly. Image-to-text examples also give the model image prefixes without caption conditioning, so the model encounters that unconditional image context. A separate module covers the detailed role of image-to-text supervision in the training budget.
I like the clarity of this setup: one decoder, and the sequence tells you what is being conditioned on what. But sharing the model does not make text and images the same problem; the tokenizers, task ordering, and loss balance still matter. How well this works depends on the training mix as much as on the shared architecture.
Read the deep dive
Why People Wanted It
Text-to-image systems often borrow a pretrained language model to interpret captions. That can help with text conditioning, but it also means language pretraining may consume a much larger token budget than the paired image-caption data used for generation. Halcyon asks a narrower question: can one model learn to read captions and generate images using only paired caption supervision, without a separately pretrained language model?
The answer starts with a shared sequence, not a text model attached to an image model. The text tokenizer, image tokenizer, and multimodal decoder are all trained from scratch.
The Core Move
Treat text and image tokens as members of one vocabulary, then train a decoder-only Transformer to predict the next token in their shared sequence. The training objective is weighted next-token cross-entropy: text-token loss plus a scaled image-token loss, normalized by the number of non-padding text and image tokens.
Why One Vocabulary Can Serve Two Modalities
Now the version for readers who want more of the machinery: the shared vocabulary is made by combining three token groups, while retaining a single embedding table and output head. Text uses a byte-level BPE tokenizer trained on captions and OpenWebText; images use a discrete tokenizer trained on the image data, based on an encoder-decoder design with finite scalar quantization.
The image tokenizer downsamples each spatial dimension by a factor of 8. For a 256 × 256 image, its 32 × 32 grid is flattened in raster order into 1,024 image tokens. The image codebook has size 64,000. The text vocabulary has 16,384 tokens; adding 256 special tokens yields a joint vocabulary of 80,640. These are the paper's stated construction details, not requirements of autoregressive modeling in general.
Special markers tell the decoder where image content begins and ends: [SOV] and [EOV] bracket image tokens, [EOS] ends a sequence, and [PAD] marks padding. Padding positions do not contribute to the loss.
Why Sequence Order Defines the Task
Here is the part that gets technical: all four formats are ordinary next-token sequences, but their ordering changes what the model must condition on. Text-only data is packed as text followed by [EOS]. An image-only sequence starts at [SOV], then contains image tokens and [EOV]. In image-to-text, the image comes first and the caption follows it; in text-to-image, the caption comes first and the image follows it.
That ordering makes the conditioning direction explicit. Image-to-text asks the decoder to continue from an image prefix into caption tokens; text-to-image asks it to continue from a caption into image tokens. Image-containing sequences are padded to a fixed length, and multiple image-caption pairs are not packed together. The formats share machinery, but not the same prediction problem. What should happen when the model needs an unconditional image branch?
Why The Loss Balances Token Types
Now for the objective's precise bookkeeping: the text loss sums next-token log probabilities over non-padding text tokens, including special tokens; the image loss does the same over non-padding image tokens. The total is their sum after multiplying the image loss by a weight, then dividing by the total count of those text and image tokens. The image-loss weight is set to 0.5, to keep image tokens from dominating the training loss.
Classifier-free guidance adds an unconditional route by dropping the caption from a fraction of text-to-image examples during training. At inference, the conditional and unconditional image-token logits are combined as , where is the guidance scale. In words: start from unconditional logits, then move toward the caption-conditioned logits by the guidance scale. Image-to-text examples also expose the model to an image prefix without caption conditioning, so that prefix trains the unconditional image branch. The exact weighted-loss and guidance expressions are stated here because they are given in the paper; the mechanism is the shared next-token setup plus explicit balancing and conditioning. What does that shared setup buy, and what does it make harder?
Minimal Causal Code
This plain loop shows the causal update: each position predicts the next token, and padding is excluded from the loss. It leaves out the paper's separate text/image loss weighting so the sequence-level prediction is easy to see.
import torch
import torch.nn.functional as F
# tokens: [batch, sequence], containing text, image, and special token IDs
# model returns next-token logits at every position
logits = model(tokens[:, :-1])
targets = tokens[:, 1:]
loss_sum = 0.0
valid_count = 0
for position in range(targets.shape[1]):
valid = targets[:, position] != PAD_ID
position_loss = F.cross_entropy(
logits[:, position], targets[:, position], reduction="none"
)
loss_sum = loss_sum + (position_loss * valid).sum()
valid_count = valid_count + valid.sum()
loss = loss_sum / valid_count
loss.backward()
optimizer.step()
optimizer.zero_grad()
What You Gain, and What You Lose
A shared decoder and vocabulary let one model learn text continuation, image continuation, and cross-modal conditioning without a separately pretrained language model. Training from scratch also makes the source of language supervision explicit: captions paired with images, rather than a large text-only pretraining stage.
The tradeoff is that the model has to learn useful text representations and image generation together from the available training mixture. The shared output space does not make the modalities interchangeable: image tokens and text tokens have different roles, and the objective needs an explicit weight to balance their contributions. Nor does a unified sequence eliminate design choices; tokenizers, sequence order, padding, and the conditional/unconditional mix all shape what the model learns.
Halcyon makes the conditioning direction a sequence-order choice: put the caption before the image to generate, or the image before the caption to describe.