paper-9be54172

The short version

The problem

Many image generators depend on enormous amounts of text-only training before they can use written prompts. That makes them costly to build, and it is not clear that this extra language training actually helps them make better images.

The idea

Halcyon learns from scratch using examples that pair an image with its caption, rather than first studying a huge collection of text on its own. It reads and produces words and image pieces in one sequence; for example, it can learn to make an image from a caption and to describe an image using the same model. The paper also tests a way to teach image description without giving up the image-generation training budget.

What it got

A 0.52-billion-parameter Halcyon model scored 48.0 on a quality measure where lower is better, beating a larger comparison system’s 63.3 while using 18.2 times less total training compute. The experiments also found that text-only pretraining did not improve image generation in the tested setups.