Image AI beats LLM-based rivals without language pretraining
Why it matters
Radical Numerics trained Halcyon, a family of open early-fusion autoregressive text-to-image models (0.1B to 4B parameters), entirely from scratch on 100 million image-text pairs, with no language pretraining. In controlled tests against baselines retrained on the same data, Halcyon-1.6B beat the JiT-T2I diffusion model (35.3 vs 41.0 gFD, lower is better) with 275x less total pretraining compute, counting the language model, image tokenizer and generator; Halcyon-0.52B beat LlamaGen (48.0 vs 63.3 gFD) with 18.2x less, and scored 44.91 vs 30.48 on GenEval. Giving 10-50% of the budget to text-only pretraining made images worse (gFD 48.0 to 51.6-65.1), and starting from a pretrained language model gave no gain. The same models also learn image captioning at no extra pretraining cost. Ten Halcyon checkpoints (5 sizes, 2 variants each) are on Hugging Face. Caveat: results are at 256x256 in the authors' own setup.
Read the source: Radical Numerics (opens in a new tab)
All hits from 17 Oct 2026More ResearchGet the daily hits by email