OBIT
← ALL RESEARCH
DREAMINGGOOSE · ABSTRACT + INTERIM RESULTS

Can pretraining compute
survive the rewrite?

Pretraining an autoregressive Transformer takes trillions of FLOPs. We want to know how much of that training can be carried over when we change both the model's architecture and the way it generates text.

PRIOR WORK

Dream (Ye et al., 2025) and DiffuLLaMA (Gong et al., 2025) adapt pretrained autoregressive Transformers into bidirectional diffusion LMs, but keep the Transformer backbone. RADLADS (Goldstein et al., 2025) distills attention into RAD-RWKV7 recurrence, but preserves next-token behavior.

DreamingGoose changes both the architecture and the generation objective, one stage at a time. It distills an autoregressive Transformer teacher into DualGoose, a bidirectional recurrent diffusion model with no attention.

CONVERSION STEPS
1
Distill. We use RADLADS-style distillation to train a causal RAD-RWKV7 student to match the attention teacher's hidden states and logits.
2
Add a backward stream. We initialize it from mirrored forward weights, then train a timestep-conditioned merge while keeping the forward stream frozen.
3
Adapt to diffusion. We continue pretraining with a masked discrete diffusion objective.
Changing the architecture and objective in separate stages lets us see where capabilities are lost.
CONVERSION RESULTS

We tested models at four sizes, from 135M to 8B parameters, using rented machines with one or two GPUs.

SmolLM2-135M
The full pipeline runs on a single 3090, including distillation, bidirectional conversion, and diffusion adaptation. In stage-1, KL falls from 2311 to 550. The fidelity adjustments used at larger scales don't help at this size. The token budget has the largest effect.
0.716 AGREEMENT
Qwen3-0.6B
The same pipeline runs unchanged on a single 3090. With 0.5M training tokens, agreement with the larger teacher reaches 0.568. It rises to 0.633 at 1.1M tokens, suggesting that the token budget limits the conversion at this scale. This reproduces the RADLADS token-scaling result in a small model.
0.633 AGREEMENT
Qwen3-1.7B
We converted four recurrent backbones with roughly 30M tokens each. Agreement ranges from 0.727 to 0.761, with perplexity retention from 0.85 to 0.93.
0.761 AGREEMENT
Qwen3-8B
At 4.7x the parameter count, agreement reaches 0.779 and perplexity retention is 0.873. Student perplexity is 26.0, compared with the teacher's 22.7.
0.779 AGREEMENT
Agreement measures how often the student and teacher choose the same top-1 token on held-out text. The larger matched conversions reach 0.73 to 0.78. These interim figures come from our internal run records and have not been published or peer-reviewed.
REMAINING EVALUATION

The conversion runs span 135M through 8B parameters on rented hardware. At roughly 30M distillation tokens, the larger models recover substantial agreement with their autoregressive teachers.

We still need to establish which capabilities survive the switch to diffusion. We're rebuilding that evaluation before publishing the paper.

The target architecture is described in DualGoose →