OBIT
← ALL RESEARCH
DUALGOOSE · ABSTRACT

Bidirectional recurrence,
without attention.

Diffusion language models generate text by repeatedly refining a masked sequence. Each prediction can use context from both sides of a token. Most of these models use Transformer denoisers, so they still pay the quadratic cost of self-attention.

BACKGROUND

DiffuMamba (Singh et al., 2025) showed that Mamba-based, linear-time scans can work as diffusion denoisers. Other recurrent architectures have received less attention.

RWKV-7's rank-1 state update is a strictly more expressive recurrence than Mamba-2's diagonal decay. We want to test whether that difference helps a denoiser combine context from both directions.

ARCHITECTURE
Paired streams
We process each noisy sequence in both directions using RAD-RWKV7, a variant of RWKV-7 designed to replace attention with recurrence.
A learned merge
A learned merge combines the two states into a representation of the full sequence. We train it with a masked discrete diffusion objective.
Timestep conditioning
The merge depends on the diffusion timestep. When many tokens are masked, it concentrates on gathering context. As fewer masks remain, it refines that information. This lets the merge change during denoising instead of simply concatenating the two streams.
EVALUATION

We analyze how much the paired forward and backward states can represent, then compare small-to-mid-scale models against Transformer, bidirectional Mamba-2 (DiffuMamba and DiffuMamba-H), causal-only, and sparse-attention-hybrid baselines. We measure denoising perplexity, sampling efficiency, scaling with sequence length, and retrieval using MQAR and S-NIAH probes.

We want to find out whether RWKV-7 can denoise as well as these alternatives at the same parameter count. This work also supports our effort to convert autoregressive Transformers into diffusion models without attention. We aren't trying to build a frontier model in this evaluation.

We study that conversion in DreamingGoose →

The abstract and architecture are complete. Evaluation is in progress, and we'll publish the full paper with the results.