Introducing DWARF-55M-Base

Explain Like I'm Normal

DWARF is a new model architecture that ditches traditional heavy attention mechanisms for a 'Dynamic Sparse Query-Gather' approach. It uses only one full attention layer mixed with several sparse layers, claiming to maintain high-quality retrieval even at triple its trained context window. This could significantly lower the compute costs for training and running small but capable local models.