Dense Model

A dense model uses all of its parameters for every token. It is the classic architecture, as opposed to a mixture of experts.

Dense models are often smaller and more predictable in behaviour. In our benchmark the dense 27B beat much larger models on tool use, strategy and factual drawing.