Mamba

Mamba is a neural network architecture based on selective state space models. It was proposed as an efficient alternative to the Transformer for sequence modeling, especially where long inputs make full attention expensive.

Transformers use self-attention to compare tokens across a sequence. That gives strong context handling but can be costly as sequence length grows. Mamba instead uses a state-space mechanism with input-dependent selection, allowing sequence processing with more favorable scaling characteristics.

In language modeling, Mamba is relevant because it explores a different architecture path for Large Language Models (LLMs). It is not simply a smaller transformer or a Mixture of Experts (MoE). It changes the sequence-processing primitive itself.

Important tradeoffs include:

• better efficiency on some long-sequence workloads;
• different hardware kernels and implementation requirements;
• less ecosystem maturity than transformer models;
• different behavior on tasks that rely on exact token-to-token interactions; and
• evolving evidence about where state-space models match or fall short of attention.

Mamba should therefore be treated as an architecture family, not as a drop-in guarantee of better LLM performance. Benchmark results depend on model size, data, training recipe, and workload.

The original paper, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, introduces the architecture and its selective state-space layer.