Mixture of experts (MoE) is an architecture where many specialist “expert” networks sit behind a router that activates only a subset per token. That lets models grow total parameter count while keeping per-token compute closer to a smaller dense model — with routing and load-balancing tradeoffs.
Dense models use every parameter on every token; MoE models wake only a few experts, aiming for more capacity per watt.
Open and closed labs both ship MoE variants; comparing them requires knowing active parameters, not just total parameters on a slide.
When Latent covers model cards and leaks, MoE routing details often explain why two “similar-sized” models feel different in production.
They can advertise huge parameter counts and capacity while controlling inference cost relative to an equally large dense model.
Routing instability, expert imbalance, and harder debugging — a wrong expert path can look like random quality regressions.