Mixtral of experts | Mistral AI

Mixtral 8x7B is a sparse mixture-of-experts model with open weights, released under the Apache 2.0 license.

Architecture: decoder-only SMoE with 8 expert groups; each token routes to two experts; 46.7B total parameters but 12.9B used per token.

Capabilities: 32k-token context, multilingual support (English, French, Italian, German, Spanish), strong code generation, and finetunable to an instruction-following model scoring 8.3 on MT-Bench.

Performance: matches or outperforms Llama 2 70B and GPT3.5 on most benchmarks; compared to Llama 2 70B it runs about 6x faster in inference.

Pretraining used data from the open Web, with experts and routers trained simultaneously.