Sparse Mixture-of-Experts

SPARSE MIXTURE-OF-EXPERTS ARCHITECTURE 100 experts. 3 active per token. INACTIVE: 968B parameters ACTIVE: 32B parameters total: 1,000,000,000,000 active per token: 32,000,000,000 --- The architecture that lets a trillion-parameter model run on reasonable hardware. Most of it is asleep. The rest is doing your bidding. Bella+Canvas 3001 unisex jersey tee. 100% combed and ring-spun cotton. Blueprint schematic printed front-center.