Does anyone know if there were any attempts to test Mamba on really large scale? To me this model looks as the most promising successor to the transformer architecture. Does anyone know why it might not be the case or what are other alternatives?
Interesting direction for research but not a model you’d want to use today. The paper looks at a 3b model built on llama3.2-3b, modified for mamba, and they’re comparing to a distilled version of r1 with 1.5b params.
Does anyone know if there were any attempts to test Mamba on really large scale? To me this model looks as the most promising successor to the transformer architecture. Does anyone know why it might not be the case or what are other alternatives?
Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Large Model: https://news.ycombinator.com/item?id=43447254
Interesting direction for research but not a model you’d want to use today. The paper looks at a 3b model built on llama3.2-3b, modified for mamba, and they’re comparing to a distilled version of r1 with 1.5b params.