Qwen/Qwen3.5-122B-A10B
Image-Text-to-Text • 125B • Updated • 524k • • 619
robbie
robb-0
AI & ML interests
i like semiotics and hermeneutics, happens that I train image adapters. 🐶🐕🦊🧸
Recent Activity
liked a model about 23 hours ago
Qwen/Qwen3.5-122B-A10B liked a model about 23 hours ago
Qwen/Qwen3.5-35B-A3B liked a model about 23 hours ago
Qwen/Qwen3.8-27BOrganizations
liked 3
models about 23 hours ago
liked 3
models 13 days ago
upvoted 2
collections 13 days ago
reacted to FlameF0X's post with 🚀 13 days ago
Post
2627
Hello HuggingFace!
I would like to share a small preview of a language model that I have been experiencing for a while. In the first attached image there is a small sample of the Myosotis-1, an attempt to make a small 100m parameter flagship model that is built on my bizarre architecture that is somewhat similar to an S4/S5 model with WKV added. I call it FWKV (Feed-Forward WKV). The model is currently still in training because of the nature of RNN-like models. It can also be seen that the model has insane prompt processing and token generation speed (evaluation done on a 2x Titan XP); even for its small size, some similar Transformer models do struggle to get the same results without custom kernels (some Transformer models can achieve this level of throughput on cheap hardware).
In the second image, you can see checkpoint 20k of the model in its next token prediction state (this means it can't chat), ranking in the top 100 on AxiomicLabs/Open_SLM_Leaderboard (the results have not been submitted since the model is not done training).
- Why not just use Transformers?
Have you seen any pure non-Transformers SLMs besides RWKV and Mamba?
- Should you expect this project to become the next LFM or another very fast language model thing on some Raspberry Pi?
No, the model is still an experiment; it's very sensible and prone to collapse (by the time of this post, it can be seen in the 1st image).
- Should you use it?
Maybe not yet; the architecture itself is still very "naive"—that's how I could call it at its current level. If you just want to play with it and see what you could do or how fast the model is on your hardware, then you can do it.
Once the training is finished and I feel satisfied with the model next token prediction (the base model) and "assistants" (the instruction-tuned model) capabilities, I will make open weights at
FWKV with full support of the 🤗 Transformers.
I would like to share a small preview of a language model that I have been experiencing for a while. In the first attached image there is a small sample of the Myosotis-1, an attempt to make a small 100m parameter flagship model that is built on my bizarre architecture that is somewhat similar to an S4/S5 model with WKV added. I call it FWKV (Feed-Forward WKV). The model is currently still in training because of the nature of RNN-like models. It can also be seen that the model has insane prompt processing and token generation speed (evaluation done on a 2x Titan XP); even for its small size, some similar Transformer models do struggle to get the same results without custom kernels (some Transformer models can achieve this level of throughput on cheap hardware).
In the second image, you can see checkpoint 20k of the model in its next token prediction state (this means it can't chat), ranking in the top 100 on AxiomicLabs/Open_SLM_Leaderboard (the results have not been submitted since the model is not done training).
- Why not just use Transformers?
Have you seen any pure non-Transformers SLMs besides RWKV and Mamba?
- Should you expect this project to become the next LFM or another very fast language model thing on some Raspberry Pi?
No, the model is still an experiment; it's very sensible and prone to collapse (by the time of this post, it can be seen in the 1st image).
- Should you use it?
Maybe not yet; the architecture itself is still very "naive"—that's how I could call it at its current level. If you just want to play with it and see what you could do or how fast the model is on your hardware, then you can do it.
Once the training is finished and I feel satisfied with the model next token prediction (the base model) and "assistants" (the instruction-tuned model) capabilities, I will make open weights at
- 10 replies
reacted to KlondikeDev's post with 🔥 13 days ago
Post
2407
Boris-1.7-D60M-n30M out NOW!
opencerebral/Boris-1.7-D60M-n30M
Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.
Then, Boris-2 will begin training!
opencerebral/Boris-1.7-D60M-n30M
Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.
Then, Boris-2 will begin training!
upvoted 4
collections 13 days ago
liked 3
models 13 days ago
reacted to Banaxi-Tech's post with 🤗 13 days ago
Post
2894
We're excited to release BananaMind 2.1 Pico Preview!
It includes the first preview of our BananaMind 2.1 architecture!
This model gets near BananaMind 2 Micro performance at half the parameters and 37.5x less tokens!
Thats insane!
The current architectures includes about 500K parameters of the total 1.5M parameters in n-gram embeddings and the layer 2 is run twice.
It also includes XSA and the XSA refresh gate.
We're still going to improve the architecture in the final release.
Check it out at:
Follow us for more models:
BananaMind
@Banaxi-Tech
@vovaRL
@DedeProGames
It includes the first preview of our BananaMind 2.1 architecture!
This model gets near BananaMind 2 Micro performance at half the parameters and 37.5x less tokens!
Thats insane!
The current architectures includes about 500K parameters of the total 1.5M parameters in n-gram embeddings and the layer 2 is run twice.
It also includes XSA and the XSA refresh gate.
We're still going to improve the architecture in the final release.
Check it out at:
Follow us for more models:
@Banaxi-Tech
@vovaRL
@DedeProGames
- 3 replies
upvoted 2
collections 13 days ago