We optimized a reasoning model to run on a consumer 4090 (24M params). Beat the SOTA by 3x.
Hey everyone,
I wanted to share a repo we (Bitterbot AI) just open-sourced that challenges the "bigger is better" trend in AI engineering.
Current reasoning models (for tasks like ARC-AGI-2) usually require massive compute clusters. They suffer from "drift"—they forget the algorithm while trying to execute the pixels.
Our Engineering Fix: We built a Dual-Stream architecture (TOPAS-DSPL). Instead of one monolithic transformer, we split the compute into two streams:
Logic Stream: Handles the algorithmic planning (The CPU).
Canvas Stream: Handles the state updates (The GPU).
The Results:
Size: Only 24M parameters (Tiny).
Performance: 24% on the ARC-AGI-2 Eval set (vs ~8% for standard architectures of this size).
Hardware: The whole training pipeline runs on a single RTX 4090.
We’ve released the full stack—Data Augmentation, Training Loop (with MuonClip optimization), and the TTT (Test-Time Training) Evaluator.
Repo here: https://github.com/Bitterbot-AI/topas_DSLPv1
If you have a 4090 and want to test the inference speed or the TTT pipeline, I’d love to hear your results. We're trying to see how far we can push efficient architectures before scaling up.
Very cool — squeezing 24M-parameter reasoning models onto a 4090 with a 3× improvement over SOTA is impressive.
In practice, I’ve found that the real cost isn’t just raw throughput but how often performance wins hold up under real workloads (mixed batch sizes, varied input lengths, and memory fragmentation).
Curious — when you optimized for the 4090, what was the biggest bottleneck you had to tackle first — memory bandwidth saturation, kernel launch overhead, precision trade-offs (FP16/INT8), or something else? And did that bottleneck shift when you moved from micro-benchmarks to end-to-end reasoning pipelines?