Replies: 2 comments 9 replies
|
I might be missing something but the model is relatively large. I have it running on 88GB of VRAM, albeit across RPC nodes and I only get 10-12 t/s with 128K context. And I also get relatively low GPU utilization. You could try shrinking your context to 2048 or 1024 so more of your model fits in VRAM though it should only improve t/s marginally. And less than 32K context is kind of useless outside of benchmarks. But hey, at least you can run it successfully on a relatively new version (7819). :)
|
|
Try reap version 🤷 |
BTW it's already running at 14 threads. @FNsi did try MiniMax-M2.1 REAP 30 and REAP 40. Reap 30 produced 2.4 t/s. Still trying other models. But GLM-4.7-Flash only semi-succeeded my test that I give all models; (Write a fast as possible prime number generator in rust.). |
|
Getting 17 t/s with Qwen3-Next-80B-A3B-Thinking-UD-Q4_K_XL. GPU is utilized between 20-30%, more than previous models. So that is quite decent for me. But it seems to be stuck in a loop.. Here is an extremely fast prime number generator using Sieve-of-Eratosthenes optimized in pure rust without external crates: This is a standard implementation. But to make it faster, we can skip even numbers except 'two' and use bit packing. Actually here's the fastest possible sieve in rust with all optimizations: Wait here's the correct code that works and runs fast: But this is not skipping even numbers. To make it faster, we can skip all evens except 'two' and mark them separately. Here's the fastest possible implementation in rust for generating primes up-to n: Actually, here's the fastest and most efficient code from a known source: It did this until context was full (65k). |
|
Qwen3-Next-80B is solid. I'm using it for non coding tasks and I only get around 39 t/s despite offloading it all to VRAM. So your performance is excellent considering the resources. Aside from that one all other numbers look reasonable given the hardware. If you have the coin to spare, I'd encourage you to get a DGX Spark if AI coding is your focus. The base model of Asus Ascent GX10 with 1TB NVME is only $3K (B&H). Nvidia/OSS community have significantly improved performance on Blackwell and you can get mid 50s t/s for GPT-OSS 120B. You can look for posts from "eugr" on this forum or nvidia for detailed info on benchmark and optimizing performance. The other option to maybe consider is using a cloud service. Personally, I found setting up AWS's Bedrock with AWS Bedrock Gateway reasonably straightforward. For about $0.01 - $0.02 for a few thousand tokens in-out, you can get Kimi K2 to augment your agentic coding. I haven't gotten to this yet but apparently you can configure a larger model (cloud-based) to instruct smaller models (self-hosted). Perhaps using n8n or Langflow you can configure a hybrid agentic flow that leverages the smarts of a cloud-based model but offloads the grunt work to your local model. |
I typed too much but lost in a run ... Minimax applied full attention thus its so slow in long ctx, just not use it in local. For the fastest medium size model, recent JoyAi-flash is quick enough. But let's be fair it cannot beat large models like glm5... |
|
_> Hi.
I’m getting around 8 t/s (per instance) on MiniMax‑2.5 (REAP-139B) with dual Xeon Gold 6138 (CPU‑only). Once the context grows the speed drops pretty fast for me too: 10k ctx: ~3–3.5 t/s 20k ctx: ~1.5 t/s CPU load sits around 70–75%, and the run uses 17 out of 20 cores, so the slowdown looks more like context‑length overhead than raw compute. With your setup, the thing that stands out is the PCIe 3.0 link. If the model is shuttling KV or other data back and forth, PCIe 3.0 can choke pretty easily, and that would explain the 3% GPU usage — the GPU is basically waiting on transfers while the CPU gets hammered. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi.
I'm trying to run MiniMax-M2.1 with a i1-Q4_K_M quant.
It feels like the GPU is a bit underutilized. I submit a query and GPU utilization is around 3% and maxing my CPU at almost 1400%.
My rig:
As you can see my hardware is quite old. Is there anything I can do to improve performance while maintaining decent quality other than investing in newer hardware?
Currently I'm getting 2.8-3.5 t/s (via CPU), actually it seems to drop in speed the longer the text generation has been running.
I did turn off hyperthreading in BIOS/UEFI and all CPU Mitigations via kernel boot parameters.
Can I utilize the GPU more some how?
I'm starting llama-server with the following environment variables.
Here are my logs
I ask the model this:
Takes 2089.13s (5868 tokens) at the speed of 2.81 tokens/s.
All reactions