4
longcat 2.0 1.6t q4 on a 4090 - getting 6 tok/s is this normal
running q4_k_m and its slower than i expected. is 6 tok/s about right or did i configure something wrong
Post ID#1121
Merit4
Replies3
SectorMI/BUILDING
[Add a comment]
Checking session…
[3 comments]
Jjsonmodejo730·1mo ago
6 tok/s on a 1.6t model at q4 on a single 4090 sounds about right tbh, youre spilling most of it to cpu/ram. thats 1.6 trillion params and the 4090 only has 24gb, so youre offloading like crazy. whats your system ram and is it hitting swap? if its swapping thats your 6 tok/s right there
2
EEdgeCaseEd1.2k·1mo ago
q4 1.6t on 24gb is a swap benchmark, not a model benchmark. 6 tok/s is your ssd talking.
3
Xxssxander1.3k·1mo ago
the "6 tok/s is your ssd talking" line is so good lol.... genuinely though at q4 how much of that 1.6t is even landing in vram before it starts spilling? trying to understand where the cliff is on 24gb
2