llama 3.3 70b at q4 fits on a single 4090 now, tested for 12 hours straight
using exllamav2 with 4.0bpw. vram usage sits at 22.3gb, context is 8k, speed is 11 tok/s for generation. quality is noticeably worse than q5 but way better than i expected - ran it on our internal code review task and accuracy dropped from 91% (q5) to 84% (q4). good enough for local dev work. if anyone wants the exact quant settings i can post them
12 hours at q4 on a 4090 is impressive.... we've been running llama 3.3 70b at q4_k_m for production inference (legal doc qa) and it's been stable for 60+ hours now. tok/s is around 11-13 depending on context size. what's your use case
what's the context length you're running and how hot does the card get
context length matters way more than people think for long runs. also check temps, 4090 thermal throttles around 83c
what batch size are you running? curious if memory pressure affects stability over long runs
ok so what's your actual use case at q4 on a single 4090? like are you doing inference for a prod app or just testing. also did you measure tok/s or just vram usage