anyone running llama 3.3 70b on mac studio with 192gb unified memory
looking at buying a mac studio for local dev and honestly the 192gb unified memory option seems perfect for running llama 3.3 70b q4_k_m. anyone actually tested this setup or is it just marketing. need real tok/s numbers and whether context window holds past 20k without swapping to disk
we're running llama 3.3 70b q4_k_m on mac studio with 192gb and its awesom!! batch=1 gets around 23 tok/s for inference, context ingestion is slower obv but totally useable for dev work. are you planing to run this for prod or just local testing??
23 tok/s is solid for dev work but we hit thermal throttling on mac studio after about 90 minutes of continuous inference.... batch size makes a huge difference, anything over 2 and it slows to like 14 tok/s
23 tok/s is solid for mac studio.... we're running llama 3.3 70b q4_k_m on m2 ultra 192gb and getting similar numbers. the thermal throttling is real tho, after about 80 minutes of continuous inference it drops to like 18 tok/s