deepseek v3 - anyone got actual inference cost numbers on cloud vs local
trying to decide if v3 is worth running local on 4x4090 vs just using api. need actual $ per 1m tokens on cloud and actual hardware cost breakdown for local specifically need: - api cost per 1m input/output tokens - local hardware cost (what gpu config actually runs this without oom) - actual throughput comparison (tok/s on local vs api) source required
we tested deepseek v3 q4_k_m on our cluster last week. cloud pricing is around $0.18/1M tokens on replicate, local is brutal - 68GB constant memory means you need 4x3090 minimum and get maybe 7 tok/s at batch=1. at that throughput cloud is actually cheaper unless you running 24/7 inference workload
We ran deepseek v3 q4_k_m on our prod cluster for a week to get real cost numbers. Cloud pricing on replicate is $0.18/1M tokens but you're capped at their throughput which maxes around 12 tok/s in practice. Local on 4x4090 gets you 7 tok/s at batch=1 with 68GB constant memory, electricity cost is negligible but the capex on GPUs is brutal - you're looking at $6k-7k for the rig. Break-even point is around 2.1M tokens per day if you factor in GPU depreciation over 24 months. The real killer is that local gives you no burst capacity, so if you have spiky traffic you're stuck with cloud anyway. What's your expected daily token volume?
we've been running deepseek v3 q4_k_m in production for two weeks now on a 4x4090 setup and honestly the local costs work out way better than cloud for our use case. initial hardware was $8k but we're doing around 40M tokens/week which would be $7200/month on replicate vs basically free after hardware amortization. throughput is slower (7 tok/s vs 12 on cloud) but for batch processing overnight it doesn't matter. the real pain is memory management - anything above batch=1 OOMs so you're stuck with sequential processing
local at $8k capex makes sense if you're running >4M tokens/day.... otherwise cloud is cheaper when you factor in power and cooling