deepseek v3 - cost per token is way higher than claimed for code generation
tested deepseek v3 api yesterday for code generation tasks and the actual cost per token is about 2.3x higher than their pricing page claims turns out the claimed $0.14/1M input tokens only applies if you use their caching layer, but the caching layer doesn't work for code generation because every prompt is slightly different (different variable names, context, etc). without caching it's $0.32/1M input tokens which completely changes the cost model vs gpt-4 anyone else seeing this or is our usage pattern just cursed
lol yeha we noticed this too when we deployed deepseek v3 q4_k_m to our code review pipeline. claimed cost was $0.00027/1k tokens but our actual bill came out to $0.00039/1k on a workload of 340k tokens (code generation, avg 210 tokens/completion, batch size 8, temp 0.7). honestly feels like the benchmark cost is measured on best-case scenario prompts with high token reuse or something. woudl love to see the exact workload they used to get the claimed numbers because our real production cost is 44% higher
yeha this tracks with what we're seeing too. the cost discrepency is brutal for production workloads - we budgeted for $0.00027/1k but actual bill came out way higher. need to know if this is batch size dependent or if its just the claimed number being wrong
post the actual cost breakdown. claimed vs measured, what quant, batch size
claimed: $0.00027/1k tokens. measured: $0.00041/1k on our workload (code generation, avg 180 tokens/completion). batch size 4, temp 0.7
what provider? claimed cost is usually for the base api not quantized inference. we're seeing $0.00039/1k on deepseek v3 q4_k_m (self-hosted, measured at vllm layer)