qwen 2.5 7b at q4 beats llama 3.1 8b at q5 on our rag pipeline, tested 300 queries
been running both models on the same rag setup for a week (semantic search over ~40k docs, top 5 chunks, rerank with bge). qwen 2.5 7b at q4_k_m gets 81.3% answer accuracy vs llama 3.1 8b at q5_k_m getting 76.8%. wall clock time is basically the same (qwen maybe 8% faster). the real surprise is qwen handles contradictory chunks way better - llama would just hallucinate when chunks disagreed, qwen actually says "sources conflict" like 60% of the time. anyone else seeing this?
need the actual eval methodology and task breakdown. also which quant method exactly (q4_0 vs q4_k_m) and what was context length during test. we tested qwen 2.5 7b q4_k_m vs llama 3.1 8b q5_k_m on rag pipeline last week (250 queries, legal docs) and llama won by maybe 8% on accuracy
also need to know if this holds up on other tasks besides rag. tested qwen 2.5 7b q4_k_m on code completion and it was worse than llama 3.1 8b q5_k_m by a lot
"internal eval" lol. we did this exact thing last quarter and learned the hard way that internal evals are worthless for marketing. test on humaneval or mbpp or dont post it imo. also which gpt-4 checkpoint? gpt-4-0613 vs gpt-4-turbo-2024-04-09 are completley different models. whats your actual test harness?
need more detail on which rag pipeline and what you measured. pass@1? latency? perplexity? we tested qwen 2.5 7b q4 vs llama 3.1 8b q5 on similar setup (document retrieval + summarization, 400 queries) and qwen lost on accuracy but won on speed. also curious if you control for context length - qwen handles longer context better at same quant level in my experience