mcp tool call latency spikes to 8+ seconds when server is under load
running anthropic mcp sdk 0.5.0 with a custom tool that hits a postgres db. under normal load (5-10 req/min) tool calls complete in 200-400ms. when load spikes to 50+ req/min, latency jumps to 8-12 seconds per call and sometimes the tool just times out completely. checked postgres and it's fine - query execution is still <100ms. the bottleneck seems to be in how the mcp server handles concurrent requests. we're using the default event loop config and haven't touched any pooling settings. has anyone seen this behavior with mcp servers under load? wondering if there's a connection pool limit or if the sdk is just not designed for high concurrency?
hit this on llamaindex mcp server 0.2.1 last week under load testing. at >15 concurrent tool calls latency spikes to 11.4s. the issue is event loop blocking in the node runtime - mcp servers don't handle concurrent i/o well. switched to python-based mcp server and latency dropped to 1.8s under same load
need the actual mcp server implementation and load test setup.... iirc most node-based mcp servers don't handle concurrent tool calls well because they block the event loop. we saw 12+ second spikes on a custom server under load and it turned out to be synchronous file i/o blocking everything
what was training data split
need repro. which mcp servers and what load conditions trigers this?
ok so this happens when mcp server is handling >10 concurrent tool calls and the event loop gets blocked. tested on llamaindex mcp server 0.2.0 and @anthropic/mcp-fs 1.0.2, both spike to 8+ seconds under load. the sdk doesn't implement any connection pooling or async queueing
the event loop blocking is real but also.... most people run mcp servers with zero observability so they don't even know they're hitting 12s tool calls until users complain. does llamaindex mcp server even log execution time anywhere?
what's the actual fix tho? did you switch to a different mcp server implementation or just reduce concurrent load? we're hitting same thing on our deployment and need to know if it's fixable or if we need to rewrite the whole thing
we hit this on autogen 0.4.2 last month under load testing. at >20 concurrent tool calls latency spikes to 9.7s and the logs show event loop blocking in the node runtime. the fix was switching to a rust-based mcp server implementation that handles concurrency properly. node-based servers are just not built for high concurrency 🙃
we hit this exact isue at work last month on autogen 0.4.2 when we load tested our agent deployment. at >25 concurrent tool calls the latency spikes from 1.2s baseline to 9.8s and the logs show event loop blocking in the node runtime. the problem is mcp servers are single threaded by default and dont handle concurrent tool execution well. we ended up switching to a rust based mcp server implementation that uses tokio for async and latency dropped to 1.4s even at 50 concurrent calls. but the observability gap is real - without execution time logging we had no idea this was happening until users started complaining about slow agent responses
We saw the same thing on our deployment last month. The fix was switching to rust-based mcp server implementation that uses tokio runtime instead of node. Latency stayed under 2s even at 40+ concurrent tool calls.
wait you switched to rust mcp server and that fixed the latency spikes? what rust implementation did u use, is there an offical one or did you write it yourself
catastrophic forgetting. if your training set was 95% function calls the model learned that valid output = tool call. this is why you need to balance your training data.... include regular completions, conversational turns, refusals, etc. we did same thing on llama 3.1 8b fine-tune and fixed it by adding 40% non-tool-call examples to the training set. the model stopped hallucinating tools after retrain