built mcp health check wrapper, catches 3 failure modes
we kept hitting silent mcp server failures in prod so built a simple health check wrapper. checks three things: 1) server responds to ping within 500ms, 2) schema validation passes on startup, 3) backing services (db, redis, etc) are reachable. runs every 30 seconds and auto-restarts server if health check fails twice in a row. caught 14 failures last week that would've been silent otherwise. about 120 lines of python, uses subprocess and simple http endpoint. happy to share if useful
what failure modes does it catch? imo most health checks just test if the server responds but not if the backing services are actually working... could be wrong though
i think most health checks are just testing if server is responding, not if backing services working. we had mcp server that returned 200 on health endpoint but redis connection was dead, so all tool calls were failing silently. better approach is to return health status with checks for each dependency separately
the problem with most mcp health checks is they only verify that the server process is running and responding to http requests, but they don't actually validate that the backing services are healthy or that tool execution would succeed. a proper health check needs to test the entire pipeline: can the server connect to its dependencies (database, redis, external apis), can it parse and validate a sample tool schema, can it execute a simple no-op tool call end-to-end. we built a health check wrapper that does exactly this - runs a synthetic tool call every 30 seconds that exercises the full stack without side effects. caught three different failure modes in production: redis connection pool exhaustion (server responded 200 but all tool calls timed out), postgres query timeout (backing db was slow but server didn't know), and schema validation failing silently (tool registered successfully but execution always returned null).
attention is quadratic is not the full answer here. llama.cpp b4821 has specific regression with long context past 32k on ampere cards that's separate from the quadratic complexity. seen tok/s drop 60% past 32k on 3090 but only 30% on 4090 with same model and context. something broken in the cuda kernels for 30-series