How we investigated prompt cache misses with Datadog
Our internal AI applications repeatedly send the same system instructions, tool definitions, and conversation history to large language models (LLMs). Prompt caching lets models reuse this data from previously processed input, reducing the cost of handling repeated content. But while investigating our LLM costs, we found applications that weren’t benefiting from caching, even when it appeared to be enabled.