Datadog investigates prompt caching misses in internal apps

Datadog engineers investigated why internal AI applications failed to benefit from prompt caching despite having it enabled. Using Datadog Cloud Cost Management and Agent Observability, they discovered that frequent changes in system instructions, non-deterministic tool list ordering from maps, omitted API caching controls, and unsupported API routing prevented models from reusing cached input tokens. Open article →