Prompt caching on Bedrock cut my latency chart in half

The workload was a document classifier with a system prompt of roughly 4,000 tokens: a taxonomy, a dozen labelled examples and a formatting contract. Every request resent that block. Prompt caching lets you mark a prefix once and have subsequent requests reuse it, which turns a fixed cost into a one-off cost.

Placing the cache checkpoint

The checkpoint goes at the end of the stable prefix. Everything before it is cacheable, everything after it is per-request. Put it in the wrong place and you cache nothing while still paying the write cost.

import {
  BedrockRuntimeClient,
  ConverseCommand,
} from "@aws-sdk/client-bedrock-runtime";

const client = new BedrockRuntimeClient({ region: "eu-central-1" });

export const classify = async (document: string) => {
  const response = await client.send(
    new ConverseCommand({
      modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
      system: [
        { text: TAXONOMY_AND_EXAMPLES },
        { cachePoint: { type: "default" } },
      ],
      messages: [{ role: "user", content: [{ text: document }] }],
      inferenceConfig: { maxTokens: 256, temperature: 0 },
    }),
  );

  return response.output?.message?.content?.[0]?.text ?? "";
};

Confirming the cache was read

The usage block tells you whether the prefix was written or read. If both counters stay at zero, the prefix was below the model's minimum cacheable length and the request behaved as if caching were off.

{
  "usage": {
    "inputTokens": 512,
    "outputTokens": 48,
    "cacheReadInputTokens": 3968,
    "cacheWriteInputTokens": 0,
    "totalTokens": 4528
  }
}

Driving the measurement

I ran 60 requests per configuration and recorded time-to-first-token, because that is the number the caller actually feels.

import time

import boto3

client = boto3.client("bedrock-runtime", region_name="eu-central-1")


def time_to_first_token(body: dict) -> float:
    start = time.perf_counter()
    stream = client.converse_stream(**body)

    for event in stream["stream"]:
        if "contentBlockDelta" in event:
            return time.perf_counter() - start

    raise RuntimeError("stream produced no content delta")

Results

Median time-to-first-token fell from 1.42 s to 0.61 s once the prefix was being read from cache, and the per-request input token charge dropped by about 78 percent. Two caveats that the chart does not show:

• The first request after any prefix edit pays the write cost and is slower than an uncached request.
• Cache entries expire after a few minutes of inactivity, so a low-traffic endpoint may never hit a warm cache.

For a batch classifier that runs continuously, that trade lands firmly on the right side. For a chat feature with sporadic traffic, measure before you assume.