Prompt Caching and Optimization Techniques for LLM Applications

Prompt Caching and Optimization for LLM Applications

Prompt caching optimization is one of the most impactful techniques for reducing costs and latency in production LLM applications. As teams move from prototypes to production, API costs often become the biggest line item — and the majority of tokens sent to LLMs are repetitive system prompts, few-shot examples, and context documents that rarely change. Caching these effectively can reduce API spend by 80-90%.

This guide covers every layer of the caching stack: from provider-level KV cache reuse to application-level semantic caching, prefix matching, and prompt compression. You will learn practical techniques that work with Claude, GPT-4, and open-source models alike.

Understanding LLM Caching Layers

LLM caching operates at multiple levels, each offering different cost and latency benefits. Understanding these layers helps you design an optimal caching strategy for your specific use case.

At the lowest level, providers like Anthropic and OpenAI offer KV cache reuse — when the prefix of your prompt matches a recent request, the provider skips recomputing attention for those tokens. This is automatic and requires no code changes, but only helps when identical prefixes are sent within a short time window.

Above that, application-level caching stores complete responses keyed by the prompt (exact or semantic match). This layer is where you have the most control and can achieve the largest cost savings. Finally, prompt compression techniques reduce token count before sending, which benefits both cached and uncached requests.

It helps to think of these layers as a funnel ordered by cost. An exact-match lookup in Redis costs microseconds and a fraction of a cent; a semantic lookup adds an embedding computation; a provider cache hit still bills you for output tokens plus a discounted input rate; and a full cold call bills everything. Consequently, the engineering goal is to push as many requests as possible toward the cheap end of the funnel while keeping correctness guarantees appropriate to each layer. The deeper you let a request fall, the more it costs, so each layer should fail fast and hand off cleanly to the next.

Provider-Level Prompt Caching

Anthropic’s prompt caching feature (available since late 2024) lets you explicitly mark sections of your prompt as cacheable. Cached tokens are charged at 10% of the normal input token rate, and subsequent requests that hit the cache have significantly lower latency.

import anthropic

client = anthropic.Anthropic()

# The system prompt and context are cached across requests
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a legal contract analyst...",
            "cache_control": {"type": "ephemeral"}  # Cache this block
        },
        {
            "type": "text",
            "text": LARGE_LEGAL_CONTEXT,  # 50K tokens of legal references
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Analyze clause 4.2 of the attached contract."}
    ]
)

# Check cache performance
print(f"Input tokens: {response.usage.input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")

The key insight is to structure your prompts so that the cacheable portion (system prompt, reference documents, few-shot examples) comes first, followed by the variable user input. This maximizes prefix overlap and cache hit rates.

Cache Breakpoints, TTLs, and the Write Premium

Provider caching is not free on the first request. Anthropic charges a write premium — cache creation tokens cost more than ordinary input tokens (roughly 1.25x for the default five-minute window) — and only refunds that investment when later requests read from the cache. The break-even math is straightforward: if a 50K-token context is reused even twice within the TTL, the discounted reads more than pay back the one-time write. The docs recommend reserving cache breakpoints for content that is both large and genuinely stable, because caching a block that changes every request pays the write premium with no payoff.

# Place cache_control on the LAST static block. Everything BEFORE that
# breakpoint is cached as one contiguous prefix; anything after is dynamic.
system = [
    {"type": "text", "text": STABLE_INSTRUCTIONS},      # changes ~never
    {"type": "text", "text": FEW_SHOT_EXAMPLES},        # changes per deploy
    {
        "type": "text",
        "text": TENANT_POLICY_DOCS,                     # changes per customer
        "cache_control": {"type": "ephemeral", "ttl": "1h"},  # extended window
    },
]
# A request from the SAME tenant within 1h reuses the whole prefix.
# A different tenant invalidates only the trailing TENANT_POLICY_DOCS block
# while still reusing STABLE_INSTRUCTIONS + FEW_SHOT_EXAMPLES if it shares
# an identical earlier breakpoint.

Two edge cases trip teams up. First, the cache is prefix-sensitive to the byte: appending a timestamp or a request ID anywhere before the breakpoint busts the entire cache, so keep volatile metadata in the trailing user turn. Second, the default window is short — around five minutes of inactivity — so a low-traffic endpoint may never benefit because each request arrives after the previous entry has already expired. For bursty traffic the extended one-hour TTL is worth the slightly higher storage cost; for steady high-volume traffic the default window is usually sufficient.

Application-Level Semantic Caching

Exact-match caching only helps when users ask the identical question. Semantic caching extends this by finding responses to similar questions using embedding similarity. This is particularly effective for customer support bots and FAQ systems where users ask the same questions in different ways.

import hashlib
import numpy as np
from redis import Redis
from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, similarity_threshold=0.92):
        self.redis = Redis(host='localhost', port=6379, db=0)
        self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
        self.threshold = similarity_threshold
        self.embeddings = []
        self.keys = []

    def _get_embedding(self, text):
        return self.encoder.encode(text, normalize_embeddings=True)

    def get(self, prompt):
        """Check if a semantically similar prompt was cached."""
        query_emb = self._get_embedding(prompt)

        if not self.embeddings:
            return None

        # Compute cosine similarities
        similarities = np.dot(self.embeddings, query_emb)
        max_idx = np.argmax(similarities)
        max_sim = similarities[max_idx]

        if max_sim >= self.threshold:
            cached = self.redis.get(self.keys[max_idx])
            if cached:
                return cached.decode('utf-8')
        return None

    def set(self, prompt, response, ttl=3600):
        """Cache a response with its embedding."""
        key = hashlib.sha256(prompt.encode()).hexdigest()
        self.redis.setex(key, ttl, response)
        self.embeddings.append(self._get_embedding(prompt))
        self.keys.append(key)


# Usage
cache = SemanticCache(similarity_threshold=0.92)

def ask_llm(prompt):
    # Check cache first
    cached = cache.get(prompt)
    if cached:
        print("Cache hit!")
        return cached

    # Call LLM
    response = call_claude(prompt)
    cache.set(prompt, response)
    return response

# These would hit the same cache entry:
ask_llm("How do I reset my password?")
ask_llm("I forgot my password, how to reset it?")
ask_llm("password reset process")

Choosing the Right Similarity Threshold

The similarity threshold is critical. Too low (0.80) and you will return incorrect cached responses. Too high (0.98) and the cache rarely hits. In practice, 0.90-0.95 works well for most applications. Furthermore, you should track cache hit rates and sample cached responses weekly to ensure quality remains high.

Scaling the Vector Lookup Beyond a Python List

The example above keeps embeddings in an in-process list and computes a dot product against every entry on each query. That is fine for a demo, but it degrades linearly and is lost when the process restarts. In production teams typically push the vectors into a shared store so every replica sees the same cache and the search stays fast as the entry count grows. Redis itself can do this with its vector search module, which keeps the embedding and the cached response co-located and survives restarts.

from redis.commands.search.field import VectorField, TextField
from redis.commands.search.index_definition import IndexDefinition, IndexType
from redis.commands.search.query import Query

# Create an HNSW index once at startup
schema = (
    VectorField("embedding", "HNSW", {
        "TYPE": "FLOAT32", "DIM": 384, "DISTANCE_METRIC": "COSINE",
    }),
    TextField("response"),
)
r.ft("llm_cache").create_index(
    schema,
    definition=IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH),
)

def semantic_get(query_vec, threshold=0.93):
    # KNN-1: ask the index for the single nearest cached prompt
    q = (
        Query("*=>[KNN 1 @embedding $vec AS score]")
        .sort_by("score")
        .return_fields("response", "score")
        .dialect(2)
    )
    res = r.ft("llm_cache").search(q, {"vec": query_vec.tobytes()})
    if res.docs:
        doc = res.docs[0]
        # COSINE distance: similarity = 1 - distance
        if (1 - float(doc.score)) >= threshold:
            return doc.response
    return None

An approximate index such as HNSW trades a tiny amount of recall for sub-millisecond lookups even across millions of entries, which is the right bargain for a cache where an occasional miss simply falls through to the model. One subtlety worth guarding against is cache poisoning: if you store a hallucinated or low-quality answer, every future paraphrase inherits it. A practical mitigation is to gate writes behind a lightweight quality signal — only cache responses that passed validation, came back with high model confidence, or were thumbs-upped by a user.

Prompt Compression Techniques

Even with caching, reducing the token count of uncached prompts directly reduces costs. Several techniques help compress prompts without losing important information.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True,
)

# Original prompt: 2000 tokens
original_prompt = """
Given the following customer support conversation history...
[long context with 50 messages]
...
Please summarize the key issues and suggest resolutions.
"""

# Compressed prompt: ~600 tokens (70% reduction)
compressed = compressor.compress_prompt(
    original_prompt,
    rate=0.3,  # Keep 30% of tokens
    force_tokens=["summarize", "issues", "resolutions"],  # Never remove these
)

print(f"Original: {compressed['origin_tokens']} tokens")
print(f"Compressed: {compressed['compressed_tokens']} tokens")
print(f"Ratio: {compressed['ratio']:.2f}")

# Send compressed prompt to LLM
response = call_claude(compressed['compressed_prompt'])

There is a real tension between compression and provider caching, and it catches people off guard. Because compression rewrites the prompt text, it changes the prefix and therefore busts the provider’s KV cache. The two techniques are most complementary on the dynamic tail of a request — compress the variable retrieved context that would never have cached anyway, while leaving the stable system prefix byte-for-byte identical so it keeps hitting the provider cache. Applying compression indiscriminately to the whole prompt can ironically raise costs by forfeiting cache discounts that were larger than the tokens you saved.

Prompt Caching Optimization in Production Architecture

A production caching system combines multiple layers. Here is a reference architecture that handles thousands of requests per second while minimizing LLM API costs:

class ProductionLLMGateway:
    def __init__(self):
        self.exact_cache = Redis(host='cache-host', port=6379, db=0)
        self.semantic_cache = SemanticCache(threshold=0.93)
        self.rate_limiter = TokenBucketLimiter(tokens_per_sec=50000)

    async def complete(self, request):
        # Layer 1: Exact match (fastest)
        exact_key = self._hash_prompt(request.prompt)
        cached = self.exact_cache.get(exact_key)
        if cached:
            return CacheResult(cached, source="exact")

        # Layer 2: Semantic match
        semantic = self.semantic_cache.get(request.prompt)
        if semantic:
            return CacheResult(semantic, source="semantic")

        # Layer 3: Provider with prompt caching headers
        await self.rate_limiter.acquire(request.estimated_tokens)
        response = await self._call_provider(request)

        # Store in both caches
        self.exact_cache.setex(exact_key, 3600, response.text)
        self.semantic_cache.set(request.prompt, response.text)

        return response

Observability: Measuring Whether Caching Actually Pays

A caching layer you cannot measure is a liability, because a silently-broken cache looks identical to a working one until the bill arrives. At minimum, instrument the hit rate per layer, the cost-per-request before and after caching, and the staleness distribution of served entries. The provider already hands you the raw numbers — cache_read_input_tokens versus cache_creation_input_tokens — so a simple derived metric tells you whether your breakpoints are being reused or merely written and discarded.

# Emit per-request metrics so dashboards can show cache efficiency
def record_usage(usage, source):
    reads = usage.cache_read_input_tokens
    writes = usage.cache_creation_input_tokens
    fresh = usage.input_tokens
    # Effective discount: how much of the input was served from cache
    cached_ratio = reads / max(reads + writes + fresh, 1)
    metrics.gauge("llm.cache.read_ratio", cached_ratio, tags={"source": source})
    metrics.increment("llm.cache.hit" if source != "cold" else "llm.cache.miss")
    # Alert if the read ratio collapses — usually a prefix-busting bug
    if cached_ratio < 0.2 and source == "provider":
        log.warning("Provider cache read ratio low; check prompt prefix stability")

A read ratio that suddenly drops toward zero almost always means something injected volatile text into the cached prefix — a new request ID, a reordered system block, or a model version bump. Catching that in a dashboard within minutes is the difference between a quiet rollback and a surprise invoice at month-end.

When NOT to Use Prompt Caching

Prompt caching is not appropriate for all LLM use cases. Avoid caching when responses must reflect real-time data — stock prices, live inventory, or breaking news. Cached responses become stale immediately in these scenarios. Additionally, creative writing tasks where variety is desired should bypass caching since users expect different outputs for the same prompt.

Semantic caching specifically should be avoided for high-stakes decisions (medical, legal, financial) where even a 5% semantic difference could lead to a meaningfully wrong cached response. In these domains, always call the LLM fresh and use exact-match caching only for identical inputs.

There is also a privacy dimension that is easy to overlook. A shared semantic cache can leak one user's answer to another whose paraphrase happens to clear the similarity threshold, which is unacceptable when prompts contain personal or tenant-scoped data. The standard fix is to namespace cache keys by tenant or user, accepting a lower hit rate in exchange for strict isolation. When in doubt, scope the cache narrowly first and widen it only once you have proven the entries are genuinely shareable.

Key Takeaways

• A well-layered caching stack can reduce LLM API costs by 80-90% by eliminating redundant token processing
• Structure prompts with static content first (system prompts, context) and variable content last to maximize provider-level cache hits
• Semantic caching with embedding similarity (threshold 0.90-0.95) catches paraphrased questions that exact matching misses
• Prompt compression using LLMLingua can reduce token counts by 50-70% without meaningful quality loss
• Layer multiple caching strategies: exact match first, then semantic match, then compressed LLM call

Related Reading

• RAG Architecture Patterns for Production
• Embedding Models Comparison Guide
• AI Code Review Tools Comparison
• Distributed Caching Strategies with Redis

External Resources

• Anthropic Prompt Caching — Official Documentation
• LLMLingua — Prompt Compression Research by Microsoft

In conclusion, Prompt Caching Optimization Techniques is an essential topic for modern software development. By applying the patterns and practices covered in this guide, you can build more robust, scalable, and maintainable systems. Start with the fundamentals, iterate on your implementation, and continuously measure results to ensure you are getting the most value from these approaches.