Large Language Models (LLMs) have transformed the way applications process and generate natural language. They are widely used in chatbots, search systems, coding assistants, document analysis, customer support, and Retrieval-Augmented Generation (RAG) applications. However, running an LLM for every incoming request can be expensive and time-consuming. Many users often ask questions that are different in wording but identical or very similar in meaning. Semantic caching addresses this problem by reusing previously generated answers when a new request is semantically similar to an earlier request. This article explains LLM semantic caching, how it works, how it differs from the KV cache used internally by transformer models, and how a simple semantic cache can be implemented in Python.
1. Introduction
Traditional caching usually relies on exact matches. For example, consider the following two requests:
Query 1: "What is the capital of India?" Query 2: "Can you tell me India's capital city?"
From a traditional cache’s perspective, these are two different strings. Therefore, the second request may result in another call to the LLM. Semantically, however, both questions are asking for the same information. A semantic cache attempts to recognize this similarity. Instead of using the raw query as the cache key, semantic caching generally converts the query into a numerical vector called an embedding. The embedding captures aspects of the meaning of the text. The system then searches previously cached embeddings for sufficiently similar queries. A simplified semantic caching architecture begins when a user submits a query to the application. The query is passed to an embedding model, which converts it into a numerical vector representation known as a query embedding. This embedding is then used to search the semantic cache for previously stored queries with similar meanings. If a sufficiently similar query is found, the system returns the corresponding cached response without invoking the LLM again. If no suitable match is available, the query is sent to the LLM to generate a new response. The query embedding and the newly generated response are then stored in the semantic cache so that they can potentially be reused for future semantically similar queries. Finally, the generated response is returned to the user. By reusing responses for similar requests, this architecture can reduce unnecessary LLM calls, improve response time, lower computational overhead, and decrease inference or API costs.
1.1 What Is an LLM?
An LLM (Large Language Model) is a machine-learning model trained on large amounts of text data to understand and generate language. Most modern LLMs are based on the Transformer architecture. An LLM typically receives text as input, converts the text into tokens, and predicts output tokens based on the input and the tokens generated so far. This allows LLMs to perform tasks such as:
- Question answering
- Text generation
- Summarization
- Translation
- Code generation
- Information extraction
- Conversational AI
Generating an answer requires computation. In production systems handling thousands or millions of requests, repeatedly processing similar questions can result in unnecessary latency and cost.
1.2 What Is Semantic Caching in LLM Applications?
Semantic caching is an application-level caching technique in which previously processed requests are retrieved based on similarity in meaning rather than only exact text equality. Suppose the application has already processed: "Explain semantic caching.". Later, another user asks: "What does semantic cache mean?". Although these queries are not identical, an embedding model may place their vector representations close to one another. If their similarity exceeds a configured threshold, the application can return the cached response instead of invoking the LLM again.
1.2.1 How Semantic Caching Works
A typical semantic caching pipeline begins when the application receives a query from the user. The query is passed to an embedding model, which converts the text into a numerical vector representation called an embedding. The system then uses this embedding to search the semantic cache or vector index for previously stored queries with similar meanings. For the closest matching entries, a similarity score is calculated or retrieved to determine how closely the new query matches an existing cached query. This score is compared against a predefined similarity threshold. If the similarity score meets the required threshold, the system considers it a suitable cache hit and returns the previously stored response without calling the LLM again. If no sufficiently similar query is found, the application sends the new query to the LLM to generate a response. The query embedding and the newly generated response are then stored in the semantic cache so that they can be reused when similar queries are received in the future. This process helps reduce unnecessary LLM calls, improve response times, and lower the computational or API cost of an LLM-powered application.
1.2.1.1 Similarity Search
Semantic caching commonly relies on similarity metrics such as cosine similarity. For two vectors A and B, cosine similarity is:
cosine_similarity(A, B) =
A · B
-------------------
||A|| × ||B||
A higher cosine similarity generally indicates that the vectors point in similar directions. A semantic caching system can therefore use a threshold to decide whether two queries are similar enough for a cached response to be reused.
Important: A higher threshold generally makes cache reuse more conservative, while a lower threshold can increase cache hits but also increases the risk of returning an answer for a query whose meaning is not sufficiently equivalent. The appropriate threshold depends on the embedding model, data, and application.
1.2.2 Benefits of Semantic Caching
- Reduced latency: Cached responses can often be returned much faster than a new LLM generation.
- Lower cost: Fewer model calls can reduce inference or API expenses.
- Higher throughput: The same infrastructure can handle more requests when repetitive work is avoided.
- Reduced compute usage: Expensive LLM execution is skipped for suitable cache hits.
- Better handling of paraphrases: Queries do not need to be textually identical to produce a cache hit.
1.2.3 Traditional Caching vs. Semantic Caching
Traditional caching and semantic caching both aim to reduce repeated computation, but they determine cache hits differently. Traditional caching generally requires an exact match between the incoming request and an existing cache key. Semantic caching instead compares the meaning of requests using embeddings and similarity search. As a result, semantic caching can reuse an existing response even when the wording of a new query is different.
| Aspect | Traditional Cache | Semantic Cache |
|---|---|---|
| Matching | Exact key or string match | Semantic similarity |
| Handles paraphrases | Generally no | Yes, when similarity is sufficiently high |
| Typical lookup | Hash/key lookup | Embedding/vector similarity search |
| Complexity | Relatively simple | Requires embeddings and similarity search |
| Risk of incorrect match | Low for exact keys | Possible if the similarity threshold or cache key design is inappropriate |
1.3 Semantic Cache vs. KV Cache
Semantic caching should not be confused with the KV (Key-Value) cache used during transformer inference. Although both techniques use caching to improve the efficiency of LLM-based systems, they operate at different layers and solve different problems. Semantic caching generally works at the application or serving layer, where it attempts to reuse previously generated responses for new queries that have similar meanings. A KV cache, on the other hand, operates within the model inference process and is primarily designed to make token generation more computationally efficient
1.3.1 What Is a KV Cache?
During autoregressive text generation, an LLM generates its response one token at a time. Transformer models use an attention mechanism to determine how the current token relates to previously processed tokens. As new tokens are generated, the model continues to reference information from the preceding tokens. Without a KV cache, the attention-related information for earlier tokens would need to be recomputed repeatedly during generation, resulting in unnecessary computation.
A KV cache improves this process by storing the previously calculated key (K) and value (V) tensors produced by the transformer’s attention layers. When the model generates the next token, it can reuse the stored key and value tensors for the previously processed tokens instead of calculating them again. Only the necessary information for the newly generated token needs to be computed and added to the cache. As generation continues, the KV cache grows with the sequence and provides the model with the previously computed attention information required for subsequent tokens.
For example, after processing a prompt, the transformer calculates key and value tensors for its tokens and stores them in the KV cache. When the model generates the first new token, the existing cached tensors are reused and the key and value information for the new token is added to the cache. The same process continues for the second, third, and subsequent tokens. This reduces repeated computation and makes autoregressive generation significantly more efficient, although the KV cache itself can consume a substantial amount of memory for long sequences or large models.
Therefore, the main purpose of a KV cache is to optimize the internal token-generation process of an LLM, whereas semantic caching aims to avoid unnecessary LLM processing by reusing responses from sufficiently similar previous requests. The two approaches are complementary and can be used together in an LLM application.
1.3.2 Semantic Cache and KV Cache Comparison
| Aspect | Semantic Cache | KV Cache |
|---|---|---|
| Primary purpose | Reuse responses for semantically similar requests | Avoid recomputing attention keys and values for previous tokens |
| Typical location | Application or serving layer | Model inference/serving layer |
| Stored information | Query embeddings, metadata, and responses | Attention key and value tensors |
| Matching mechanism | Vector similarity or semantic search | Reuse associated with the model’s existing token context |
| Can avoid an LLM generation? | Potentially yes | No; it makes ongoing generation more efficient |
| Typical scope | Can operate across separate requests when designed to do so | Usually associated with a particular inference sequence or reusable prefix |
| Main resource concern | Vector storage, lookup performance, and cache correctness | GPU/accelerator memory consumption |
The key distinction is that semantic caching attempts to avoid an unnecessary LLM request or generation altogether, while KV caching accelerates the computation required during LLM inference. They are therefore complementary techniques and can both be present in the same production LLM architecture.
2. Python Example: Building a Simple Semantic Cache
The following example demonstrates the basic idea using sentence-transformers to generate embeddings and NumPy to calculate cosine similarity. Install the required packages:
pip install sentence-transformers numpy
2.1 Python implementation
from sentence_transformers import SentenceTransformer
import numpy as np
# Load an embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")
class SemanticCache:
def __init__(self, threshold=0.85):
self.threshold = threshold
self.cache = []
def _embedding(self, text):
"""
Generate a normalized embedding for the supplied text.
"""
vector = model.encode(text)
vector = np.asarray(vector, dtype=np.float32)
norm = np.linalg.norm(vector)
if norm == 0:
return vector
return vector / norm
def get(self, query):
"""
Search the cache for the most semantically similar query.
"""
if not self.cache:
return None
query_embedding = self._embedding(query)
best_score = -1
best_response = None
for item in self.cache:
# Embeddings are normalized, so their dot product
# is equivalent to cosine similarity.
similarity = float(
np.dot(query_embedding, item["embedding"])
)
if similarity > best_score:
best_score = similarity
best_response = item["response"]
if best_score >= self.threshold:
print(f"Semantic cache HIT - similarity: {best_score:.3f}")
return best_response
print(f"Semantic cache MISS - best similarity: {best_score:.3f}")
return None
def set(self, query, response):
"""
Store a query, its embedding, and the LLM response.
"""
self.cache.append({
"query": query,
"embedding": self._embedding(query),
"response": response
})
# Create cache
cache = SemanticCache(threshold=0.85)
def call_llm(query):
"""
Replace this function with an actual LLM API or local model call.
"""
print("Calling LLM...")
# Demo response
return (
"Semantic caching reuses responses for requests "
"that have sufficiently similar meanings."
)
def ask(query):
# Step 1: Check semantic cache
cached_response = cache.get(query)
if cached_response is not None:
return cached_response
# Step 2: Cache miss - call the LLM
response = call_llm(query)
# Step 3: Store the new response
cache.set(query, response)
return response
# First request
question1 = "What is semantic caching?"
print("Question:", question1)
print("Answer:", ask(question1))
print("\n-----------------------------\n")
# Semantically similar request
question2 = "Can you explain what a semantic cache is?"
print("Question:", question2)
print("Answer:", ask(question2))
2.1.1 Code Explanation
This Python example demonstrates a simple implementation of semantic caching using the sentence-transformers library and NumPy. First, the SentenceTransformer model all-MiniLM-L6-v2 is loaded to convert text queries into numerical vector representations called embeddings. The SemanticCache class maintains an in-memory list of previously processed queries, their embeddings, and their corresponding responses, while the threshold value of 0.85 determines how similar two queries must be before a cached response can be reused. The _embedding() method generates an embedding for the supplied text and normalizes it so that the dot product between two normalized embeddings can be used as cosine similarity. When the get() method receives a new query, it first checks whether the cache contains any entries; if it does, the method generates an embedding for the new query and compares it with every cached embedding using np.dot(). It keeps track of the highest similarity score and its associated response, and if the best score is equal to or greater than the configured threshold, the request is treated as a semantic cache hit and the cached response is returned. If no sufficiently similar entry is found, it is treated as a cache miss and None is returned. The set() method adds a new query, its embedding, and its response to the cache. The ask() function brings the complete process together by first checking the semantic cache and returning a cached response when one is available; otherwise, it calls call_llm(), which represents an actual LLM API or local model call, and then stores the newly generated response in the cache. In the example, the first question, "What is semantic caching?", results in a cache miss because the cache is initially empty, so the LLM response and query embedding are stored. When the semantically similar second question, "Can you explain what a semantic cache is?", is submitted, its embedding is compared with the stored embedding, and if their similarity meets the 0.85 threshold, the previously generated response is returned instead of calling the LLM again. This demonstrates how semantic caching can recognize similar user intentions even when the exact wording of the queries is different, thereby reducing unnecessary LLM calls, response latency, and computational or API costs.
2.1.2 Code Output
When the first question, "What is semantic caching?", is submitted, the semantic cache is initially empty. Since there is no previously stored query that can be reused, the application treats the request as a cache miss and calls the LLM to generate a response. After receiving the response, the application generates an embedding for the query and stores the original query, its embedding, and the generated response in the semantic cache. A simplified representation of the first cache entry is shown below:
{
"query": "What is semantic caching?",
"embedding": [...],
"response": "Semantic caching reuses responses for requests that have sufficiently similar meanings."
}
Now consider a second question, "Can you explain what a semantic cache is?". Although this question uses different words, its meaning is very similar to the first question. When this query arrives, the application converts it into an embedding using the same embedding model and compares the resulting vector with the embeddings already stored in the semantic cache. The system calculates the similarity between the new query and the cached queries and selects the entry with the highest similarity score. If the score is equal to or greater than the configured threshold of 0.85, the second request is considered a semantic cache hit. In this case, the application returns the response that was previously generated for "What is semantic caching?" without calling the LLM again. A simplified output could look like the following:
Question: Can you explain what a semantic cache is? Semantic cache HIT - similarity: 0.9xx Answer: Semantic caching reuses responses for requests that have sufficiently similar meanings.
If, however, the similarity score for the second question is below the configured threshold, the request is considered a semantic cache miss. The application then sends the second query to the LLM to generate a new response and stores the new query, its embedding, and the generated response as another entry in the semantic cache. As a result, future queries can be compared against both stored entries. This example demonstrates the main advantage of semantic caching: queries do not need to be textually identical to reuse an existing response; they only need to be sufficiently similar in meaning according to the embedding model and configured similarity threshold.
3. Conclusion
Semantic caching is a useful optimization for LLM-powered applications. Instead of caching only exact requests, it uses semantic representations such as embeddings to identify requests with similar meanings and potentially reuse their previously generated responses. This can reduce LLM calls, improve response latency, decrease computational requirements, and lower operating costs. However, successful implementation requires careful management of similarity thresholds, cache expiration, personalization, security, data freshness, and model or prompt changes. Semantic caching and KV caching address different parts of the LLM stack. Semantic caching operates primarily at the application or serving layer and attempts to reuse answers across suitable requests. KV caching operates within model inference and avoids recomputing attention information for previously processed tokens. Used together, these techniques can contribute to a more efficient LLM system: semantic caching reduces unnecessary model executions, while KV caching makes the model executions that still occur more computationally efficient.
