Tuning a semantic cache for an LLM proxy

Each last user message is turned into an embedding — a vector of a few hundred to a few thousand dimensions representing its meaning. The incoming question is compared to already-cached vectors via cosine similarity: a score between -1 and 1, where 1 means "identical meaning".

If the best score clears a configured threshold, the cache returns the stored answer without ever calling the LLM provider. Below it, it's a miss: the request goes through normally, and its response feeds the cache for next time.

A threshold set too low (permissive) surfaces answers to different questions — a false hit, invisible to the user who gets a plausible but wrong answer. A threshold set too high (strict) almost never fires, and the cache pays for itself in nothing.

There's no universal threshold: it depends on traffic variance. A support agent answering highly repetitive questions ("what's my order status?") tolerates a permissive threshold. An agent handling open-ended, nuanced requests needs a strict one, or it will conflate neighboring but distinct intents.

Exposing a raw threshold ("0.93") to someone who doesn't know their embedding distribution is asking them to guess. A better interface offers named profiles — strict, permissive, aggressive — each mapped to a threshold calibrated upfront for a given embedding model.

The catch: these thresholds don't travel across embedding models. 0.95 on `text-embedding-3-small` means nothing for a different model — the geometry of the vector space changes. Any calibration has to be redone if the embedding model changes.

A response carrying tool calls should never be cached as-is: it corresponds to an action taken in a specific context, and replaying it for a semantically close but different question can trigger the wrong action.

A streamed response the client disconnected from mid-generation shouldn't be cached either — that would serve a truncated response as complete on the next matching request.

Comparing a request against thousands of cached entries gets expensive if done naively. Two practical levers: cap how many candidates are scored per lookup (most recent first — conversational traffic's temporal locality makes old entries progressively less relevant), and vectorize the comparison as a single matrix product instead of a loop.