Plate 02
fast-semantic-cache
A cache for AI calls: repeated questions come back in milliseconds, without paying the API again.
Problem
LLM applications pay for, and wait on, the same answer over and over: repeated or near-identical questions become brand-new API calls.
Solution
A gateway in front of the LLM provider answers from cache when it is safe to: an exact tier keyed on the full payload hash and an optional lexical tier (MinHash/LSH with a deterministic verifier), isolated per client.
Decisions
- 01Exact by default: the approximate tier ships disabled and runs in shadow mode until it proves it does not get answers wrong.
- 02I invalidated my own benchmarks after finding leakage between the calibration and test sets, and repositioned the product based on that.
- 03Requests with tools, images or that depend on the current time never come from cache.
Stack
- Python
- FastAPI
- Redis
- Pydantic v2
- MinHash/LSH
- k6
- pytest