Is KV Cache in a high dimensional vector space? [D]

I've been doing some research on this question:

At inference time a large part of a model's working memory lives in the KV cache, plus whatever external memory the harness bolts on. I've been poking at the storage-and-retrieval side of this, treating that cache as an index, and what stands out is that it isn't a flat list. It's a structured set of vectors with a navigable geometry, since the keys carry the model's learned sense of what relates to what.

Because that geometry is navigable, attention over it is really a similarity search: the query scores against the stored keys and blends the matching values. Full attention just runs that search exhaustively, scanning everything on every step.

• Full attention effectively searches that geometry exhaustively. Every query scores broadly against the available keys and retrieves from the corresponding values.
• Once you stop treating the KV cache as a flat array and start treating it as a search space, indexing becomes possible.
• That means you can organize old KV into regions, route a query toward likely regions, and only run local attention over a subset.
• The interesting part is that relevance is not uniformly distributed. Queries tend to concentrate on relatively small neighborhoods of old context.
• So the engineering question becomes less “how do I store all of this?” and more “how do I navigate to the right part cheaply?”

I'm new here and don't want to break rules around self promotion or spam so not posting any links atm. Would be cool to get other peoples thoughts on this.

Update: I framed this post badly. I wrote it like I was asking a conceptual question, but I had already built and measured the mechanism. That was my mistake. The actual result is much more specific: on frozen Qwen3.5-2B at 32k, geometric routing cuts physical KV reads by roughly 16–31× while still retrieving the planted long-range needle; window-only and random-routing controls collapse. I’ve put up a minimal runnable demo so people can reproduce it on their own documents.

https://github.com/Regan-Milne/kvspace/tree/main/demo