hckrnws
LensVLM: Compressing long context as images, expanding only relevant pages
(huggingface.co)
89
10
by victormustar
by victormustar
The thing I’d love to do with a system like this is train it to be KV cache ordering independent (ie permutation invariant at the page level). Basically each page’s KV cache should be understandable by the model in any ordering - which would allow you to go one step further and treat the KV cache of the vision encoded page as the chunk for the model to reason over.
Then all these zoom in for more detail tricks will extend naturally.
We are living in the age where people can compose a model to do almost anything you can describe, literally. I personally have accrued a rough 'gang' of models I use to check each other before they wreck each other, and nothing gets past a repo review cycle. All I am now is the final builder, distributable packager, for a fleet of little purpose-built byte-chuckers.
Am I correct in assuming that a principle use case for LensVLM is in the training side of things .. ?
Maybe to save on doing the client-side layout and rendering?
I'm reminded of the MSPaint IDE:
Since every character in text shares compressible ligaments with every other character, maybe? Or, more finite, every pixel representing a ligature component in a character can be compressed against other pixels in the path set.
I think, at scale, this is very important - folks training models on near-petabyte sized corpus would have cause to want to use this in pipelines, I imagine ..
There's no reason why we shouldn't be able to expose active relevant information that is only relevant for the next request: current agents running, time, etc.
There's also no reason why we shouldn't have a cheaper lossy input which uses way less bytes per token - see deepseek flash 4.1.
[dead]
[dead]