For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.
But isn't what everyone want almost the complete opposite of this? For smaller LLMs you can already run them with consumer hardware, sometimes consumer hardware 2x, but it's the rest of the stuff where distributed inference could actually "democratize" access, as almost no one has 500GB VRAM available at home, or whatever the latest SOTA local models take nowadays.
It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals
Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...
> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.
Run LLMs at home, BitTorrent‑style - https://news.ycombinator.com/item?id=37546810 - Sept 2023 (125 comments)
Peer to peer, consumers get to route their request to a host with compatible model. Consumers have to also contribute GPU but it does not have to be equal - I have not thought through the fairness part. Perhaps initially it starts with "create your friends group and have access to all the host nodes".
---
[0]: https://www.gnus.ai/
You could even hand out some kind of digital 'currency' to the people proportional to the amount of work their doing!
In order to be as decentralized as Bitcoin and related PoW mechanisms, a proof-of-work mechanism should have
* the ability to create an unbounded number of instances of the problem deterministically from numeric seeds
* the ability to scale difficulty up and down using a difficulty parameter
* in a way where anyone can easily confirm that a given problem instance corresponds to a given seed, and anyone can easily confirm that a given solution correctly solves a given problem instance
I'm not aware of a proof-of-useful-work mechanism that meets these criteria. (I'm also not aware of an argument that it's impossible to have one. Zero-knowledge proofs might actually go far toward making it possible in the future. but it's kind of complicated.)
[dead]
[dead]
If you run a local model at a good pace, you're emitting tokens at, what, 60 tokens/sec? That's, very roughly, 300 bytes/sec. You can get that speed with dialup from the 80's! And when it takes a half second just to ingest the input tokens (to get to the part where you start generating the reply), you're not gonna care about even a crazy 250ms latency.
Unless you think that the model should be downloaded on-the-fly in response to a user request, there's zero reason for an LLM to need any sort of fast connection.
Also, this is the reason I want to start with trust based groups only - invite people you already know. I have friends who have Macs with 32GB or 48GB unified memory but sending prompts to them is not a out of the box thing.
* typo
[dead]