hckrnws
Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint
(modal.com)
91
18
by charles_irl
by charles_irl
I thought the memory snapshotting part in particular was clever since most container based systems don't bother (VM/firecracker based ones can use UFFD and call it a day), but by having emulated syscalls you can actually do single-process restore pretty well.
I am a bit dubious of the use of fuse (though it clearly works well!), and I wonder if ublk (what I ended up using) might alleviate some of the pain/magic in fuse tuning. I'd personally also be looking at forking gvisor to take a memfd which you enable UFFD on for the page loading (I have some firecracker patches where I do the same). It's nice because you can optimistically push pages, rather than waiting for the requests to come in. The series of three codesandbox blog posts are good background reading.
Major AI Labs all have secured their own compute in the form of hardware, data center, and power generation. That means their resource pool is fixed, and they can do all sorts of tricks to pre-load, pre-allocate, etc... to improve on inference latency.
Cold start is usually a solution for "cloud" environment when your pool is flexible, and you only pay for what you use. Its effectiveness lowered in bare-metal settings as folks do not care about scaling up and down as much.
So my question is: who is this for? AWS and GCP running Anthropic models?
I'm currently planning to deploy using Amazon SageMaker, but a cold start takes a whopping ~9 minutes: 6 minutes for instance provisioning + 3 minutes for PyTorch initialization. My Docker image is ~14 GB, and the weights are a few GB. How long would it take to cold start this configuration on Modal?
SageMaker's performance makes it pretty much useless without many warm instances around (= tens of thousands of dollars per month), because users won't be happy if they have to randomly wait 9 minutes
Edit: "The solution is to disaggregate the container launcher (runc for Docker, runsc for gVisor) from the container image delivery" is exactly what I've done! I've not built a lazy FUSE on top of it (yet! except for cache mounts in BuildKit), but it's on my TODO list. I guess I'm mainly curious what stops bytes from being shared in your case.
[dead]
[dead]
Here's my 2cents: improve cold starts also means utilizing resources more effectively.
From cloud providers to end users - every ms both adds up and translates to additional waste of electricity/hardware and costs.
[deleted]
There are a few limitations with snapshotting, e.g. it generally fails when using multiple GPUs, which we document here: https://modal.com/docs/guide/memory-snapshots.
FROM some/image RUN pip install torch==2.7.1
and
FROM another/image RUN pip install torch==2.7.1
will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.
[deleted]