I've been running local image models on an older laptop recently, and memory behavior surprised me more than raw inference time.
One experiment briefly pushed private memory past 18 GB before I changed the allocation behavior. After terminating the worker/process, it dropped dramatically.
It made me realize how different "the model fits in memory" is from "the whole inference pipeline behaves well in memory."
1 comments