The Old Pain
Two years ago, self-hosting a 7B parameter model meant: CUDA version conflicts, Python dependency hell, ONNX conversion failures with cryptic error messages, and inference servers that crashed under load. The barrier to entry was not technical knowledge — it was patience.
What Changed
VLLM matured into a genuinely reliable inference server. Ollama made local model management feel like package management. HuggingFace's Transformers library stabilized its API. Container images with pre-built CUDA dependencies became widely available. None of these individually solved the problem, but together they lowered the floor significantly.
What Still Sucks
Multi-GPU inference remains finicky. Model versioning across environments is still largely manual. And the moment you need to serve more than one model simultaneously, you are back to capacity planning and resource allocation decisions that require real expertise. The tools got better, but self-hosting AI is still not a "set and forget" operation.