So here’s the deal: my wife wanted to generate wallpapers on her laptop, my kid wanted to run some fantasy-style avatars on their phone, and I needed it for myself too. One machine, and nobody was budging. After about an hour and a half of tinkering, we settled on a plan: backend stays ComfyUI, frontend gets a layer of Caddy for basic auth, remote access goes through Tailscale—no need to open ports on the router.
The biggest thing a lot of tutorials skip: single GPU doesn’t have real concurrency. Both ComfyUI and A1111 just queue up tasks one by one. If two people submit at the same time, the second one has to wait—it won’t fail, but it’ll queue. My 3090 does SDXL in about 30-40 seconds per image at 20 steps. If the whole family generates five images each, the queue clears in maybe 10-15 minutes, which is fine for daily use. What really screws things up is when someone gets trigger-happy and runs a 50-step Flux or a batch of 20 images at once—then everyone’s stuck waiting.
VRAM is the hard limit. If you want to load Flux models without offloading, 24GB is where it starts feeling comfortable. 12GB shared setups are painfully slow. Caddy only protects the web UI; the API isn’t isolated. Anyone who knows the address and credentials can still hit the endpoints directly. For family use, that’s fine, but if you’re opening it up to outsiders, you’ll need a beefier auth system.