Quick Start

This guide takes you from a fresh install to a running model in about five minutes — once with vLLM (Hugging Face Transformers) and once with llama.cpp (GGUF). Each step shows both the TUI path and the headless CLI path, so pick whichever fits your workflow.

Before you start

Make sure you've finished Installation: llmux is on your PATH, and llmux env-check reports Status: OK. The one-line install works from any directory; LLMUX_ROOT is only needed for a manual checkout in a nonstandard location.

Path A — Serve a model with vLLM

vLLM serves Hugging Face Transformers models directly from a model id. We'll use Qwen/Qwen3-0.6B because it's small and downloads fast.

1. Create a profile

quick-setup creates both a profile and a matching config from a single model id — the same flow as the TUI's Quick Setup.

llmux

On the dashboard, press n to open Quick Setup, pick the vLLM backend, enter Qwen/Qwen3-0.6B, and confirm. llmux generates the profile and config, with a live memory estimate to check it fits your GPU.

llmux profile quick-setup Qwen/Qwen3-0.6B --gpu-id 0

This prints Created profile + config: qwen3-0-6b — the profile name is auto-derived from the model id. Confirm it landed:

llmux profile list

2. Start the container

Select the qwen3-0-6b row and press u (or enter → Start). A version picker appears for the vLLM image — pick Local Latest or an explicit release. llmux runs its cross-backend port/GPU conflict check, then streams the Compose output live.

llmux up qwen3-0-6b

llmux up auto-detects the backend and streams docker compose output. On a fresh host it resolves Docker Hub's current stable vLLM release to an explicit version and pulls it if missing. Add --tag v0.20.1 --pull to choose a specific version.

vLLM never starts from :latest

llmux refuses vllm/vllm-openai:latest because its contents drift while the tag name doesn't. The picker resolves stable choices to a semver tag; nightly stays available as an explicit rolling option.

3. Check health

vLLM exposes an OpenAI-compatible server. Once the container is up, hit its /health endpoint on the profile's port (default 8000):

curl http://localhost:8000/health

An empty 200 OK response means the server is ready. You can confirm status across both backends at any time:

The dashboard row shows running/stopped state, port, throughput, model, and backend. Press r to refresh.

llmux ps                    # table of every profile + container status
llmux ps --json --running   # machine-readable, running containers only

4. Watch the logs

Select the profile and press l to open the log stream. Press f to toggle auto-follow.

llmux logs qwen3-0-6b              # follow live (Ctrl-C to stop)
llmux logs qwen3-0-6b --no-follow --tail 100   # snapshot of recent lines

5. Stop the container

Select the profile and press d (or enter → Stop).

llmux down qwen3-0-6b

Path B — Serve a model with llama.cpp

llama.cpp serves GGUF files. A llama.cpp profile needs GGUF metadata (model_file, hf_repo, hf_file) — llmux downloads the file automatically on first start, so there's no manual hf download.

1. Create a profile

quick-setup supports both backends. For llama.cpp, pass the Hugging Face GGUF repository and filename.

Press n on the dashboard, pick the llama.cpp backend, and provide the GGUF repo and file. llmux generates the profile and config.

llmux profile quick-setup \
  --backend llamacpp \
  --hf-repo unsloth/gemma-3-4b-it-GGUF \
  --hf-file gemma-3-4b-it-q4_k_m.gguf \
  --gpu-id 0

The profile name is derived from the repository unless you pass --name.

2. Start the container

Select the gemma-3-4b row and press u. On first start llmux auto-downloads the GGUF file before bringing the container up — you'll see the download progress in the stream.

llmux up gemma-3-4b

llama-server fetches the GGUF inside the container into the shared HF_CACHE_PATH, then serves it through the OpenAI-compatible API.

GGUF download stuck?

For a gated repo, check HF_TOKEN in .env.common and confirm you accepted the model license. For a public repo, verify the hf_repo and hf_file values. No host-side hf CLI is needed.

3. Check health

llama.cpp's server is also OpenAI-compatible and exposes /health on the profile's port (default 8080):

curl http://localhost:8080/health

llmux ps --running    # confirm the llama.cpp container is up

4. Watch the logs

Select the profile and press l.

llmux logs gemma-3-4b

5. Stop the container

Select the profile and press d.

llmux down gemma-3-4b

Next steps

You've now served a model with both backends. To go further:

Profiles — the full profiles.yaml schema, defaults inheritance, and per-backend fields.
Model Configs — tune engine flags per model in config/<backend>/<name>.yaml.
Container Lifecycle — image version policy, the conflict gate, and render-env.
Using the TUI — the dashboard, keyboard reference, and in-TUI editing.
CLI Reference — every subcommand and flag for scripts, agents, and CI.