CLI Reference

llmux                      # launch the interactive TUI (default)
llmux tui                  # launch the TUI explicitly
llmux top [PROFILE]        # btop-style monitor in a plain terminal (no TUI)
llmux <group> <command>    # headless control
llmux <alias>              # top-level shortcut (see below)
llmux --version            # print the installed version and exit

Command map

Common concepts

top

The btop-style system monitor, rendered in a plain terminal without the Textual TUI. Good for low-bandwidth SSH or dumb terminals. Every GPU is shown always — util, memory, temperature, power, PCIe rx/tx — whether or not anything is running, plus a panel per running model (throughput and KV-cache braille graphs, cache-hit, requests, latency percentiles: TTFT/E2E p50/p95/p99 and phases). With no argument it shows every running model; pass a profile name to narrow it to that one. Press q or Ctrl+C to exit. The same monitor is reachable inside the TUI with the t key.

llmux top [PROFILE]

Backend detection

Profile names are globally unique across both backends. Most commands resolve the matching backend (vllm or llamacpp) automatically by scanning profiles.yaml:

• If the name exists, its backend is used.
• Pass --backend / -b to require a specific backend explicitly.
• If it exists in neither, the command fails with a clear usage error.

--backend always wins when supplied.

JSON output

Commands that produce structured data accept --json to emit a JSON payload (UTF-8, 2-space indent) instead of the default padded table or text. Commands with --json support:

Exit codes

Lifecycle commands propagate the underlying docker / docker compose exit code. Ctrl-C while streaming (up, logs, build-dev) exits with 130. system env-check exits 1 when .env.common is missing or has issues.

container — container lifecycle

llmux container up|prepare|down|logs|ps|benchmark|stats|render-env …

Start, stop, inspect, benchmark, monitor, and re-render profile containers. Every container command except the grouped namespace itself is also exposed through the corresponding top-level alias.

container up

Start a profile's container. Streams docker compose output to stdout.

llmux container up PROFILE [OPTIONS]

# Start a vLLM profile (official image, highest local versioned tag)
uv run llmux container up qwen3-0-6b

# Start it from a locally-built dev image
uv run llmux container up qwen3-0-6b --dev --tag main-20260515

# One-off: ignore a stale pinned image_tag and use the compose default
uv run llmux up qwen3-0-6b --default-image

# Start a llama.cpp profile with an explicit backend
uv run llmux up gemma-3-4b --backend llamacpp

# llama.cpp dev image from a personal fork
uv run llmux up gemma-3-4b --dev \
    --repo-url https://github.com/me/llama.cpp.git --branch master

container prepare

Download the model and render the runtime files — without starting the server. Everything up does up to docker compose up: renders the profile env (plus llama.cpp's command override), makes sure the image is on disk (pulling it if not), then downloads the weights into the HF cache from HF_CACHE_PATH. No container is left running and no GPU is touched.

llmux container prepare PROFILE [OPTIONS]

uv run llmux prepare qwen3-0-6b
uv run llmux prepare gemma-3-4b --backend llamacpp

container down

Stop a profile's container.

llmux container down PROFILE [OPTIONS]

uv run llmux container down qwen3-0-6b

container logs

Stream container logs (Ctrl-C to stop following).

llmux container logs PROFILE [OPTIONS]

# Follow live logs
uv run llmux logs qwen3-0-6b

# Print the last 50 lines and exit
uv run llmux container logs qwen3-0-6b -n 50 --no-follow

container ps

List profiles and their container status across backends.

llmux container ps [OPTIONS]

Table columns: backend, profile, status, port, gpu, container, model. JSON rows additionally include a boolean running field.

Status preserves Docker lifecycle and health: healthy, unhealthy, starting, running, paused, restarting, removing, created, exited, dead, unknown, or stopped. Model labels come from the vLLM config's model (then profile model_id) or the llama.cpp config's alias (then hf_repo, then model_file).

uv run llmux ps
uv run llmux ps --running --json
uv run llmux container ps -b vllm

container render-env

Re-render .runtime/<backend>/<profile>.env from profiles.yaml. Useful after editing profiles.yaml by hand. Prints the rendered path(s).

llmux container render-env [PROFILE] [OPTIONS]

# Re-render a single profile
uv run llmux render-env qwen3-0-6b

# Re-render every llama.cpp profile
uv run llmux container render-env --backend llamacpp

container benchmark · bench

Benchmark a running container's generation speed via /v1/chat/completions. A warmup call is discarded, then --runs measured calls report the median tok/s (a single cold call conflates model warmup with steady-state decode).

llmux bench PROFILE [OPTIONS]

uv run llmux bench qwen3-0-6b --runs 5
uv run llmux bench qwen3-0-6b --json

container stats · stats

Live token throughput (tok/s) for every running container, polled from its Prometheus /metrics endpoint. Rates are deltas between successive samples, so the first line appears one interval in. An unavailable metric reads n/a; an HTTP request failure is reported as an error rather than displayed as normal missing data.

llmux stats [OPTIONS]

uv run llmux stats                 # live, Ctrl-C to stop
uv run llmux stats --once --json   # one machine-readable sample

profile — profile CRUD + quick-setup

llmux profile list|show|new|edit|rename|clone|delete|quick-setup …

Profiles live in profiles.yaml at the repo root — the single source of truth for both backends. Fields not applicable to a backend stay at their defaults.

profile list

List all profiles across (or within) backends.

llmux profile list [OPTIONS]

Columns: backend, name, port, gpu_id, config, model.

uv run llmux profile list
uv run llmux profile list -b vllm --json

profile show

Show a profile's full record. Environment keys containing TOKEN, SECRET, PASSWORD, PASSWD, API_KEY, PRIVATE_KEY, CREDENTIAL, or AUTH keep their names but replace their values with set and length metadata in both text and JSON output. Other environment values remain visible for automation compatibility.

llmux profile show NAME [OPTIONS]

uv run llmux profile show qwen3-0-6b --json

profile new

Create a new profile entry in profiles.yaml. The name must match ^[a-z0-9][a-z0-9_-]*$ — lowercase only, because docker compose project names are lowercase-only and a cross-backend profile must validate on both sides. Creation fails if the name already exists in either backend.

llmux profile new NAME [OPTIONS]

# vLLM profile
uv run llmux profile new qwen3-8b --backend vllm --model Qwen/Qwen3-8B \
    --gpu-id 0,1 --port 8000 --set VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1

# llama.cpp profile
uv run llmux profile new gemma-3-4b --backend llamacpp --port 8080 \
    --model-file gemma-3-4b-it-q4_k_m.gguf \
    --hf-repo unsloth/gemma-3-4b-it-GGUF --hf-file gemma-3-4b-it-q4_k_m.gguf

profile edit

Edit fields of an existing profile. Only the options you pass change; everything else is left untouched.

llmux profile edit NAME [OPTIONS]

# Pin a llama.cpp profile to a locally-built dev image
uv run llmux profile edit gemma-3-4b --image-tag llamacpp-dev:mtp_main

# Add an env var, remove another
uv run llmux profile edit qwen3-8b --set VLLM_ATTENTION_BACKEND=FLASHINFER --unset OLD_VAR

profile rename

Rename a profile. The profile's container must be stopped — it is named after the profile, so renaming under a running container would orphan it.

An unset config link is pinned to the old name, so the profile keeps resolving to the config file it already used. An unset container name follows the new profile name.

llmux profile rename OLD NEW [OPTIONS]

uv run llmux profile rename qwen3-8b qwen3-8b-fp8

profile clone

Copy a profile to a new name, keeping the same linked config. The clone reuses the source's port — change it before running both.

llmux profile clone SRC DST [OPTIONS]

uv run llmux profile clone qwen3-8b qwen3-8b-tuned

profile delete

Delete a profile (and optionally its linked config YAML).

llmux profile delete NAME [OPTIONS]

uv run llmux profile delete old-profile --with-config --yes

profile quick-setup

Create a profile and a config in one step — mirrors the TUI's "Quick Setup" for both backends. vLLM takes a model id; llama.cpp takes --hf-repo and --hf-file. If --name is omitted, a name is derived from the model or repository; collisions get a numeric suffix.

llmux profile quick-setup [MODEL] [OPTIONS]

uv run llmux profile quick-setup Qwen/Qwen3-8B --gpu-id 0,1 --gpu-mem 0.92

uv run llmux profile quick-setup --backend llamacpp \
  --hf-repo unsloth/gemma-3-4b-it-GGUF \
  --hf-file gemma-3-4b-it-q4_k_m.gguf

config — config (YAML) CRUD

llmux config list|show|new|edit|clone|rename|flags|from-recipe|delete …

Per-model engine configs live in config/<backend>/<name>.yaml:

• vLLM — config/vllm/<name>.yaml: model: + gpu-memory-utilization: plus arbitrary vllm serve flags.
• llama.cpp — config/llamacpp/<name>.yaml: a flat dict of llama-server flags.

New config names (new, clone destination, rename destination, and recipe --name) must match ^[a-z0-9][a-z0-9_-]*$. Read/edit/delete and clone-source lookups also accept legacy on-disk names matching ^[a-zA-Z0-9][a-zA-Z0-9._-]*$, so an older or hand-written file can still be inspected and removed. If the same name exists in both backends, disambiguate with --backend.

config list

List config files.

llmux config list [OPTIONS]

Columns: backend, name, model, params (number of keys in the YAML).

config show

Print a config YAML.

llmux config show NAME [OPTIONS]

config new

Create a new config YAML. For vLLM, the file is seeded with model: and gpu-memory-utilization:; for llama.cpp it starts empty. --set values are YAML-parsed, so --set foo=2 stores an int and a bare --set bar stores true.

llmux config new NAME [OPTIONS]

# vLLM config
uv run llmux config new qwen3-8b --model Qwen/Qwen3-8B --gpu-mem 0.9 \
    --set max-model-len=8192 --set enable-prefix-caching

# llama.cpp config
uv run llmux config new gemma-3-4b --backend llamacpp \
    --set ctx-size=8192 --set n-gpu-layers=999 --set flash-attn

config edit

Patch fields in an existing config.

llmux config edit NAME [OPTIONS]

uv run llmux config edit qwen3-8b --set max-model-len=16384 --unset enable-prefix-caching

# Toggle a flag off without losing its value, then back on
uv run llmux config edit qwen3-8b --disable enforce-eager
uv run llmux config edit qwen3-8b --enable enforce-eager

A disabled parameter is stored as a # llmux:disabled <key>: <value> comment marker — inert to the server, visible to llmux. Hand-written # comments in the file are preserved across edits. See Toggling parameters on / off.

config clone

Clone SRC config to DST within the same backend. Useful when profile quick-setup --copy-from isn't enough — for example, when you want to clone a tuned config to a new name without creating a profile in the same step. Routes through the backend's canonical serializer so the resulting YAML is byte-identical to a TUI-edited config.

llmux config clone SRC DST [OPTIONS]

uv run llmux config clone qwen3-0-6b qwen3-0-6b-experiment

config flags

Inspect the flags supported by the actual vLLM or llama.cpp image. Discovery is cached by image identity, so a moving tag is re-read after its image changes.

llmux config flags [OPTIONS]

uv run llmux config flags --profile qwen3-8b
uv run llmux config flags --backend llamacpp --image ghcr.io/me/llama:v1 --json

config from-recipe

Create a vLLM config from the model's official vllm-project/recipes recipe. The TUI shows an interactive review window (see Import a vLLM recipe); this headless form picks a variant with --variant, or auto-picks the highest-quality one that fits the detected GPU.

llmux config from-recipe MODEL_ID [OPTIONS]

uv run llmux config from-recipe Qwen/Qwen3-32B --list
uv run llmux config from-recipe Qwen/Qwen3-32B --variant fp8 --feature reasoning
uv run llmux config from-recipe Qwen/Qwen3-32B --variant awq --json

# Borrow the base model's recipe for a quantized checkpoint
uv run llmux config from-recipe cpatonn/Qwen3-32B-AWQ \
    --recipe-from Qwen/Qwen3-32B --variant awq

# Merge into a config you already have
uv run llmux config from-recipe Qwen/Qwen3-32B --name my-model --merge

config rename

Rename a config YAML and repoint every profile that referenced it. Refused while any referencing profile's container is running.

llmux config rename OLD NEW [OPTIONS]

uv run llmux config rename qwen3-0-6b qwen3-0-6b-fp8

config delete

Delete a config YAML.

llmux config delete NAME [OPTIONS]

image — Docker image inventory + dev build

llmux image list|pull|remove|build-dev …

image list

List local (and optionally remote) Docker images.

llmux image list [OPTIONS]

Columns: source (local / local-dev / remote), repository, tag, size, created.

uv run llmux image list --dev
uv run llmux image list --backend llamacpp --dev
uv run llmux image list --remote --json

image pull

docker pull <repo>:<tag> — surfaces raw docker output and exit code.

llmux image pull TAG [OPTIONS]

uv run llmux image pull v0.20.1
uv run llmux image pull nightly

image remove

Wraps docker rmi <ref> so you can drop a local image without leaving the CLI. Use --force to drop images that are still referenced by a stopped container.

llmux image remove REF [OPTIONS]

uv run llmux image remove llamacpp-dev:fork-master
uv run llmux image remove vllm/vllm-openai:v0.20.0 --force

image build-dev

Build a <backend>-dev:<tag> image from source, streaming docker build output. The build pipeline (git clone/update → docker build → image labels) is shared; the tags applied are <prefix>:<custom_tag or branch-YYYYMMDD> and <prefix>:<branch> (a stable alias to the latest build).

• vLLM → vllm-dev:<tag> (target vllm-openai, docker/Dockerfile)
• llama.cpp → llamacpp-dev:<tag> (target server, .devops/cuda.Dockerfile)

llmux image build-dev [OPTIONS]

# vLLM dev image from an unmerged PR branch
uv run llmux image build-dev --backend vllm --branch my-feature \
    --repo-url https://github.com/me/vllm.git

# llama.cpp dev image, pinned to a single GPU arch for a fast build
uv run llmux image build-dev --backend llamacpp --branch master --cuda-arch 89

See Dev Builds for the full workflow, and Architecture for how the pipeline works.

system — system info & env validation

llmux system gpu|mem-estimate|disk|env-check …

system gpu

Print an nvidia-smi summary, one row per GPU. Also available as the top-level alias llmux gpu.

llmux system gpu [OPTIONS]

Columns: index, name, memory_used, memory_total, utilization, temperature. Prints (no GPUs detected — is nvidia-smi installed?) if none are found.

uv run llmux gpu
uv run llmux system gpu --json

system mem-estimate

Estimate a Hugging Face model's VRAM footprint via hf-mem — the same engine the TUI's memory estimator uses. Reads HF_TOKEN from .env.common (or the environment) for gated models.

llmux system mem-estimate MODEL_ID [OPTIONS]

uv run llmux system mem-estimate Qwen/Qwen3-8B
# → ~17.2GB (model: 15.3GB + KV: 1.9GB)

JSON output sets status to ok, over, or error. A malformed, non-finite, or non-positive GPU memory reading produces an error row with total_gb, ratio, and over set to null. Unless another known GPU is over capacity, aggregate any_over is also null. Text output labels the row UNKNOWN; either format exits 1 when any GPU memory reading is unknown.

system disk

Show backend storage and df -h usage. vLLM reports its configured host Hugging Face cache; if the cache has not been created yet, the command measures the nearest existing ancestor and still succeeds. Its JSON output distinguishes this with hf_cache_exists: false and identifies the measured path as df_target. llama.cpp reports its model directory, GGUF inventory, and HF GGUF cache. The JSON success and failure payloads both identify the selected backend.

llmux system disk [OPTIONS]

uv run llmux system disk --backend llamacpp
# Model dir : /home/me/Project/llmux/models
# GGUF files: 2 (total 17.1 GB)
# df ...: used 583G, avail 285G (68%)

uv run llmux system disk --backend vllm --json

system env-check

Validate .env.common and report key paths. Checks that the file exists, is owner-only rather than group/world-accessible, and that required paths such as HF_CACHE_PATH are set and absolute. HF_TOKEN is optional, and the runtime injects the selected vLLM version at start time. Also available as llmux env-check.

llmux system env-check [OPTIONS]

Exits 0 when everything checks out, including when only non-fatal warnings are present, and exits 1 when .env.common is missing or any issue is found. Human output prints non-fatal messages under Warnings: before Status: OK. JSON preserves them in warnings, keeps issues empty, and reports status: ok. Text output masks HF_TOKEN; JSON exposes only set and length, never the token value.

uv run llmux env-check
uv run llmux system env-check --json

See .env.common Reference for every variable this command validates.

update — check for a newer llmux

Ask GitHub whether this checkout is behind the latest Release, and update to it. llmux already checks on every interactive start (only a failed lookup backs off for 15 minutes), so this command is for asking on demand — right after a release, or from a script.

llmux update [OPTIONS]

llmux update --check          # "llmux is up to date (v2.6.0)."
llmux update --check --json   # machine-readable status
llmux update -y               # pull + refresh the installed tool, no prompt

Exit codes: 0 when the state is known (up to date, or reported/applied), 1 when the check could not be completed (offline, rate-limited, not a git checkout, shallow clone) or the update failed. An update is refused — with the reason — when the checkout is not on a clean main; pull it yourself there.

If the Git fast-forward succeeds but refreshing the installed uv tool fails, the checkout has already advanced but the update still exits 1. The error prints the manual refresh command; the TUI reports the same partial failure and stays open.

The TUI has the same action on U.

Top-level aliases

For convenience — and so agents can write llmux up <profile> directly — the most-used commands are mirrored at the top level:

These aliases share the exact arguments, options, and defaults of their canonical commands.