Installation

Install llmux, complete the first-run wizard, and verify that Docker can reach your NVIDIA GPU.

Requirements

llmux orchestrates GPU containers, so it expects a Linux host with NVIDIA hardware and a working Docker + CUDA stack.

Requirement	Notes
Linux + NVIDIA GPU	One or more NVIDIA GPUs with a recent driver installed (`nvidia-smi` should work).
Docker + NVIDIA Container Toolkit	Docker Engine with the NVIDIA Container Toolkit so containers can see the GPU.
Python 3.10+	Managed through uv — uv installs and isolates the tool for you.

Why Docker?

llmux never installs vLLM or llama.cpp on your host. Every profile runs in an official container image, and llmux only renders the Compose files and streams logs. That keeps the two toolchains fully isolated from each other and from your system Python.

GPU passthrough must already work

Before installing llmux, confirm docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi prints your GPUs. If it doesn't, fix the NVIDIA Container Toolkit setup first — llmux can't work around a broken GPU runtime.

Install llmux

The installer clones llmux into ~/.llmux, installs uv if needed, and puts the llmux command on your PATH:

curl -fsSL https://raw.githubusercontent.com/Changroro/llmux/main/install.sh | sh

Once installed, the llmux binary lives in ~/.local/bin/llmux and runs from any directory:

llmux --help

Upgrading and uninstalling

Upgrade: run llmux update, press U in the TUI, or run the installer again.
Uninstall: uv tool uninstall llmux

Run without installing globally

For a manual development install, clone the repo and run uv tool install --editable .. From inside that checkout, uv run llmux also works without a global command. Set LLMUX_ROOT=/path/to/llmux only when a manually installed command must use a different checkout.

Initial setup

Run llmux once. The wizard creates the shared host configuration; Quick Setup creates profiles when you need them:

llmux

1. Complete the shared-settings wizard

On the first launch, llmux asks for the Hugging Face cache path, a legacy model-directory path used by disk views, and an optional Hugging Face token. The token prompt is hidden. The wizard creates the directories and writes the gitignored .env.common file for you.

Manual setup must keep the token private

The wizard writes .env.common with owner-only mode 0600. If you create it manually, run install -m 600 .env.common.example .env.common before editing it; a plain copy can leave HF_TOKEN readable by other local users.

The variables you can set:

Variable	Purpose
`HF_TOKEN`	Hugging Face access token for gated model downloads. Leave empty if you only use public repos.
`HF_CACHE_PATH`	Host path mounted into containers as the Hugging Face cache. Must be an absolute path.
`MODEL_DIR`	Legacy host directory used for old GGUF detection and disk views. Current downloads use `HF_CACHE_PATH`.
`LORA_BASE_PATH`	vLLM only: host path holding LoRA adapters (mounted into the container as `/app/lora`).
`LLAMACPP_IMAGE`	llama.cpp server image tag — defaults to `ghcr.io/ggml-org/llama.cpp:server-cuda`.
`LLAMACPP_REPO_URL`	Source repo URL used by `llmux image build-dev` for llama.cpp dev builds. Defaults to the upstream `ggml-org/llama.cpp` repo.
`LLAMACPP_BRANCH`	Branch cloned for llama.cpp dev builds. Defaults to `master`.
`TZ`	Container timezone, e.g. `Asia/Seoul`.

HF_CACHE_PATH must be absolute

env-check rejects a relative HF_CACHE_PATH. Use a full path such as /home/$USER/.cache/huggingface so the bind mount resolves correctly.

Gated models need HF_TOKEN

Public models download fine with an empty HF_TOKEN. For gated repos, use a token with access and accept the model license on Hugging Face. Both backends download inside their containers, so no host-side hf CLI is required.

2. Create a profile with Quick Setup

Press n in the dashboard, or use the CLI. Both vLLM and llama.cpp are supported, and llmux writes profiles.yaml plus the matching backend config:

llmux profile quick-setup Qwen/Qwen3-0.6B --gpu-id 0

profiles.yaml remains the single source of truth. For a manual setup, run install -m 600 profiles.example.yaml profiles.yaml and then edit it by hand; profile environment values can contain secrets.

The template defines a defaults block per backend plus one example profile for each engine:

version: 1

defaults:
  vllm:
    port: 8000
    gpu_id: "0"
    tensor_parallel_size: 1
    enable_lora: false
  llamacpp:
    port: 8080
    gpu_id: "0"

profiles:
  - name: qwen3-0-6b
    backend: vllm
    config_name: qwen3-0-6b       # → config/vllm/qwen3-0-6b.yaml
    model_id: Qwen/Qwen3-0.6B
    gpu_id: "0"

  - name: gemma-3-4b
    backend: llamacpp
    config_name: gemma-3-4b       # → config/llamacpp/gemma-3-4b.yaml
    model_file: gemma-3-4b-it-q4_k_m.gguf
    hf_repo: unsloth/gemma-3-4b-it-GGUF
    hf_file: gemma-3-4b-it-q4_k_m.gguf

Any field you omit from a profile inherits from defaults.<backend>. You don't need to hand-write the example profiles — the Quick Start generates a fresh profile for you with llmux profile quick-setup.

You never hand-write .env files

At launch, each selected profile is rendered to .runtime/<backend>/<name>.env and passed to docker compose --env-file. profiles.yaml is the whole interface — the runtime renders are gitignored.

See the Profiles guide for the full schema.

3. Verify the installation

llmux env-check validates .env.common and reports the resolved project paths. Run it from inside the repo (or with LLMUX_ROOT set):

llmux env-check

A healthy setup ends with Status : OK. If something is missing or unsafe, the command lists each issue and exits non-zero — for example a missing or group/world-accessible .env.common, or a relative HF_CACHE_PATH. HF_TOKEN is optional.

You can also confirm the GPU runtime is visible to llmux:

llmux gpu          # nvidia-smi summary, one row per GPU

Machine-readable checks

env-check, gpu, and every list/show command accept --json for clean piping into scripts and agents. Token JSON reports only whether a token is set and its length, never its value.

Next steps

With llmux installed and configured, head to the Quick Start to create your first profile and serve a model in about five minutes.