Skip to content

Local models with vLLM

This page provides instructions for installing vLLM in an isolated uv virtual environment on the HPC using CUDA 12.1, GCC 11.2.1, and uv.

Start a job on a GPU node#

If you're submitting an interactive job in Open OnDemand, ensure that Number of GPUs is set to at least 1.

If you're on a HPC login node, run an interactive job on a GPU node:

1
2
3
# If you have leased GPU resources on the HPC, substitute your queue name for 'backfill2'
srun -A backfill2 --time=03:00:00 --gres=gpu:a4500:1 --cpus-per-task=8 --pty /bin/bash
# Wait for the job to start...

Note

We recommend not connecting to nodes that have GTX 1080 Ti GPUs, so you will need to specify the GPU type in the --gres option. Refer to our GPU jobs page for a list of models currently supported.

Configure the environment#

1
2
3
4
5
6
# Ensure environment modules are loaded 
module load python-uv gnu/11.2.1 cuda/12.1 webproxy

cd ~
uv venv vllm-env --python 3.12 --seed
source vllm-env/bin/activate

Install vLLM#

Note

We do not yet have CUDA 13 on the HPC (it's coming soon!), so you cannot install vLLM using the --torch-backend auto option. Instead, you will have to perform additional steps to install a CUDA 12.1-compatible version of vLLM.

1
2
3
export VLLM_VERSION=0.24.0    # <- Check the vllm website for the current version
curl -s "https://api.github.com/repos/vllm-project/vllm/releases/tags/v${VLLM_VERSION}" \
 | grep -oE 'vllm-[^"]+\.whl' | sort -u

In the output, find the bare filename line that includes cu129, manylinux, and x86_64:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
vllm-0.24.0-cp38-abi3-manylinux_2_28_aarch64.whl
vllm-0.24.0-cp38-abi3-manylinux_2_28_x86_64.whl
vllm-0.24.0+cpu-cp38-abi3-manylinux_2_34_aarch64.whl
vllm-0.24.0+cpu-cp38-abi3-manylinux_2_34_x86_64.whl
vllm-0.24.0+cu129-cp38-abi3-manylinux_2_28_aarch64.whl
vllm-0.24.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl    # <---- This is the one we want
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0%2Bcpu-cp38-abi3-manylinux_2_34_aarch64.whl
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0%2Bcpu-cp38-abi3-manylinux_2_34_x86_64.whl
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0%2Bcu129-cp38-abi3-manylinux_2_28_aarch64.whl
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0-cp38-abi3-manylinux_2_28_aarch64.whl
vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0-cp38-abi3-manylinux_2_28_x86_64.whl

Copy and paste that line into an environment variable and install it with uv:

1
2
export VLLM_BACKEND=vllm-0.24.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
uv pip install "https://github.com/vllm-project/vllm/releases/download/v$VLLM_VERSION/$VLLM_BACKEND" --torch-backend=cu129

Verify the installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# When you run this...:
python -c "import vllm, torch; \
print('vllm', vllm.__version__); \
print('torch', torch.__version__); \
print('cuda available:', torch.cuda.is_available()); \
print('device:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NONE')"

# ...you should see output similar to the following:
vllm 0.24.0
torch 2.11.0+cu129
cuda available: True
device: NVIDIA RTX A4500

Use vLLM#

If you're starting a new session on the HPC, ensure you're connected to a GPU node:

1
2
3
4
5
6
# Ensure necessary modules are loaded
module load python-uv gnu/11.2.1 cuda/12.1 webproxy

# Ensure uv environment is activated
cd ~
source vllm-env/bin/activate

This is a basic "smoke test" using a small model:

Create a file called smoke.py:

1
2
3
4
5
6
7
8
cat > smoke.py <<'EOF'
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")   # small model for a quick test
params = SamplingParams(temperature=0.7, max_tokens=1024)
out = llm.generate(["Explain what an HPC scheduler does in one sentence."], params)
print(out[0].outputs[0].text)
EOF

Then run it:

1
python smoke.py

Note

The output from running smoke.py is very verbose. The first time this runs, it will download the Qwen model to your home directory. It may take up 3 minutes; subsequent runs should be slightly faster

For more information about using vLLM, refer to the documentation on the vLLM website.