Local models with vLLM

This page provides instructions for installing vLLM in an isolated uv virtual environment on the HPC using CUDA 13.3, GCC 13.3, and uv.

Start a job on a GPU node#

If you're submitting an interactive job in Open OnDemand, ensure that Number of GPUs is set to at least 1.

If you're on an HPC login node, run an interactive job on a GPU node:

1
2
3

# If you have leased GPU resources on the HPC, substitute your queue name for 'backfill2'
srun -A backfill2 --time=03:00:00 --gres=gpu:a4500:1 --cpus-per-task=8 --pty /bin/bash
# Wait for the job to start...

Note

vllm is not supported on nodes that have GTX 1080 Ti GPUs, so you will need to specify the GPU type in the --gres option. Refer to our GPU jobs page for a list of models currently supported.

Configure the environment#

# Ensure environment modules are loaded 
module load python-uv gnu/13 cuda/13.3 webproxy

cd ~
uv venv vllm-env --python 3.12 --seed
source vllm-env/bin/activate

Install vLLM#

1	`uv pip install vllm --torch-backend auto`

Verify the installation:

# When you run this (it will take a few seconds)...:
python -c "import vllm, torch; \
print('vllm', vllm.__version__); \
print('torch', torch.__version__); \
print('cuda available:', torch.cuda.is_available()); \
print('device:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NONE')"

# ...you should see output similar to the following:
vllm 0.27.1
torch 2.13.0+cu130
cuda available: True
device: NVIDIA RTX A4500

Use vLLM#

If you're starting a new session on the HPC, ensure you're connected to a GPU node:

# Ensure necessary modules are loaded
module load python-uv gnu/11.2.1 cuda/12.1 webproxy

# Ensure uv environment is activated
cd ~
source vllm-env/bin/activate

This is a basic "smoke test" using a small model:

Create a file called smoke.py:

cat > smoke.py <<'EOF'
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")   # small model for a quick test
params = SamplingParams(temperature=0.7, max_tokens=1024)
out = llm.generate(["Explain what an HPC scheduler does in one sentence."], params)
print(out[0].outputs[0].text)
EOF

Then run it:

1	`python smoke.py`

Note

The output from running smoke.py is very verbose. The first time this runs, it will download the Qwen model to your home directory. It may take up 3 minutes; subsequent runs should be slightly faster

For more information about using vLLM, refer to the documentation on the vLLM website.