Running vLLM locally is useful when you want a model behind an HTTP endpoint without sending prompts to a hosted inference service. It isn't a promise that any laptop will work, or that your model will beat another framework by a fixed margin.
This recipe targets vLLM 0.21.0 on NVIDIA/Linux, using its CUDA 12.9 wheel and Python 3.12. That release was published on 2026-05-15, before this guide's original 2026-05-21 date. This correction pins the workflow to that documented release; it doesn't claim to cover every newer version. The commands were checked against documentation and source, not run on a GPU.
Use a Linux x86-64 machine with an NVIDIA GPU of compute capability 7.5 or higher, the release's documented minimum. Use Python 3.12 in a fresh virtual environment; the release documents Python 3.10–3.13, not every future version above 3.10. The selected CUDA wheel is tagged manylinux_2_34_x86_64, so it requires a compatible Linux userspace, including glibc 2.34 or newer.
The precompiled stack is CUDA 12.9 with PyTorch 2.11.0. For this recipe, use a GPU-supported NVIDIA Linux driver at least as new as 575.51.03, the toolkit-driver version NVIDIA lists for CUDA 12.9 GA. This is a conservative recipe baseline, not a claim that no older driver can work through CUDA compatibility mechanisms. Driver support for your particular GPU still matters.
Don't treat an arbitrary installed driver, a working nvcc, or a generic "CUDA 12.1+" requirement as sufficient. nvidia-smi is a useful driver/GPU diagnostic, not proof that this wheel will run. A locally installed CUDA toolkit is not a substitute for matching driver and wheel dependencies; using this prebuilt wheel doesn't require following the separate source-build toolkit/compiler recipe.
The CUDA requirements pin torchaudio 2.11.0 and torchvision 0.26.0 alongside PyTorch. Let the selected wheel install its declared dependencies in the clean environment. Don't reuse an environment with an unrelated PyTorch/CUDA combination: the tagged installation docs warn about binary incompatibility even between differently built packages with the same PyTorch version. A different backend or source build needs its own instructions.
This is not an Apple Silicon or Metal setup. The same tagged docs describe experimental Apple Silicon CPU support requiring a source build and point to the community-maintained vllm-metal plugin for Metal GPU acceleration. Neither is this NVIDIA wheel. A blanket "vLLM can't run on a Mac" would be too broad; don't infer plugin support or performance from this recipe.
These are shell commands for the stated Linux environment. Installation, dependency resolution, imports and GPU execution remain untested here.
python3.12 -m venv .venv-vllm source .venv-vllm/bin/activate python -m pip install --upgrade pip python -m pip install \ 'https://github.com/vllm-project/vllm/releases/download/v0.21.0/vllm-0.21.0%2Bcu129-cp38-abi3-manylinux_2_34_x86_64.whl' \ --extra-index-url https://download.pytorch.org/whl/cu129
That exact CUDA 12.9/x86-64 wheel is listed in the release's asset API. The cp38-abi3 filename is an ABI tag, not an instruction to use Python 3.8. Python 3.12 is within the documented range.
This selects the vLLM release and wheel, but doesn't freeze every transitive dependency. Keep the resolved environment and model revision if you need a reproducible deployment. You also need network access, disk space for packages and model files, and permission to create the environment. Don't share writable runtime caches with untrusted users.
The example uses Qwen/Qwen2.5-0.5B-Instruct, not Llama. Its public model card identifies an Apache-2.0 release. Its configuration declares Qwen2ForCausalLM, an architecture listed in vLLM 0.21.0's supported-model table. That establishes documented architecture compatibility, not a measured result for this exact checkpoint on your machine.
It's a smaller, publicly downloadable example, not a hardware-fit guarantee. Model weights aren't the entire memory budget: context length, KV cache, concurrency and runtime allocations matter too. The model configuration records BF16 as its dtype; this recipe explicitly requests FP16 instead, rather than assuming BF16 support across the stated NVIDIA hardware range. That conversion and model quality have not been tested here.
Use a separate downloader environment so its CLI version doesn't override the inference runtime's dependencies. Hugging Face Hub 1.0.0 uses hf; that release removed huggingface-cli on 2025-10-24, before the original guide's publication. Run the downloader commands below in another shell, in the same project directory.
python3.12 -m venv .venv-hf source .venv-hf/bin/activate python -m pip install 'huggingface_hub==1.0.0' hf download Qwen/Qwen2.5-0.5B-Instruct \ --revision main \ --local-dir ./models/qwen2.5-0.5b-instruct deactivate
main is mutable. For a repeatable run, replace it with a reviewed commit revision and retain that revision with the environment record. No model commit is claimed as pinned here. Download the complete repository, not just the weights: serving also needs the configuration and tokenizer files.
If you choose a gated model instead, a token alone isn't enough. Llama 3.1's repository requires acceptance of its access conditions through the relevant Hugging Face account. Use an account that actually has access, then authenticate the downloader with an appropriately scoped token, such as through hf auth login. Don't paste tokens into shared scripts or shell history. Approval, license obligations and model-specific compatibility remain your responsibility. A model-not-found error can also mean a wrong identifier, missing local files or another download problem; it isn't proof of missing authentication.
Return to the runtime environment and serve the downloaded local directory.
source .venv-vllm/bin/activate vllm serve ./models/qwen2.5-0.5b-instruct \ --served-model-name local-qwen \ --host 127.0.0.1 \ --port 8000 \ --dtype float16 \ --max-model-len 2048
The context limit bounds this example's configuration; it doesn't certify that it fits your GPU. Startup can still fail on driver, dependency or memory constraints. If memory is insufficient, reconsider context, concurrency or the model rather than relying on an undocumented "8B fits in 16GB" assurance.
Binding the HTTP endpoint to 127.0.0.1 keeps that endpoint off other machines' interfaces. It isn't a full security boundary against local users, and doesn't guarantee that every internal dependency binds only to loopback. vLLM's security docs warn that distributed communication can expose internal services even on a single host. Use host/network isolation and appropriate firewall protections; keep model/runtime caches private and trusted.
Don't switch to 0.0.0.0 just to make this local example convenient. Remote serving needs a separately designed authentication, TLS, endpoint allowlisting and network-access setup. The tagged security docs explicitly warn that --api-key does not protect all sensitive HTTP endpoints, so an API key alone isn't that setup. No exposure probe or security incident is claimed here.
Once the server reports that it is ready, this is a documentation-checked request shape, not a request executed for this article:
curl --fail http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local-qwen","messages":[{"role":"user","content":"Explain continuous batching in two sentences."}],"max_tokens":64}'vLLM exposes OpenAI-compatible endpoints, not a universal API drop-in. Its docs list endpoint-specific unsupported or ignored parameters and chat-template requirements. Check your actual client and endpoint. The request's model name must match the served name, and chat requests need the selected tokenizer's appropriate chat template. Changing only a base URL doesn't establish compatibility with every OpenAI payload or framework.
Run this in the vLLM environment as a separate workload. Stop the server first so the two processes aren't competing for GPU memory. This example is documentation/source-checked and GPU-untested. The stray t from the original SamplingParams call has been removed; no syntax checker was executed in this correction turn.
from vllm import LLM, SamplingParams
llm = LLM(
model="./models/qwen2.5-0.5b-instruct",
dtype="float16",
max_model_len=2048,
)
params = SamplingParams(temperature=0.0, max_tokens=64)
outputs = llm.generate(["Explain continuous batching in two sentences."], params)
for output in outputs:
print(output.outputs[0].text)This calls raw text generation. It doesn't apply the model's chat template or reproduce the server's chat endpoint, and shouldn't be presented as an equivalent chat-quality test. Slow startup alone also doesn't identify one cause: loading, profiling, compilation and warmup need to be distinguished using actual logs.
For vLLM 0.21.0, use vllm bench latency. The source defines --num-iters and --num-iters-warmup. The original singular python -m vllm.benchmark.latency module and --num-runs flag aren't the documented interface for this release.
Stop the server before trying this standalone benchmark. This is an unexecuted command example, not retained benchmark evidence:
vllm bench latency \ --model ./models/qwen2.5-0.5b-instruct \ --dtype float16 \ --max-model-len 2048 \ --batch-size 1 \ --input-len 128 \ --output-len 32 \ --num-iters-warmup 1 \ --num-iters 3
The lengths and iteration counts are example inputs, not performance findings. This is a synthetic single-batch latency workload; it doesn't establish real-user throughput, cross-framework superiority or application latency. Useful comparisons need the same hardware, model revision, precision, workload and measurement method, plus retained results.
The earlier "fastest," "2–4x," tokens-per-second, installation-time and build-time promises have been removed because no retained test evidence supports them. That doesn't establish that an earlier author never ran anything; it means those claims can't be verified from the available record. No install, authentication test, GPU inference or benchmark was performed for this correction.
Primary documentation and release-tagged source checked for this correction on 2026-10-07:
main references are mutable.mergedAt value records ingestion, not a replacement publication date; no original publishedAt is supplied.