Deploy guide
Deploy vLLM on Vast.ai
Serve an OpenAI-compatible chat completions API from a Vast.ai GPU, with one public URL, health checks and automatic replacement when an instance dies. Cheapest card that fits 7B–8B models in FP16: RTX 3090 (24 GB) at $0.10/hr — about $73/month if it ran 24/7.
Quickstart
| Image | vllm/vllm-openai:latest |
|---|---|
| Command arguments | --model meta-llama/Llama-3.1-8B-Instruct |
| Container port | 8000 |
| Health endpoint | /health |
| Minimum VRAM | 24 GB |
- Create a deployment with the vllm/vllm-openai:latest image. In the dashboard choose New deployment, paste vllm/vllm-openai:latest, and set the container port to 8000.
- Pick the GPU pool. Select SKUs with at least 24 GB of VRAM — the cheapest right now is RTX 3090 (24 GB) at $0.10/hr.
- Set the health endpoint and start. Use /health as the health endpoint so a worker only receives traffic once the container answers. Start the deployment and open the public URL it gets.
Things that bite
- The image already exposes port 8000 and answers /health once the model is loaded; use /health as the health endpoint so a worker only takes traffic when it can actually serve.
- Model weights download on first boot. Add an HF_TOKEN environment variable for gated repositories, and give the deployment a longer health grace period than usual (a cold start can take minutes).
- For larger models raise the GPU class: 70B-class weights in FP16 do not fit one 80 GB card, so pick a pool of 80 GB SKUs and set max GPUs per deployment accordingly.
- The cheapest card that fits is not always the fastest: an L4 holds an 8B model in FP16, but a newer consumer or L40S-class card answers faster — pick the pool by what you optimise for, not only by the lowest price.
What it costs
GPU time is Vast.ai's own price, billed per second to your Vast.ai account; CUDAnetes adds a flat subscription (free plan or $19/mo on Pro) and takes no markup. A fleet running 24/7 for a month costs $73 per card at the cheapest live offer, before the subscription.
See the live GPU price index for current rates, or the Vast.ai vs RunPod comparison for the fixed-price alternative.
Questions
What does it cost to run vLLM on Vast.ai?
GPU time is billed by Vast.ai per second at the marketplace rate shown on this page; orchestration is free on the Free plan and $19/mo on Pro. An 8B model on one 24 GB card typically costs a few cents per hour at the low end of the market.
Do I need my own Vast.ai account?
Yes. CUDAnetes uses your Vast.ai API key, so instances are rented on and billed to your own account, and CUDAnetes takes no markup on GPU time.
What happens when the GPU dies mid-request?
The health tick marks the worker unhealthy, stops routing new requests to it, and provisions a replacement; the deployment keeps serving from the healthy workers meanwhile.
Deploy it now
Free plan: one deployment with up to five GPUs, no credit card. Any Docker image, one public URL.