Skip to main content

vLLM

How to serve a large language model with vLLM on a CGC GPU as an OpenAI-compatible API: load a model from Hugging Face or a volume, run the server and call the API.

vLLM is a fast and easy-to-use library for LLM inference and serving.

Choose and load a model​

Pick any model you want to serve. In this example we'll use Llama‑3.2‑3B‑Instruct (~7 GiB).

Create a volume​

Please be sure to mount a volume that is large enough to accommodate all the files. If you plan to create a model repository volume, provision at least 200 GB of storage. You can expand the volume later if needed.

cgc volume create -s 200 -sc <storage_class> models

Use HuggingFace repository​

If the chosen model is available on the HF repository, you can just pass its name when running a new vLLM instance.
Please be aware that you need to mount the previously created volume as there are constraints on ephemeral storage. Also, you don't want to download the model every time you create a new vLLM instance.

For further steps go to How to run section.

Put the model in the volume​

The quickest way is to do it manually using the filebrowser GUI. You can run:

# Create if it doesn't exist
cgc compute filebrowser create

# Mount the new volume
cgc volume mount models -t filebrowser

To check the app token and URL under which it's available:

cgc compute list -d

When logging into filebrowser's web interface, use username admin and the app token as a password.

In the filebrowser, download all the model files from Hugging Face and drag them into the volume.

warning

Ensure you download the model in Hugging Face's original PyTorch format, not GGUF.

The vLLM image also ships the Hugging Face CLI, so from any running instance that has the volume mounted you can download the whole repository straight into it:

hf download meta-llama/Llama-3.2-3B-Instruct --local-dir /media/models/Llama-3.2-3B-Instruct

How to run​

Huggingface model​

To run an HF model, you need to make sure that you are allowed to download the model.
You'll need to generate a new HF Token. Some models are restricted, so you'll need to go to the model card website and accept the terms and conditions.

cgc compute create vllm-openai -n <name> -g <gpu_count> -gt <gpu_type> -c <cpu_count> -m <ram> -v <volume_name> -e hf_token=<HF_TOKEN> -- --model=<MODEL_NAME>

To run a Bielik example: Before applying this command, change the HF token and go to the Bielik model card to accept the terms and conditions.

cgc compute create vllm-openai -n bielik-26 -e hf_token=<HF_TOKEN> -g 1 -gt NVIDIA-RTX-A5000 -c 2 -m 24 -v models -- --model=speakleash/Bielik-11B-v2.6-Instruct-FP8-Dynamic

CGC automatically uses your volume to download the model. The weights are written to --download-dir in Hugging Face's cache layout, for example /media/models/huggingface/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/<revision>/.

warning

That cache directory is not a model directory you can serve later. Only the weight files are written to the volume; config.json, the tokenizer and the rest of the metadata stay in the container's own cache and disappear with the instance. Passing the snapshot path to --model fails with Invalid repository ID or local directory specified. To serve a model from the volume, put a complete copy of the repository there first - see Put the model in the volume.

Your model​

Point --model at a directory on the volume that holds a complete copy of the model repository - config.json, the tokenizer files and the weights - such as the one created in Put the model in the volume.

cgc compute create -n <name> -c 4 -m 8 -g 1 -gt NVIDIA-RTX-A5000 -v models vllm-openai -- --model=/media/models/Llama-3.2-3B-Instruct

Whatever volume you pass with -v is mounted under /media/models, so this path does not depend on the volume's name.

Parameters​

  • -n - instance name that will be used in URLs
  • -c - CPU cores, no more than 4 is needed in most cases
  • -m - memory, no more than 8 GiB is needed in most cases
  • -g, -gt - GPU count and type. Be sure that vRAM is sufficient for the model.
  • -v - volume with models

Required arguments​

  • --model <path> - path to the model directory (e.g., /media/models/Llama-3.2-3B-Instruct) or a Hugging Face repository ID (e.g., speakleash/Bielik-11B-v2.6-Instruct).

Default arguments​

  • --host 0.0.0.0 - binds the server to all available IP addresses
  • --port 8000 - sets the port for the server to listen on
  • --api-key <APP_TOKEN> - sets the API key for the server, which is shown in cgc compute list -d command output
  • --download-dir /media/models/huggingface - specifies the directory where models are downloaded, defaults to /media/models/huggingface
  • --max-model-len 4096 - the maximum sequence length for the model, defaults to 4096
  • --safetensors-load-strategy prefetch - enables prefetching of safetensors files, which speeds up model loading by fetching weight files in parallel

Optional environment variables​

  • HF_TOKEN - Your Hugging Face token. This is required if you are downloading a gated model.
  • TRUST_REMOTE_CODE - If provided, allows the execution of remote code from the model's repository. By default, remote code is not trusted.
  • TENSOR_PARALLEL_SIZE - The number of GPUs to use for tensor parallelism, allowing you to run models that are too large for a single GPU.
  • MAX_NUM_BATCHED_TOKENS - The maximum number of tokens in a single batch.
  • VLLM_USE_MODELSCOPE - Set to true to use ModelScope for model loading.
  • VLLM_USE_PRECOMPILED - Set to true to use precompiled kernels, which can speed up startup time.
  • VLLM_USE_V1 - Specifies the version of the vLLM engine to use.
  • UV_TORCH_BACKEND - Defines the backend for UV Torch.

Using -e you can pass all the environmental for the vllm serve command. A list of all possible arguments and environment variables can be found here.

API usage​

Your endpoint lives at:

https://<name>.<NAMESPACE>.cgc-waw-01.comtegra.cloud/

Fetch the API token:

cgc compute list -d

Example call​

curl -H "Authorization: Bearer $API_TOKEN" \
     -H "Content-Type: application/json" \
     -X POST \
     -d '{
           "model": "/media/models/Llama-3.2-3B-Instruct",
           "messages": [
             {"role":"user","content":"Hello!"}
           ]
         }' \
     https://<name>.<NAMESPACE>.cgc-waw-01.comtegra.cloud/v1/chat/completions

Embedding models​

The same template can serve an embedding model. The example below uses intfloat/multilingual-e5-large (~2.2 GiB, 1024 dimensions):

cgc compute create vllm-openai -n <name> -c 2 -m 8 -g 1 -gt NVIDIA-RTX-A5000 -v <volume_name> -- --model=intfloat/multilingual-e5-large --runner pooling --max-model-len 512
  • --runner pooling switches the server into embedding mode.
  • --max-model-len 512 repeats the default argument with a lower value. e5 models only support 512 positions, and the server refuses to start with the default 4096.

The server then serves the model under /v1/embeddings:

curl -H "Authorization: Bearer $API_TOKEN" \
     -H "Content-Type: application/json" \
     -X POST \
     -d '{
           "model": "intfloat/multilingual-e5-large",
           "input": ["query: what is CGC?"]
         }' \
     https://<name>.<NAMESPACE>.cgc-waw-01.comtegra.cloud/v1/embeddings

The same call with the OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url="https://<name>.<NAMESPACE>.cgc-waw-01.comtegra.cloud/v1",
    api_key="APP_TOKEN")

res = client.embeddings.create(model="intfloat/multilingual-e5-large",
    input="query: what is CGC?")

print(len(res.data[0].embedding))

The e5 models expect a prefix on every input: query: for search queries and passage: for the documents you index.