Serving Llms Vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

By Orchestra Research · Official Docs · Date unknown

Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Platforms: linux, macos.

hermes skills install official/mlops/inference/serving-llms-vllm

• #skills
• #optional
• #mlops
• #vllm
• #inference serving
• #pagedattention
• #continuous batching