vLLM

Visit product

An open-source inference and serving engine for running large language models efficiently across many architectures and accelerator types.

Links

Built by

Inferact

Cosigned by

No cosigns yet

vLLM develops the serving layer between a language model and the applications using it. Its focus is throughput and memory efficiency, using techniques such as PagedAttention and continuous batching to make better use of hardware. An OpenAI-compatible interface simplifies integration, while ongoing model and hardware support extends where it can run. The project makes serving efficiency a reusable part of the open-model ecosystem.

Built with

PyTorch

The project declares PyTorch as a build requirement and its GPU requirements pin PyTorch; this supports an implementation dependency rather than mere compatibility.

Be the first Cosign

Put your name behind vLLM.

Add it to your Products I use list. You can also share why you use it.

Product name

Story

Product name

Story

Product name

Story

Product name

Story

Product name

Story

vLLM

Built with

PyTorch

Put your name behind vLLM.