Deployment & Inference ★ 19.0k

SGLang

Serving engine built for structured outputs and heavy prefix reuse.

SGLang pairs a fast runtime with RadixAttention for prefix caching, which pays off when many requests share a long system prompt. It also has first-class support for constrained decoding, so JSON-mode agents stay reliable under load.

Best for: Structured output, shared-prefix workloads, agents

Deploy: Self-hostable

#Inference #Serving #GPU #Open Source

Related tools

Deployment & Inference ★ 62.0k

vLLM

High-throughput LLM inference engine with PagedAttention — the default choice when you need to squeeze a GPU.

Self-hosted production inference on NVIDIA/AMD GPUs Read more →

Deployment & Inference ★ 12.0k

Text Generation Inference

HuggingFace's Rust-based serving stack with tensor parallelism and streaming built in.

Multi-GPU serving inside the HuggingFace ecosystem Read more →

Deployment & Inference

TensorRT-LLM

The last few percent of GPU throughput, paid for in build complexity and hardware lock-in.

High-volume serving of a stable model set on fixed NVIDIA hardware Read more →

Deployment & Inference ★ 152.3k

Open WebUI

The front end you hand to users when the model layer is already done.

Internal chat deployments on top of self-hosted models, with real user accounts Read more →