HF-native LLM serving
swap to any HF model with one config change
// official site: github.com ↗
AI / LLM · PRO TIER
TGI (HuggingFace)pro
TGI (Text Generation Inference) is HuggingFace's official production LLM serving stack — continuous batching, tensor parallelism, quantization (bitsandbytes, GPTQ, AWQ, EETQ), streaming, and OpenAI-compatible API. The HF ecosystem-native answer to vLLM.
// What it is
A closer look.
TGI (Text Generation Inference) is HuggingFace's official production LLM serving stack — continuous batching, tensor parallelism, quantization (bitsandbytes, GPTQ, AWQ, EETQ), streaming, and OpenAI-compatible API. The HF ecosystem-native answer to vLLM.
If your stack already uses HuggingFace models + Spaces + Inference Endpoints, TGI is the natural self-hosted equivalent.
// Use cases
What it's for.
Concrete scenarios where teams pick TGI (HuggingFace) over the SaaS alternative.
Production-grade inference
continuous batching, streaming
Broad quantization support
bitsandbytes, GPTQ, AWQ, EETQ
OpenAI-compatible API
(newer versions)
Multi-shard inference
tensor parallelism for big models
Tested HF models
TGI tests every major HF model on release
// Who it's for
Built for these teams.
If your team profile matches one of these, TGI (HuggingFace) is a strong fit out of the box.
Teams already on HF stack
Spaces, Inference Endpoints users
AI startups
wanting wide model compatibility
Researchers
needing to swap models frequently
Production teams
valuing HF's testing + maintenance commitment
Hosting providers
offering HF-aligned LLM tier
// Differentiators
Why teams pick TGI (HuggingFace).
When evaluating self-hosted options for this category, here are the dimensions on which TGI (HuggingFace) consistently lands above the alternatives.
- ✓Apache 2.0 — fully open
- ✓HF integration — every new HF model tested on release
- ✓Best quantization breadth — more formats than vLLM
- ✓Streaming — first-class server-sent events
- ✓Simpler model swap — than vLLM (any HF model path works)
- ✓HF backing — long-term maintenance commitment
// Integrations
Connects to.
The stack you'll plug TGI (HuggingFace) into — services, protocols, and adjacent apps in the BluixApps catalog.
◇
OpenAI v1 endpoints
/v1/chat/completions, /v1/completions
◈
TGI native
/generate, /generate_stream
◆
HF Hub
direct model loading
▣
Pair with
LangChain (TGI client), LlamaIndex, OpenWebUI
▦
Multi-shard
--num-shard N for tensor parallelism
▩
Quantization flags
// Adoption & deployment
Notable users & community
- 9k+ GitHub stars
- HuggingFace corporate backing
- Used inside HF Inference Endpoints (production at scale)
- Used by enterprises wanting HF compatibility
- Featured in HF model card "Use in TGI" buttons
What we ship
- Docker (ghcr.io/huggingface/text-generation-inference:latest)
- Default model: meta-llama/Meta-Llama-3.1-8B-Instruct (configurable via /opt/tgi/.env)
- Persistent volume: /opt/tgi/data
- Port 8001 (separate from vLLM if co-installed)
--max-input-length 4096 --max-total-tokens 8192defaults- Install report at
/root/bluixapps/tgi.txt - Quantization options documented
- TGI vs vLLM positioning explained
- HF_TOKEN environment variable for gated models
- GPU pre-flight check via
bluixapps_ensure_nvidia_runtime - Backup hook covers model cache
// Tips & operations
Run it properly.
Operational guidance from running this in production — what to lock down, what surprises people.
// PERFORMANCE
VRAM by model
similar to vLLM (16 GB for 7B fp16, 26 GB for 13B)
// SECURITY
HF_TOKEN
required for gated models
// OPERATIONS
Max input/total tokens
configurable per startup
// RELIABILITY
Quantization choice
AWQ usually best balance
// DEPLOYMENT
Multi-GPU
--num-shard 2 enables tensor parallel
// SCALING
Streaming
SSE format compatible with OpenAI client streaming
// MAINTENANCE
Production
reverse proxy + auth + monitoring (Prometheus metrics built-in)
// COSTS
vs vLLM
TGI for HF-aligned teams, vLLM for raw peak throughput
16384
// min ram (MB)
40
// min disk (GB)
8001
// access port
http
// protocol
pro
// bluixapps tier
// Alternatives in AI / LLM
Compare with
Project resources
Official sitegithub.com ↗