Spent some time today thoroughly reworking the LiteLLM gateway’s model routing strategy — writing down the full thinking and the final config here.
Why model routing is needed#
My AI infrastructure looks like this:
Telegram → Hermes → LiteLLM gateway → various modelsLiteLLM acts as a unified gateway, exposing a single OpenAI-compatible interface on top of a dozen-plus backend models. Hermes only needs to know one address, ai.XXX.com — it doesn’t need to care which model is actually running underneath. The benefit: models can be swapped, upgraded, or switched at any time, completely transparent to Hermes.
The model pool#
The final configuration:
Paid models (pay-as-you-go, very low cost)
- DeepSeek V3 — strongest for Chinese, $0.27/million input tokens, extremely cost-effective
- Gemini 3 Flash — Google’s latest fast model, multimodal, generous free quota
- Gemini 3.1 Pro — strong reasoning, used as a Pro-tier backup during rate limits
Free NVIDIA NIM models
meta/llama-3.3-70b-instruct— general-purpose English workhorseminimaxai/minimax-m2.5— strong Chinese abilitymoonshotai/kimi-k2.5— strong reasoningz-ai/glm5— Chinese backup (later swapped for the OpenRouter version due to timeouts)
OpenRouter free model pool (final safety net)
minimax/minimax-m2.5:freez-ai/glm-4.5-air:freemeta-llama/llama-3.3-70b-instruct:freenvidia/nemotron-3-super-120b-a12b:free
On-demand paid top-tier model
- Claude Sonnet 4.6 — accessed through OpenRouter, $3/$15 per million tokens, top-tier reasoning backup
The final fallback chain#
Whenever any model fails, there’s a complete degradation chain in place, with a final round-robin pool of four free models as the ultimate safety net.
