adelorenzo/konstruct
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(01-02): LLM Backend Pool — LiteLLM Router with Ollama + Anthropic + OpenAI fallback

Browse Source 
- Create llm_pool/router.py: LiteLLM Router with fast (Ollama) and quality (Anthropic/OpenAI) model groups
- Configure fallback chain: quality providers fail -> fast group
- Pin LiteLLM to ==1.82.5 (avoid September 2025 OOM regression in later releases)
- Create llm_pool/main.py: FastAPI service on port 8004 with /complete and /health endpoints
- Add providers/__init__.py: reserved for future per-provider customization
- Update docker-compose.yml: add llm-pool and celery-worker service stubs
This commit is contained in:
Adolfo Delorenzo
2026-03-23 10:03:05 -06:00
parent 0054383be0
commit ee2f88e13b
7 changed files with 370 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
packages/llm-pool/llm_pool/providers/__init__.py Normal file
View File

@@ -0,0 +1,7 @@
"""
Provider-specific customization package.
Reserved for future per-provider configuration, credential handling, and
provider-specific parameter transformations. Currently empty — the LiteLLM
Router in router.py is the sole configuration point.
"""