feat(01-02): LLM Backend Pool — LiteLLM Router with Ollama + Anthropic + OpenAI fallback
- Create llm_pool/router.py: LiteLLM Router with fast (Ollama) and quality (Anthropic/OpenAI) model groups - Configure fallback chain: quality providers fail -> fast group - Pin LiteLLM to ==1.82.5 (avoid September 2025 OOM regression in later releases) - Create llm_pool/main.py: FastAPI service on port 8004 with /complete and /health endpoints - Add providers/__init__.py: reserved for future per-provider customization - Update docker-compose.yml: add llm-pool and celery-worker service stubs
This commit is contained in:
10
packages/llm-pool/llm_pool/__init__.py
Normal file
10
packages/llm-pool/llm_pool/__init__.py
Normal file
@@ -0,0 +1,10 @@
|
||||
"""
|
||||
konstruct-llm-pool — LiteLLM Router service.
|
||||
|
||||
Exposes a FastAPI HTTP API for LLM completions with multi-provider routing
|
||||
and automatic fallback. Import the FastAPI app from main.py.
|
||||
"""
|
||||
|
||||
from llm_pool.router import complete, llm_router
|
||||
|
||||
__all__ = ["complete", "llm_router"]
|
||||
Reference in New Issue
Block a user