High-Throughput LLM ProxyONLINE
OpenLLM Gateway
Open-source self-hostable LLM routing layer managing API keys, costs, budget bounds, and failover.
METRIC 01Unified endpoint compatibility for all providers (OpenAI, Anthropic, Gemini).
METRIC 02API keys encrypted at rest using AES and scoped per user system.
METRIC 03Detailed analytics cost breakdowns and budget caps.
AI platforms suffer during LLM outages. Direct API keys expose organizations to cost overruns, rate limiting, and zero observability.
Attempted Strategy
Originally attempted to store LLM provider API keys as plaintext database rows for quick querying.
Observed Symptom
Failed early security review, exposing the platform to severe key leakage hazards if the database was compromised.
Resolution & Fix
Implemented AES-256-GCM encryption for all stored keys at the data service layer, ensuring keys are only decrypted in-memory during requests.
BunElysiaJSPrisma ORMPostgreSQLTurbo MonorepoReact