LiteLLM (YC W23)

Thrilled to launch support for using Gemini pass through API through LiteLLM (YC W23) Proxy https://lnkd.in/dxbfyk6f Use Google AI Studio Node JS with LiteLLM Proxy Added support for sending `tags` for /gemini/ pass through route: https://lnkd.in/dp3fQYM6 (fix) auth checks - Add regex matching for?models?on virtual keys / teams h/t Sandesh G. https://lnkd.in/dJGTFQCs (UI fix) UI does not reload when you login / open a new tab (redis fix) - fix?AbstractConnection.__init__() got an unexpected keyword argument 'ssl' fix) pass through endpoints - run logging async + use thread pool executor for sync logging callbacks

Thrilled to launch support for using Google Vertex Node SDK with LiteLLM (YC W23) Proxy https://lnkd.in/dtwkkV_j Added support for cost tracking for streaming + non streaming Vertex pass through request (Perf / latency improvement) improve pass through endpoint latency to ~50ms https://lnkd.in/duurQdvp (code quality improvement) added lint check for blocking creating httpx clients during LLM calling https://lnkd.in/duurQdvp Docs: Added doc on usage with Vertex JS SDK https://lnkd.in/dtwkkV_j PR: allow sending?`tags`?on vertex js pass through requests

LiteLLM (YC W23) v1.52.14 - 89% faster performance when using Google AI Studio through LiteLLM, Median latency overhead is ~50ms https://lnkd.in/dJf5mg2f (h/t Yardena Meymann, Oz Elhassid) (code quality) add linting check to ban creating?AsyncHTTPHandler?during LLM calling https://lnkd.in/dx8hyJQF (Feat) Allow passing?litellm_metadata?to pass through endpoints?https://lnkd.in/d35dRit3 (fix) pass through - allow internal users to access /anthropic (feat) Add usage tracking for streaming?/anthropic pass through routes

Excited to launch provider budget routing on LiteLLM (YC W23) v1.52.11 (h/t Michael Choi and Yunfeng Bai) ?? Start here: https://lnkd.in/dG_nuvsg Use this to set budgets for LLM Providers - example $100/day for OpenAI, $100/day for Azure ?? New prometheus metric - litellm_provider_remaining_budget_metric to track remaining budget for a provider https://lnkd.in/dTpHqpEi feat - add?fireworks_ai/qwen2p5-coder-32b-instruct PR: add?custom_llm_provider?in LiteLLM DB SpendLogs https://lnkd.in/dmDJDvVB (provider budget routing improvements) Raise error when all deployments are filtered out because of lack of budget

LiteLLM (YC W23) v1.52.10 - Launching improved Dev Docs organization for LiteLLM https://lnkd.in/gj-7Qy_S - New Docs Guides Section - Usage for Vision, Audio, Prompt Caching use cases https://lnkd.in/gVZ4bDGG - Categorized all docs for LiteLLM Proxy: https://lnkd.in/grjsM5sk - (docs) simplify left nav names + use a section for?making llm requests https://lnkd.in/gKQY3tZe - Section on Logging Features for LiteLLM Proxy: https://lnkd.in/gaM9Xesr (fix) httpx handler - bind to ipv4 for httpx handler (docs) add doc string for all /key, /team, /user and /customer endpoints (feat) add support for doc_url and redoc_url https://lnkd.in/ggWZQEYy (UI) - allow editing key alias on Admin UI?

LiteLLM (YC W23) v1.52.10 - Give access to 100+ LLMs, use all in the OpenAI API format (fix) Azure AI Studio - using?image_url?in content with both text and image_url (h/t Emerson Gomes) (Admin UI) - Remain on Current Tab when user clicks refresh (h/t Yunfeng Bai) (Admin UI) Allow editing key alias on Admin UI?(h/t Michael Choi) (patch) using image_urls with?vertex/anthropic?models (h/t Emerson Gomes) (Doc) Add section on what is stored in the DB + Add clear section on key/team based logging (h/t Yardena Meymann) https://lnkd.in/dS-X5EZY (docs) add doc string for /key/update

Thrilled to launch support for using all Vertex AI Model Garden Models on LiteLLM (YC W23) v1.52.9 https://lnkd.in/drWVBRdT (h/t Suresh Siva) Now use fine tuned llama 3.1 models through LiteLLM (fix) Fix - don't allow?viewer?roles to create virtual keys (h/t Michael Choi) (feat) Use?custom prefixes when storing virtual keys in AWS secret manager (h/t Yunfeng Bai) (fix) Fixes for Role based access checks - Added testing coverage for all Roles and Admin UI routes + fixes for Proxy Admin Viewer permissions (feat) - add bedrock/us.llama 3.1?models

??Today we're launching support for Vertex Model Garden Embedding models on LiteLLM (YC W23) v1.52.9 (h/t Suresh Siva) https://lnkd.in/d_Rf4Nty Now you can use custom embedding models like Snowflake `Snowflake/snowflake-arctic-embed` through LiteLLM ?? Added Cost tracking for?vertex_ai/imagen3 (h/t Mandar Phadate) ??? Fixes for using Non-Root Docker Container on LiteLLM (h/t Rajat Vig) https://lnkd.in/dnA9kM7K ? [Feature]: json_schema in response, support for Anthropic https://lnkd.in/dQKBED7G ? [Feature]: Stop swallowing up AzureOpenAi exception responses in litellm's implementation for a BadRequestError

LiteLLM (YC W23) v1.52.7 Excited to launch storing Virtual Keys in Amazon Web Services (AWS) Secrets Manager (h/t Yunfeng Bai) https://lnkd.in/gsDxQqw7 (feat) New Helm Pre Sync Hook to manage DB migrations https://lnkd.in/gVE4X9ye (feat) added support for bedrock/stability.stable-image-ultra-v1:0 (docs) add instructions on how to run docker builds for new contributors https://lnkd.in/g2MrUJuK (h/t Rajat Vig) (stable release) v1.52.5-stable is out now https://lnkd.in/gwfKNEY3

LiteLLM (YC W23) v1.52.6 - Added support for using Json mode with Anthropic https://lnkd.in/dQKBED7G (feat) Add cost tracking for Azure Dall-e-3 Image Generation https://lnkd.in/dsQm-mWC (code quality + test coverage improvement) Use base class to ensure basic image generation tests pass - track cost, basic image generation request with response validation (feat) helm hook to sync db schema https://lnkd.in/dUfJWqHw (fix proxy redis) Add redis sentinel support ?? Docs fix - update instructions for using LiteLLM with hypercorn https://lnkd.in/dptCB55K