The Problem Calling multiple LLMs involves provider/model-specific configurations. Even if you unify the I/O, you still need a way to handle model/provider-specific edge-cases. We faced this last week when Anthropic told us we’d violated their content policy. Since, we provide our community access to LLMs like Claude-2 etc. through our open-source . proxy server Checking queries through the OpenAI moderations endpoint would slow down queries, so we only wanted to run this on Anthropic’s models. if model in ["claude-instant-1", "claude-2"]: 
  # run moderations check

return litellm.completion(model, messages) But conditional logic like this leads to bugs. We’d faced this exact issue before, and had (abstraction library that simplified LLM API calls). built LiteLLM to solve this problem for us tldr; We want conditional logic on our server. didn’t We a way to control which models/providers it ran this check for. needed The Solution: Config Files Our solution was to have LiteLLM this for us, and control it’s logic through a config file. This , and still allowed us to . handle removed conditional logic from our server code control provider/model specific details This also enabled us to handle other scenarios like context window errors, max tokens, etc. Here’s our complete code: import litellm 
import os 

config = {
    "default_fallback_models": ["gpt-3.5-turbo", "claude-instant-1", "j2-ultra"],
    "model": {
        "claude-instant-1": {
            "needs_moderation": True
        },
        "gpt-3.5-turbo": {
            "error_handling": {
                "ContextWindowExceededError": {"fallback_model": "gpt-3.5-turbo-16k"} 
            }
        }
    }
}

# set env var 
os.environ["OPENAI_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your openai key
os.environ["ANTHROPIC_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your anthropic key

sample_text = "how does a court case get to the Supreme Court?" * 1000
messages = [{"content": sample_text, "role": "user"}]
response = completion_with_config(model="gpt-3.5-turbo", messages=messages, config=config)

print(response) # should be gpt-3.5-turbo-16k Config files currently manage: Prompt logic - picking the right model for a given prompt, as well as trimming a prompt if it’s larger than any available model Fallback logic - letting you set default fallbacks as well as model-specific ones (e.g. the context window error above). Moderations - if a provider (E.g. Anthropic) requires you to moderate your requests Over time, this will handle other model-specific parameters like setting max tokens, prompt formatting, etc. Ideas/suggestions are welcome! Conclusion LiteLLM already simplified calling LLM providers, with a drop-in replacement for the OpenAI ChatCompletion’s endpoint. With config files, it can now let you add new models in production, without changing any server-side code. Overall, LiteLLM is an excellent choice for anyone looking to add non-OpenAI models in production quickly and easily. We are actively trying to grow this project so no matter your skill level we welcome contributions! Open up an issue if you find missing features/bugs or contribute to existing issues. on GitHub if you want to follow our progress as new updates come. Star us

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

LiteLLM Configs: Reliably Call 100+ LLMs

Too Long; Didn't Read

Krrish Dholakia

STORY’S CREDIBILITY

Code License

Guide

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

LiteLLM Configs: Reliably Call 100+ LLMs

Too Long; Didn't Read

Krrish Dholakia

STORY’S CREDIBILITY

Code License

Guide

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES