Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Allow configurable_clientside_parameters for switching b/w finetuning models easily #5843

Closed
krrishdholakia opened this issue Sep 23, 2024 · 2 comments · Fixed by #5842 or #5858
Labels
enhancement New feature or request

Comments

@krrishdholakia
Copy link
Contributor

krrishdholakia commented Sep 23, 2024

The Feature

in the model_list, having a variable (like clientside_configurable_parameters with a default value of [] that contains a list of which litellm parameters can be overwritten by a client. This preserves the current behavior of no values overwritten by a client, but allows for new behavior of configuring variables that a client can override; this also gives the level of control we would want

model_list:
   - model_name: "anthropic/*"
     litellm_params:
       model: "anthropic/*"
#       api_base: "<default_api_base>" # [OPTIONAL]
       clientside_configurable_parameters: ["api_base", "base_url"]

Motivation, pitch

Allows devs to switch between finetuned models easily (changing api base), and allows admin to define which models + which params can have this behaviour (prevents security issues like https://huntr.com/bounties/4001e1a2-7b7a-4776-a3ae-e6692ec3d997)

Twitter / LinkedIn details

cc: @nathanluskey

@krrishdholakia krrishdholakia added the enhancement New feature or request label Sep 23, 2024
@krrishdholakia
Copy link
Contributor Author

We should probably be specific about what parameters we mean, updating to be configurable_clientside_auth_parameters

@krrishdholakia
Copy link
Contributor Author

Updated to configurable_clientside_auth_params based on testing, it felt more intuitive

krrishdholakia added a commit that referenced this issue Sep 24, 2024
* feat(auth_utils.py): enable admin to allow client-side credentials to be passed

Makes it easier for devs to experiment with finetuned fireworks ai models

* feat(router.py): allow setting configurable_clientside_auth_params for a model

Closes #5843

* build(model_prices_and_context_window.json): fix anthropic claude-3-5-sonnet max output token limit

Fixes #5850

* fix(azure_ai/): support content list for azure ai

Fixes #4237

* fix(litellm_logging.py): always set saved_cache_cost

Set to 0 by default

* fix(fireworks_ai/cost_calculator.py): add fireworks ai default pricing

handles calling 405b+ size models

* fix(slack_alerting.py): fix error alerting for failed spend tracking

Fixes regression with slack alerting error monitoring
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant