You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When making a GET request to the /health endpoint using a valid API key that has both user_id and team_id correctly set in the database, the LiteLLM Proxy logs show the error message User API key and team id and user id missing from custom callback. This happens despite the fact that:
No custom callback function (custom_callback) is actually configured or used in the config.yaml.
The same API key works correctly without this error when used for other endpoints like POST /key/health.
The /health endpoint itself successfully returns an HTTP 200 OK status code, indicating the proxy considers itself healthy.
The configuration requires API key authentication (user_api_key_auth: true), key validation (validate_keys: true), request logging (log_requests: true), and spend tracking (disable_spend_logs: false).
This suggests that while the core health check might be succeeding, the internal process related to logging or cost-tracking for the LLM API calls initiated by /health is incorrectly failing to retrieve the user_id and team_id associated with the provided key, leading to a misleading error message.
Steps to Reproduce:
Configure LiteLLM Proxy with a database (database_url) and the following relevant general_settings:
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY # Or some master key
database_url: os.environ/DATABASE_URL
log_requests: true
disable_spend_logs: false
disable_spend_tracking: false
log_level: INFO # or DEBUG
user_api_key_auth: true
validate_keys: true
Ensure an API key exists in the database (e.g., litellm_keys table). This key must have non-NULL values for both the user_id column (e.g., set to "admin") and the team_id column (e.g., set to "default_team").
Start the LiteLLM Proxy server.
Make a GET request to the /health endpoint using the prepared API key in the Authorization header:
curl -X GET 'http://localhost:PORT/health' -H 'Authorization: Bearer API_KEY_WITH_USER_AND_TEAM_ID'
(Replace PORT and YOUR_API_KEY_WITH_USER_AND_TEAM_ID accordingly)
Observe the LiteLLM Proxy server logs.
(Optional Comparison) Make a POST request to /key/health using the same API key:
The GET request to /health should return 200 OK. The server logs should not show the User API key and team id and user id missing from custom callback. error message, as the key is valid, has the required IDs associated in the database, and no custom callback is defined. Logs related to the health check calls themselves might appear, but not this specific error.
Actual Behavior:
The GET request to /health returns 200 OK. However, the server logs do show the error message: User API key and team id and user id missing from custom callback.
The POST request to /key/health (step 6) returns 200 OK and does not produce this error message in the logs.
The key point is the discrepancy between /health and other endpoints (/key/health, likely also /chat/completions etc.) when using the exact same, properly configured API key. This strongly suggests an issue within the specific code path executed by /health when it initiates downstream LLM checks and attempts related logging/cost-tracking, rather than a fundamental problem with the key or database setup. The error message mentioning "custom callback" is particularly confusing as none is configured.
Relevant log output
User API key and team id and user id missing from custom callback.
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
1.67.2
Twitter / LinkedIn details
No response
The text was updated successfully, but these errors were encountered:
What happened?
When making a GET request to the /health endpoint using a valid API key that has both user_id and team_id correctly set in the database, the LiteLLM Proxy logs show the error message User API key and team id and user id missing from custom callback. This happens despite the fact that:
No custom callback function (custom_callback) is actually configured or used in the config.yaml.
The same API key works correctly without this error when used for other endpoints like POST /key/health.
The /health endpoint itself successfully returns an HTTP 200 OK status code, indicating the proxy considers itself healthy.
The configuration requires API key authentication (user_api_key_auth: true), key validation (validate_keys: true), request logging (log_requests: true), and spend tracking (disable_spend_logs: false).
This suggests that while the core health check might be succeeding, the internal process related to logging or cost-tracking for the LLM API calls initiated by /health is incorrectly failing to retrieve the user_id and team_id associated with the provided key, leading to a misleading error message.
Steps to Reproduce:
Configure LiteLLM Proxy with a database (database_url) and the following relevant general_settings:
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY # Or some master key
database_url: os.environ/DATABASE_URL
log_requests: true
disable_spend_logs: false
disable_spend_tracking: false
log_level: INFO # or DEBUG
user_api_key_auth: true
validate_keys: true
Ensure an API key exists in the database (e.g., litellm_keys table). This key must have non-NULL values for both the user_id column (e.g., set to "admin") and the team_id column (e.g., set to "default_team").
Start the LiteLLM Proxy server.
Make a GET request to the /health endpoint using the prepared API key in the Authorization header:
curl -X GET 'http://localhost:PORT/health' -H 'Authorization: Bearer API_KEY_WITH_USER_AND_TEAM_ID'
(Replace PORT and YOUR_API_KEY_WITH_USER_AND_TEAM_ID accordingly)
Observe the LiteLLM Proxy server logs.
(Optional Comparison) Make a POST request to /key/health using the same API key:
Using PowerShell example, adapt if needed
$ApiKey = "Bearer API_KEY_WITH_USER_AND_TEAM_ID"
$Uri = "http://localhost:PORT/key/health"
$Headers = @{ "Authorization" = $ApiKey; "Content-Type" = "application/json" }
Invoke-RestMethod -Uri $Uri -Method POST -Headers $Headers
Observe the server logs again.
Expected Behavior:
The GET request to /health should return 200 OK. The server logs should not show the User API key and team id and user id missing from custom callback. error message, as the key is valid, has the required IDs associated in the database, and no custom callback is defined. Logs related to the health check calls themselves might appear, but not this specific error.
Actual Behavior:
The GET request to /health returns 200 OK. However, the server logs do show the error message: User API key and team id and user id missing from custom callback.
The POST request to /key/health (step 6) returns 200 OK and does not produce this error message in the logs.
Environment:
LiteLLM Version: 1.67.2
Database: PostgreSQL 17
Operating System: Windows Server 2022
Configuration:
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL # Example: postgresql+psycopg2://user:password@host:port/dbname
log_requests: true
disable_spend_logs: false
disable_spend_tracking: false
log_level: INFO
user_api_key_auth: true
validate_keys: true
disable_anonymous_tokens_by_default: true
background_health_checks: false
health_check_details: false
litellm_settings:
drop_params: true
num_retries: 3
request_timeout: 600
allowed_fails: 10
router_settings:
routing_strategy: simple-shuffle
num_retries: 2
timeout: 630
...
Additional Context:
The key point is the discrepancy between /health and other endpoints (/key/health, likely also /chat/completions etc.) when using the exact same, properly configured API key. This strongly suggests an issue within the specific code path executed by /health when it initiates downstream LLM checks and attempts related logging/cost-tracking, rather than a fundamental problem with the key or database setup. The error message mentioning "custom callback" is particularly confusing as none is configured.
Relevant log output
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
1.67.2
Twitter / LinkedIn details
No response
The text was updated successfully, but these errors were encountered: