[Bug]: Presidio + Custom Logger: "AttributeError: 'list' object has no attribute 'usage'"

[nihal1294]

What happened?

I am running the Proxy server in a k8s cluster with a custom logger callback to forward the token usage to another server while also using Presidio for PII masking and here is a part of the config yaml below:

litellm_settings:
    drop_params: True
    json_logs: True
    store_audit_logs: True
    log_raw_request_response: True
    return_response_headers: True
    num_retries: 5
    request_timeout: 200
    callbacks: ["presidio", "custom_logger.push_usage"]

I see a strange behavior where the response_obj in async_log_success_event is updated to type list instead of litellm.types.utils.ModelResponse which results in this error
AttributeError: 'list' object has no attribute 'usage'

for the following code snippet

async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
    print("Response obj type: ", type(response_obj))
    # input_tokens = response_obj.get("usage", {}).get("prompt_tokens", 0)
    # output_tokens = response_obj.get("usage", {}).get("completion_tokens", 0)
    input_tokens = response_obj.usage.prompt_tokens if hasattr(response_obj.usage, "prompt_tokens") else 0
    output_tokens = response_obj.usage.completion_tokens if hasattr(response_obj.usage, "completion_tokens") else 0
    ...

Initially, I tried to access the token counts via .get() as shown in the comments in the code above and later moved to the other way. But in both cases, I get the error shown above.

Strangely, I only see this happen when I am using Presidio callback, and without it, the response_obj type is litellm.types.utils.ModelResponse.

LiteLLM version: 1.44.24

This should be reproducible by using the Presidio callback to get the error, and without it to see no error. And also note the change in the datatype of response_obj.
Is this expected behavior or a bug?

Relevant log output

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/litellm_core_utils/litellm_logging.py", line 1503, in async_success_handler
    await callback.async_log_success_event(
  File "/data/custom_logger.py", line 13, in async_log_success_event
    input_tokens = response_obj.usage.prompt_tokens if hasattr(response_obj.usage, "prompt_tokens") else 0
                                                               ^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'usage'

Twitter

@nihal1294

[krrishdholakia]

I think it might have to do with this -

litellm/litellm/proxy/guardrails/guardrail_hooks/presidio.py

Line 262 in aa64c34

async def async_logging_hook(

which returns

litellm/litellm/proxy/guardrails/guardrail_hooks/presidio.py

Line 303 in aa64c34

return kwargs, responses