ai-statistics 采集指标异常 || ai-statistics Acquisition indicator abnormality

[zjun-sudo]

Why do you need it?

• 开启流式响应后，ai-statistics 采集不到 input_token、output_token等值

1. 未开启流式访问，ai-log值
   

2. 开启流式访问，ai-log 值

Other related information

1. 流式请求访问

2. 非流式请求访问

3. 插件配置

    attributes:
      - apply_to_log: true
        apply_to_span: true
        key: model
        value: model
        value_source: response_body
      - apply_to_log: true
        apply_to_span: true
        key: input_token
        value: usage.prompt_tokens
        value_source: response_body
      - apply_to_log: true
        apply_to_span: true
        key: output_token
        value: usage.completion_tokens
        value_source: response_body
      - apply_to_log: true
        key: question
        value: messages.@reverse.0.content
        value_source: request_body
      - apply_to_log: true
        key: answer
        rule: append
        value: choices.0.delta.content
        value_source: response_streaming_body
      - apply_to_log: true
        key: consumer
        value: x-mse-consumer
        value_source: request_header

---

Why do you need it?

• After turning on streaming response, ai-statistics cannot collect input_token, output_token and other values

1. Streaming access is not enabled, ai-log value
   

2. Turn on streaming access, ai-log value

Other related information

1. Streaming request access

2. Non-streaming request access

3. Plug-in configuration

    attributes:
      - apply_to_log: true
        apply_to_span: true
        key: model
        value: model
        value_source: response_body
      - apply_to_log: true
        apply_to_span: true
        key: input_token
        value: usage.prompt_tokens
        value_source: response_body
      - apply_to_log: true
        apply_to_span: true
        key: output_token
        value: usage.completion_tokens
        value_source: response_body
      - apply_to_log: true
        key: question
        value: messages.@reverse.0.content
        value_source: request_body
      - apply_to_log: true
        key: answer
        rule: append
        value: choices.0.delta.content
        value_source: response_streaming_body
      - apply_to_log: true
        key: consumer
        value: x-mse-consumer
        value_source: request_header

[johnlanni]

看上去后端并没有返回token usage信息

[zjun-sudo]

看上去后端并没有返回token usage信息

插件的源码里流式和非流式采集token都依赖这个方法，所以流式的返回如果没有 token usage 的信息，插件就计算不了这些 token 值吗？

[imp2002]

所以流式的返回如果没有 token usage 的信息，插件就计算不了这些 token 值吗？

是的，各家模型的 tokenizer 有差别，统计只能依赖接口服务里给我们返回 usage 信息，流式情况下需要设置这个参数才会返回 usage 信息，可以添加完试试插件能不能正常工作

stream_options={"include_usage": True}

ref: https://cookbook.openai.com/examples/how_to_stream_completions#4-how-to-get-token-usage-data-for-streamed-chat-completion-response