Accurate and Efficient Approach for Calculating OTel Log Record Size

[liustve]

What are you trying to achieve?

We are trying to add a new condition in the OTLP BatchLogProcessor to export batches of logs based on the data size of the unserialized and uncompressed LogRecord and are looking for a recommendation for:

1. How could I accurately calculate the size of a LogRecord?
2. How could I efficiently calculate the size of a LogRecord?

What did you expect to see?

We expected the OpenTelemetry SDK to provide a standardized method for calculating LogRecord sizes, which would be valuable for efficient batching and preventing request size limit violations. Currently, the OTel specification lacks this capability, forcing implementers to create custom Log Data size estimation logic.

Additional context.

JavaScript and Python implementations lack precise methods to determine in-memory LogRecord sizes. We have a heuristic implementation that estimates the size of a LogRecord based on the log body, which typically comprises most of the data payload and then adding a large buffer window to guesstimate the size of the other components of the log.

However, this poses a second problem: efficiency.

The body of a LogRecord object is currently defined self-referentially and represents a directed graph:
https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-api/src/opentelemetry/util/types.py#L20

This recursive nature makes precise size calculation challenging, as the structure could contain cycles or deeply nested references that are difficult to measure without trading performance efficiency for accuracy.

[svrnm]

@open-telemetry/javascript-maintainers @open-telemetry/python-maintainers can you take a look since your languages have been mentioned in the examples? Is this something you see similar as @liustve that a spec update would help with this, or is this something you see highly language specific and can be solved without a spec change?