ECS standard

[rafaelma]

Hello

I apologize if this is not the correct method to reopen a close-rejected issue. ref: #45063
I hope you can reopen it for discussion after reviewing the following comment with additional information ref: #45063 (comment)

"ECS stores data in a nested format, our example will be saved as { "ecs": {"version": "8.4"} }, and the mapping definition of the ECS standard needed by e.g. elasticsearch or opensearch defines also the nested format. It’s important to note that the translation from { "ecs.version": "8.4" } to { "ecs": {"version": "8.4"} } before saving the data according to the ECS standard is done by the Elastic Agent using the decode_json_fields processor in the elastic agent ref: https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html.

For reference, you can check the mapping representation of the ECS standard for our example here: https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/ecs.json The ECS documentation https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html refers to attributes like ecs.version, but this is merely a simplified representation of a JSON nested attribute in the documentation for presentation purposes.

If you use another agent than the elastic-agent (e.g.fluent-bit) to process logs and spring delivers them as e.g. { "ecs.version": "8.4" }, you have to transform this to the nested version { "ecs": {"version": "8.4"} } before you can save it according to the standard.

Spring doesn't deliver the logs according to the ECS standard format and it will only work without problems if you collect these logs with the elastic-agent that will convert them to the right ECS format before sending them to storage."

regards,
Rafael Martinez Guerrero

[nosan]

Hi @rafaelma

Thanks a lot for providing the information. Based on your comment, as I understood the nested JSON is the correct format rather than flatten one. This means I was incorrect in my previous comment: #45063 (comment).

It seems I misunderstood the documentation at https://www.elastic.co/guide/en/ecs/8.17/ecs-ecs.html, as it states that ecs.version is a field name. Additionally, I checked this reference https://github.com/elastic/ecs/blob/8.17/generated/csv/fields.csv, and it also points to ecs.version.

My apologies, @kajh and @snicoll.

Just to clarify the situation according to the ECS standard, does ecs.version actually mean {"ecs":{"version":"8.11"}}, and is the format below the correct approach?

{
  "@timestamp": "2025-04-10T14:35:20.081935Z",
  "log": {
    "logger": "task.gh45063.Gh45063Application",
    "level": "INFO"
  },
  "process": {
    "pid": 25264,
    "thread": {
      "name": "main"
    }
  },
  "service": {
    "name": "gh-45063"
  },
  "message": "Started Gh45063Application in 0.257 seconds (process running for 0.418)",
  "ecs": {
    "version": "8.11"
  }
}

[rafaelma]

Hi @nosan and thank you for your replay.

Yes, your JSON example is correct. I agree that the documentation can be misleading. We encountered multiple issues when we started converting to ECS because the documentation wasn't clear on this matter. It took us some time to understand what was required.

The documentation includes an example here: https://www.elastic.co/guide/en/ecs/current/ecs-custom-fields-in-ecs.html#_proper_names, which shows the JSON representation of http.request.method (reference: https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-request-method).

Importantly, the mapping definitions that implement ECS specify the nested JSON structure. I believe the information in the CSV file (https://github.com/elastic/ecs/blob/8.17/generated/csv/fields.csv) is utilized by some script to create the correct nested JSON. It's much easier to define something like http.request.method and allow automation to convert it to the complex nested JSON format, which can be quite challenging to work with given its complexity.

PS.- I wonder if elasticsearch also has functionality (and not only the elastic-agent) to transform flat fields (e.g. ecs.version) to the right nested representation before saving the data. Opensearch does not this.

regards,

[snicoll]

I apologize if this is not the correct method to reopen a close-rejected issue

No worries. Adding a comment on the closed issue is what you should be doing as we prefer to have the history in one place. I've reopened the original issue.