Skip to content

ECS standard #45132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rafaelma opened this issue Apr 10, 2025 · 3 comments
Closed

ECS standard #45132

rafaelma opened this issue Apr 10, 2025 · 3 comments
Labels
status: duplicate A duplicate of another issue

Comments

@rafaelma
Copy link

rafaelma commented Apr 10, 2025

Hello

I apologize if this is not the correct method to reopen a close-rejected issue. ref: #45063
I hope you can reopen it for discussion after reviewing the following comment with additional information ref: #45063 (comment)

"ECS stores data in a nested format, our example will be saved as { "ecs": {"version": "8.4"} }, and the mapping definition of the ECS standard needed by e.g. elasticsearch or opensearch defines also the nested format. It’s important to note that the translation from { "ecs.version": "8.4" } to { "ecs": {"version": "8.4"} } before saving the data according to the ECS standard is done by the Elastic Agent using the decode_json_fields processor in the elastic agent ref: https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html.

For reference, you can check the mapping representation of the ECS standard for our example here: https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/ecs.json The ECS documentation https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html refers to attributes like ecs.version, but this is merely a simplified representation of a JSON nested attribute in the documentation for presentation purposes.

If you use another agent than the elastic-agent (e.g.fluent-bit) to process logs and spring delivers them as e.g. { "ecs.version": "8.4" }, you have to transform this to the nested version { "ecs": {"version": "8.4"} } before you can save it according to the standard.

Spring doesn't deliver the logs according to the ECS standard format and it will only work without problems if you collect these logs with the elastic-agent that will convert them to the right ECS format before sending them to storage."

regards,
Rafael Martinez Guerrero

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Apr 10, 2025
@nosan
Copy link
Contributor

nosan commented Apr 10, 2025

Hi @rafaelma

Thanks a lot for providing the information. Based on your comment, as I understood the nested JSON is the correct format rather than flatten one. This means I was incorrect in my previous comment: #45063 (comment).

It seems I misunderstood the documentation at https://www.elastic.co/guide/en/ecs/8.17/ecs-ecs.html, as it states that ecs.version is a field name. Additionally, I checked this reference https://github.com/elastic/ecs/blob/8.17/generated/csv/fields.csv, and it also points to ecs.version.

My apologies, @kajh and @snicoll.

Just to clarify the situation according to the ECS standard, does ecs.version actually mean {"ecs":{"version":"8.11"}}, and is the format below the correct approach?

{
  "@timestamp": "2025-04-10T14:35:20.081935Z",
  "log": {
    "logger": "task.gh45063.Gh45063Application",
    "level": "INFO"
  },
  "process": {
    "pid": 25264,
    "thread": {
      "name": "main"
    }
  },
  "service": {
    "name": "gh-45063"
  },
  "message": "Started Gh45063Application in 0.257 seconds (process running for 0.418)",
  "ecs": {
    "version": "8.11"
  }
}

@rafaelma
Copy link
Author

rafaelma commented Apr 10, 2025

Hi @nosan and thank you for your replay.

Yes, your JSON example is correct. I agree that the documentation can be misleading. We encountered multiple issues when we started converting to ECS because the documentation wasn't clear on this matter. It took us some time to understand what was required.

The documentation includes an example here: https://www.elastic.co/guide/en/ecs/current/ecs-custom-fields-in-ecs.html#_proper_names, which shows the JSON representation of http.request.method (reference: https://www.elastic.co/guide/en/ecs/current/ecs-http.html#field-http-request-method).

Importantly, the mapping definitions that implement ECS specify the nested JSON structure. I believe the information in the CSV file (https://github.com/elastic/ecs/blob/8.17/generated/csv/fields.csv) is utilized by some script to create the correct nested JSON. It's much easier to define something like http.request.method and allow automation to convert it to the complex nested JSON format, which can be quite challenging to work with given its complexity.

PS.- I wonder if elasticsearch also has functionality (and not only the elastic-agent) to transform flat fields (e.g. ecs.version) to the right nested representation before saving the data. Opensearch does not this.

regards,

@snicoll
Copy link
Member

snicoll commented Apr 10, 2025

I apologize if this is not the correct method to reopen a close-rejected issue

No worries. Adding a comment on the closed issue is what you should be doing as we prefer to have the history in one place. I've reopened the original issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: duplicate A duplicate of another issue
Projects
None yet
Development

No branches or pull requests

4 participants