How to avoid dropped log events when encountering type mismatches?

[hagen93]

Hi,

I'm trying to configure the Elasticsearch sink to record all our log events but have stumbled upon a conundrum. Since Elasticsearch rejects new events if the data type of the log event differs from previous events I'm missing some log events here and there. In my specific case ASP.NET and gRPC is both trying to use the property "StatusCode", but ASP.NET uses it as an integer, while gPRC uses it as a string.

How can I ensure that none of the log events gets dropped? I was thinking the best way to avoid this would be to ensure all properties are submitted to Elasticsearch as strings, thus never triggering a conflict in the first place, but I can't seem to figure out a way to achieve this behavior with the sink as it is right now. Is this type of configuration possible?

Secondly, I would expect dropped log events to at least show up as some error in the Serilog self-log, but it seems they are all silently ignored. Is this a bug or is this intended behavior?

Cheers,
A. Hagen

[mivano]

You can override the formatter, so you can make a decision on how to store the value. We see people pre or post fix the field names with the type.

Regarding logging; there is an option to inspect the batch result, so it will be send to a function or another sink.

[hagen93]

Thanks,

Do you know of any examples how to override the formatter? I saw the ElasticsearchJsonFormatter, but that looked like it was very low level. Is there any way of just applying a transformation to the data before formatting?

For the batch result, are you thinking of EmitEventFailure? I have this set to WriteToSelfLog.

Cheers,
A. Hagen

[mivano]

Have a look here: #184 (comment)

The Selflog is an option, but then you need to listen to the selflog. You can also use a function to intercept the issues and determine what to do with them.

[hagen93]

Thanks for the example.

About the Selflog, what do you mean by listen to it? For now I have been using Serilog.Debugging.SelfLog.Enable(Console.Error); which causes Serilog Elasticsearch Sink to correctly print out error messages when it cannot connect to Elasticsearch (and possibly in other cases too, but I haven't encountered them yet), however I don't see any error messages when a log event is dropped.

Cheers,
A. Hagen

[mivano]

Sending it to console.error is indeed listening to it. There are other options, like subscribing using a function so you can inspect the log event that failed and act on it.

[hagen93]

So that means I should be seeing error messages for dropped log events? Or am I misunderstanding you?

[mivano]

No you are correct. Are you using the durable option?

[mivano]

I m not a user of this sink anymore, so I m unsure what never versions of ES will do or return. I m assuming it returns an error collection. See here: https://github.com/serilog/serilog-sinks-elasticsearch/blob/dev/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs#L75

[hagen93]

Hi,

I don't use the durable mode no. The behavior I'm looking for is for the sink to log to ES if possible, or at least throw an error to STDOUT/STDERR if ES logging is not possible. My app is running containerized so storing to disk does not help me much.

I debugged the batch emit function a bit and was able to extract a response from ES when en error occured (https://pastebin.com/XubKauE2). Is this the kind of response that the sink would pick up and log to the selflog?

Cheers,
A. Hagen

[hagen93]

I just noticed that downgrading the sink to v7.1.0 makes the error log to the selflog correctly. Do I have to match the version of the sink to the version of ES I am running? Currently I'm running ES 7.2.0.

[mivano]

No in theory it should support multiple versions. In practice it looks like it does not handle the response that well and miss out on the errors returned. Will need to look into that, or if you care for a PR?

[sweetsleep]

I'm am college of hagen, i have a look at it in the coming weak and make a PR

[TooLazyToMakeAName]

@mivano Hi! I've opened up a PR to address this issue and also #282 which was previously discussed here. Could you have a look? The fundamental problem seemed to be that Elasticsearch.NET introduced some breaking changes between 6.X and 7.X which was not accounted for.