Reconsider allow expensive queries setting

[javanna]

We have not so long ago introduced a setting that allows users to disable expensive queries, which makes Elasticsearch reject some type of queries that are known to be potentially slow, like scripted queries (or based on runtime fields), wildcard queries, regex queries etc.

While we have recently added new potentially slow queries to this existing mechanism, like doc_value only queries, the setting does not cover all ways to run slow queries and may give users a false sense of security.

Additionally, we've been adding more and more ways to run slow queries, Elasticsearch is able to support them much better than it did in the past, thanks to the Async Search API. We've observed that our Solutions rely on expensive queries for core functionalities, and their UI may not even load when users disable the ability to run expensive queries, which makes the setting problematic and mush less useful.

Given that we got requests to make the setting configurable per role (#53607), we should discuss if this is a reliable mechanism that we want to build on top of, or consider other options. Some thoughts to get the discussion started:

• Is it a reasonable expectation for our Solutions to not rely on expensive queries?
• Should Solutions instead have a preemptive check that expensive queries are allowed, otherwise give a clear error message or expose degraded functionalities?
• Should Elasticsearch document that Solutions may rely on expensive queries, hence disallowing them may prevent Solutions from working at all?
• Should Solutions have some kind of mechanism that prevents users from disabling running expensive queries as they rely on them?
• How is the setting used in the wild and is it reliable in those scenarios?
• Is disallowing expensive queries to be considered a security feature?
• Do we have other options when it comes to supporting slow queries, is there still a need to disable them entirely?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[joshdover]

In terms of Solution's usage, I wonder if allowing system indices to opt out of this behavior so that we can rely on these "expensive" features in some cases where we know the impact is limited. This won't solve all potential usages but it could prevent some breakages of functionality when this setting is enabled.

[ywangd]

I wonder if allowing system indices to opt out of this behavior so that we can rely on these "expensive" features in some cases where we know the impact is limited.

For the existing bug of threadpool usage of system indices (#89073), one idea is to attach the conditional behaviour to the user (e.g. elastic/fleet-server) instead of the target resource. I think makes sense in a service point of view. It likely requires core to be aware of users which is not an easy change. But our decision should be consistent between these two issues.

[javanna]

We discussed this with the team. We were rather aligned on the current challenges with the allow expensive queries setting: it gives false sense of security, it may prevent non expensive queries while at the same time allowing alternative ways to run expensive queries. We said that often times slow is confused with expensive and we should clarify that distinction: slow is not necessarily bad, especially when async search is used. When it comes to resource usage, some queries may require more memory or CPU than others, and possibly end up crashing data nodes: these problems could be solved assigning priorities to search requests (#37867), improving resiliency and/or our current timeout mechanisms.

It is clear that there is a need to protect clusters from potentially falling apart due to bad queries, yet the allow expensive queries is not a complete solution to address that. It is not necessarily reasonable that our Solutions should fully work without relying on what we currently declare "expensive". We said we would like to collect more data points on users that use this setting and whether it does help in practice addressing the need around securing clusters. We will resume this discussion once we have more information.

[ThomThomson]

Has there been any movement on this? Currently the new Controls in Dashboard work well with allow_expensive_queries off, but we're considering migrating to use prefix queries instead of terms include for prefix searching within Keyword mapped fields. I'm getting the sense that terms include is roughly as expensive as prefix, and that it can be a little arbitrary which query types require that setting to be on.

Switching to prefix queries would greatly reduce our code complexity. The only risk is that this might prevent the Dashboard UI from loading when the allow_expensive_queries is off - just like the many of the solutions fail to load. Are we aware of many user complaints resulting in these apps not loading? Is turning off this setting common at all?

Additionally, I've been unable to find any methods which can tell Kibana plugins the value of this setting - is it even possible to check whether or not it's on in order to change pieces of the UI?