fix: Elasticsearch: Request Entity Too Large #28117

[inferno-umar]

Fix for gitea putting everything into one request without batching and sending it to Elasticsearch for indexing as issued in #28117

This issue occured in large repositories while Gitea tries to
index the code using ElasticSearch.

I've applied necessary changes that takes batch length from below config (app.ini)

[queue.code_indexer]
BATCH_LENGTH=<length_int>

and batches all requests to Elasticsearch in chunks as configured in the above config

[lunny]

This batch length is for queue but no for elastic search.
For ealstic search I think we can have a hardcode batch length. So that you code could be simpler.

[inferno-umar]

This batch length is for queue but no for elastic search. For ealstic search I think we can have a hardcode batch length. So that you code could be simpler.

I've 2 points to mention here:

1. At the end the code_indexer's job is added to queue which Gitea then runs in the background in batches or as a whole for indexing which thereby takes the above config and it just works. Kindly refer this documentation -> it has clearly mentioned
   Gitea creates the following non-unique queues:
   code_indexer

2. If I'm still wrong, I could introduce new setting in [indexer] as INDEXING_BATCH_LENGTH that'll take the length of batch just for code indexing, no need for hardcoding

[wxiaoguang]

Thank your for your PR! (It is really fast 👍)

Two suggestions in my mind:

1. BATCH_LENGTH is only for the "queue dispatching", it shouldn't be used here. I think we do not need to fine-tune this value for elasticsearch batching, so you could just hard-code a reasonable length into code at the moment.
   • The document says: BATCH_LENGTH: 20: Batch data before passing to the handler.
   • The flow is "task publisher -> queue -> task handler -> elasticsearch"
   • So the "BATCH_LENGTH" is for "queue -> task handler" dispatching, not for "task handler -> elasticsearch" batching.
   • Why not adding a new option? There are too many options now .... no user really needs it at the moment, a reasonable value is good enough.
2. There is a builtin min function in Go now 🤣

[inferno-umar]

Thanks a lot @lunny @wxiaoguang for clearing me out.
I've created a hardcoded value that pretty much batches all our indexing needs without errors to ElasticSearch. Kindly review

[wxiaoguang]

Ideally Gitea should also read the contents into batches (but not read everything into reqs in one time), to avoid OOM.

As a quick fix, this PR is better than before.

[wxiaoguang]

@lunny the commit message is wrong (again). Please manually fix the PR description before merging.