Reddit: "How is the it's performance under multiple concurrent request i.e can it handle processing of 10 PDF at once on 8 core and 16 GB machine?"

[Goldziher]

I haven't created benchmarks - it would be a great contribution if anyone wants to add this.

The design of this library is asynchrous (concurrent) but not parallel (multi-threaded). You can use coroutines to create in-thread concurrency but its not multi-threaded.

To effectively use this library you simply need to use the basic asyncio primitives - or those from anyio if you prefer abstractions and handle multiple files in a non-blocking fashion:

from asyncio import gather
from pathlib import Path
from kreuzberg import extract_file

async def handle_multiple_files(files_to_extract: list[Path]) -> list[str]:
   """Concurrently extract text from multiple files"""
    return await gather(*[extract_file(file) for file in files_to_extract])

This function will execute the extract_file functions concurrently.