[Bug]: An error occurred (TooManyRequestsException) when calling the `ListServiceQuotas` operation (AWS)

[peterschmidt85]

Steps to reproduce

1. Ensure an AWS backend is configured (ensure regions aren't specified)
2. Submit multiple runs (e.g. > 5) programmatically

Actual behaviour

1. Some of the runs are marked as No offers without being tried to provision

Expected behaviour

1. All runs submitted successfully

dstack version

0.19.12

Server logs

[17:43:05] ERROR    dstack._internal.server.services.backends:353 Got exception when requesting offers from
                    backend BackendType.AWS
                    Traceback (most recent call last):
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_interna
                    l/utils/common.py", line 21, in run_async
                        return await asyncio.get_running_loop().run_in_executor(None, func_with_args)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      File
                    "/Users/cheptsov/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/c
                    oncurrent/futures/thread.py", line 59, in run
                        result = self.fn(*self.args, **self.kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/cachetools/__in
                    it__.py", line 717, in wrapper
                        v = method(self, *args, **kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_interna
                    l/core/backends/base/compute.py", line 136, in get_offers_cached
                        return self.get_offers(requirements)
                               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_interna
                    l/core/backends/aws/compute.py", line 129, in get_offers
                        regions_to_quotas = _get_regions_to_quotas(self.session, regions)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_interna
                    l/core/backends/aws/compute.py", line 799, in _get_regions_to_quotas
                        regions_to_quotas[future_to_region] = future.result()
                                                                      ~~~~~~~~~~~~~^^
                      File
                    "/Users/cheptsov/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/c
                    oncurrent/futures/_base.py", line 449, in result
                        return self.__get_result()
                               ~~~~~~~~~~~~~~~~~^^
                      File
                    "/Users/cheptsov/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/c
                    oncurrent/futures/_base.py", line 401, in __get_result
                        raise self._exception
                      File
                    "/Users/cheptsov/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/c
                    oncurrent/futures/thread.py", line 59, in run
                        result = self.fn(*self.args, **self.kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_interna
                    l/core/backends/aws/compute.py", line 784, in get_region_quotas
                        for page in client.get_paginator("list_service_quotas").paginate(ServiceCode="ec2"):
                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/pagina
                    te.py", line 272, in __iter__
                        response = self._make_request(current_kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/contex
                    t.py", line 123, in wrapper
                        return func(*args, **kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/pagina
                    te.py", line 361, in _make_request
                        return self._method(**current_kwargs)
                               ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/client
                    .py", line 598, in _api_call
                        return self._make_api_call(operation_name, kwargs)
                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/contex
                    t.py", line 123, in wrapper
                        return func(*args, **kwargs)
                      File
                    "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/botocore/client
                    .py", line 1061, in _make_api_call
                        raise error_class(parsed_response, operation_name)
                    botocore.errorfactory.TooManyRequestsException: An error occurred
                    (TooManyRequestsException) when calling the ListServiceQuotas operation: The request
                    failed because you exceeded the rate quota for this operation.

Additional information

No response

[r4victor]

Can probably be fixed with a quota cache on a Compute level similar to #2610

[r4victor]

We already cache backend offers by resources, so the problem should not reproducible if submitting runs with same resources.

[r4victor]

The problem is not reproduced with consecutive submissions to the CLI (e.g. for i in {0..50}; do dstack apply -f .dstack/confs/task.dstack.yaml -n test-task-$i -b aws -y -d; done). It's only reproduced when submitting multiple runs via the API without getting the run plan first. This leads to runs being processed simultaneously and offers requested in parallel so the offers cache is not involved (unlike when getting a plan first). The quota cache won't help either since it won't be used by concurrent requests. The number for concurrent requests to get_offers has to be limited to mitigate the problem.

import os
import requests

url = "http://127.0.0.1:8000"
project = "main"
token = os.environ["DSTACK_TOKEN"]

for i in range(5):
    run_name = f"test-task-{i}"
    print(f"Submitting task {run_name}")
    resp = requests.post(
        url=f"{url}/api/project/{project}/runs/apply",
        headers={"Authorization": f"Bearer {token}"},
        json={
            "plan":{
                "run_spec": {
                    "run_name": run_name,
                    "configuration": {
                        "type": "task",
                        "backends": [
                            "aws"
                        ],
                        "regions": [
                            "us-west-2"
                        ],
                        "commands": [
                            "echo 1"
                        ],
                    },
                    "ssh_key_pub": "dummy",
                }
            },
            "force": False
        }
    )

[r4victor]

We may try implementing a quota cache per region, so the requests will be quick enough for the cache to be reused between concurrent requests.