Use exponentially increasing retry delays for pending runs

[r4victor]

Fixes #2420

Previously pending runs were retried with 15s delay which caused lots of job submissions being created in case of constant no capacity. The new logic is to use exponential retry delays with 10m max delay. As a result, there will be 144 retries/day max as compared to the previous 5760 retires/day (40 times diff). The retry latency will not change for failed and retried jobs unless there is long no capacity. Runs that wait for capacity for hours/days can wait additional 5-10m.

There is still an issue of too many job submissions being returned in the API, e.g. a run that sits in pending for months. This is to be addressed in a separate issue if proved to be necessary.