Skip to content

Implement run priorities #2635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 19, 2025
Merged

Implement run priorities #2635

merged 4 commits into from
May 19, 2025

Conversation

r4victor
Copy link
Collaborator

@r4victor r4victor commented May 14, 2025

Closes #2564

The PR adds a priority property to run configurations that allows controlling the order in which the runs are provisioned. Runs with higher priorities take precedence over runs with lower priorities. Previously, submitted jobs were processed in FIFO manner with older jobs processed first. Now, the jobs are first sorted by descending priority.

The priority property is updatable, so it can be changed for already submitted runs and will take effect:

Active run test-priority already exists. Detected configuration changes that can be 
updated in-place: ['priority']

Note that if a high priority run cannot be scheduled, it does NOT block other runs with lower priority from scheduling (aka Best effort FIFO). So dstack will try a higher priority run, and if it fails, moves to a lower priority one. This can lead to a lower priority run "overtaking" higher priority run and is considered ok. An alternative is Strict FIFO which blocks lower priority runs until a higher priority run is scheduled. It would be a worse default since one unschedulable run can block all runs in the project, but we may consider supporting it in the future if requested.

@r4victor r4victor marked this pull request as draft May 14, 2025 11:36
@r4victor r4victor marked this pull request as ready for review May 15, 2025 10:18
@r4victor r4victor requested a review from un-def May 15, 2025 10:18
@r4victor r4victor merged commit fe3a6d2 into master May 19, 2025
25 checks passed
@r4victor r4victor deleted the issue_2564_run_priority branch May 19, 2025 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Allow to specify run priorities
2 participants