Skip to content

Improved HPC Support through cluster_config() #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 19, 2025
Merged

Improved HPC Support through cluster_config() #297

merged 4 commits into from
Jun 19, 2025

Conversation

shikokuchuo
Copy link
Member

@shikokuchuo shikokuchuo commented Jun 5, 2025

Closes #296.

Implements cluster_config() for full granular control over HPC job schedulers such as Slurm sbatch, SGE / Torque / PBS qsub and LSF bsub.

These systems expect a shell script to be executed with the full configuration options contained within, including the loading of environment modules etc., which would otherwise not be possible to pass in a single command line.

Rather than dynamically create an actual script on the filesystem, this implementation employs a Heredoc for more optimal programmatic control.

EDIT: rebasing after merging #298, now inherently supports array jobs just by adding a line #SBATCH --array= or equivalent in the options.

FYI @michaelmayer2

@shikokuchuo shikokuchuo marked this pull request as ready for review June 5, 2025 22:04
@shikokuchuo shikokuchuo force-pushed the launcher branch 3 times, most recently from d3b5977 to 5712b61 Compare June 9, 2025 09:31
@shikokuchuo shikokuchuo merged commit afe3c30 into main Jun 19, 2025
12 checks passed
@shikokuchuo shikokuchuo deleted the launcher branch June 19, 2025 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using mirai with HPC schedulers
1 participant