v3.0x: Hostfile behavior under RM allocation

[artpol84]

Open MPI version

---

Details of the problem

When running application under SLURM allocation but using hostfile ras/slurm breaks the launch:

$ mpirun --bind-to core --map-by node -hostfile ./hfile -np 24 -mca pml ob1 -mca btl tcp,self --mca ras_base_verbose 100 <app>
[headnode:04487] mca: base: components_register: registering framework ras components
[headnode:04487] mca: base: components_register: found loaded component slurm
[headnode:04487] mca: base: components_register: component slurm register function successful
[headnode:04487] mca: base: components_open: opening ras components
[headnode:04487] mca: base: components_open: found loaded component slurm
[headnode:04487] mca: base: components_open: component slurm open function successful
[headnode:04487] mca:base:select: Auto-selecting ras components
[headnode:04487] mca:base:select:(  ras) Querying component [slurm]
[headnode:04487] mca:base:select:(  ras) Query of component [slurm] set priority to 50
[headnode:04487] mca:base:select:(  ras) Selected component [slurm]

======================   ALLOCATED NODES   ======================
        cn01: flags=0x10 slots=12 max_slots=0 slots_inuse=0 state=UP
        cn02: flags=0x10 slots=12 max_slots=0 slots_inuse=0 state=UP
=================================================================
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 24 slots
that were requested by the application:
  /hpc/local/benchmarks/hpcx_install_Sunday/hpcx-gcc-redhat7.2/ompi-v3.0.x/tests/osu-micro-benchmarks-5.3.2/osu_barrier

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
[headnode:04487] mca: base: close: component slurm closed
[headnode:04487] mca: base: close: unloading component slurm

if I explicitly disable slurm ras all works fine:

$ mpirun --bind-to core --map-by node -hostfile ./hfile -np 24 -mca pml ob1 -mca btl tcp,self --mca ras_base_verbose 100 --mca ras '^slurm' <app>
[headnode:06721] mca: base: components_register: registering framework ras components
[headnode:06721] mca: base: components_register: found loaded component simulator
[headnode:06721] mca: base: components_register: component simulator register function successful
[headnode:06721] mca: base: components_open: opening ras components
[headnode:06721] mca: base: components_open: found loaded component simulator
[headnode:06721] mca:base:select: Auto-selecting ras components
[headnode:06721] mca:base:select:(  ras) Querying component [simulator]
[headnode:06721] mca:base:select:(  ras) No component selected!

======================   ALLOCATED NODES   ======================
        cn01: flags=0x00 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
        cn02: flags=0x00 slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
=================================================================

... <app output> ...

@rhc54 is this an expected behavior?

[rhc54]

Depends - what is in the hostfile, and what did slurm allocate?

[artpol84]

hostfile is

cn01
cn02

Slurm allocation is in the first output

[ggouaillardet]

@artpol84 what if you run only 2 MPI tasks ?
would mpirun assume 1 slot per host if -hostfile is used under SLURM ?

[rhc54]

Somebody would have to check @jsquyres table for behaviors. IIRC, providing a hostname in the hostfile without specifying num slots is supposed to default to the detected number of cores (or hwthreads if use-hwthreads-as-cpus is set). So this should have selected those two nodes out of the allocation, but used whatever slurm provided as the number of slots allocated on those nodes. From the output, it looks like slurm allocated 12 slots/node, and the hostfile processor incorrectly marked the nodes as having a hard limit of only 1 slot each.

[artpol84]

@bwbarrett according to @rhc54 this is a regression. So maybe we need to consider this as a blocker as well.

[artpol84]

@hppritcha

[rhc54]

@bwbarrett I would advise against making this a blocker as we don't have anyone standing up and saying if/when they will fix it. I know I don't have time.

[bwbarrett]

@rhc54: if it's a feature regression, it's a blocker. If there's no one to work on it, then we don't release until someone comes up with resources or we explicitly decide that the regression is ok and we document it. Neither of those has happened, so blocker (or I'm wrong that it's a feature regression, in which case it's not a blocker).

[artpol84]

@ggouaillardet 2 procs work fine. So hostfile ignores RAS info or incorrectly interprets it and yes - assumes one slot per host.

[jsquyres]

Per discussion on 8 Aug 2017 webex: yes, this is definitely a change in behavior. At the moment, no one has any availability to work on it. Not sure where to go from here.