Memory leak with openib component when using MPI_AllToAllW

[patrick-legi]

Thank you for taking the time to submit an issue!

Background information

Using MPI_type_Create_SubArray and MPI_AllToAllW show a memory leak with openib component.

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

OpenMPI 3.1 and OpenMPI 4.0.5.
OpenMPI 1.10 works fine.

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

OpenMPI built from sources using gcc7.3 (built from sources too) on CentOS6 (Rocks-Cluster) distribution.
Configure command line: '--prefix=/share/apps/GCC73/openmpi/31-patch'
'--enable-mpirun-prefix-by-default'
'--disable-dlopen' '--enable-mpi-cxx'
'--without-slurm' '--enable-mpi-thread-multiple'
test_layout_array.tar.gz
alltoallw.diff.gz

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version:
  CentOS release 6.7 (Final) as an update of Rocks release 6.2 (SideWinder)

• Computer hardware:
  DELL R720, DELL C6200

• Network type:
  Infiniband QDR (Qlogic), Ethernet 10Gb

---

Details of the problem

Using MPI_type_Create_SubArray and MPI_AllToAllW show a memory leak (memory used by the processes increase continuously) when using openib components in OpenMPI 3 and OpenMPI4. Disabling this component with "mpirun --mca pml ob1 --mca btl tcp,self.... " do not show the memory leak but is not usable in production (10Gb ethernet interconnect instead of ib QDR).

Gilles Gouaillardet has provided a patch (workaround) to solve the problem and we are back in production.

Attached file:

• Small peace of code to reproduce the problem (test_layout_array.tar.gz)
• The patch of alltoallw provided by Gilles Gouaillardet (alltoaal.diff.gz)
• The memory consumption (for the test case) before and after applying the patch to OpenMPI 3.1 (png file)
  

[jsquyres]

See also the thread on the mailing list that led up to this issue: https://www.mail-archive.com/users@lists.open-mpi.org/msg34197.html

[jsquyres]

@patrick-legi @ggouaillardet The initial description mentions that Gilles supplied a patch to get this user back in production.

1. Was that patch applied to v4.0.x and v4.1.x?
2. Is this related to Patch for memory leaks in ompi-v4.0.x #8217?
3. Do we know if the same problem occurs with the UCX PML?

[gpaulsen]

@ggouaillardet are you planning to PR this to master and release branches/.

[ggouaillardet]

This is unrelated to #8217

IIRC, I did not see anything wrong with the current code, nor found the root cause of the leak.
I happened to find that doing things differently (and likely suboptimal performance wise) the leak disappeared ...

I initially planned to investigate the root cause, but could not find the time.
If you believe a PR (with likely a performance hit) is better than the leak, I can issue this PR.

[awlauria]

Removed the master label since by the description this is only when using openib, which has been purged from master, starting with v5.0.x.