Request fixes

[hjelmn]

No description provided.

[hjelmn]

@bosilca, @jladd-mlnx Two problems. One causing compilation failure and the other causing hangs.

[hjelmn]

Hmm, yalla isn't hanging but needs an update as well. incoming.

[jladd-mlnx]

@hjelmn Just so I understand. The root cause was because of a lack of support in the CM PML. The combination of OSHMEM with CM PML MXM MTL and IKRIT SML just happened to trigger the hang. OSHMEM itself was not involved in the hang. Correct?

[hjelmn]

Looks like it.

[hjelmn]

Need jenkins to confirm.

[hjelmn]

@bosilca I still see a lot of references to ompi_request_lock in ompi. I think most of them can go away. Is there a reason those were left?

[bosilca]

In all instances where the request lock protects the request_complete it is not necessary anymore. @thananon started to remove them but apparently he stopped after ob1.

[hjelmn]

@thananon Should look at the ones in pml/ucx and see if they are still relevant. I got yalla and cm.

[hjelmn]

Unrelated error? error posting send request error 12

[hjelmn]

:bot:retest:

[bosilca]

The second part of the error is more interesting "Cannot allocate memory. size = 14"

[jladd-mlnx]

It's an OpenIB error on PML OB1. It's a thread multiple test.

taskset -c 6,7 timeout -s SIGSEGV 10m /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace-2/ompi_install1/bin/mpirun -np 2 -bind-to core -mca btl_openib_if_include mlx4_0:1 -x MXM_RDMA_PORTS=mlx4_0:1 -x UCX_NET_DEVICES=mlx4_0:1 -x UCX_TLS=rc,cm -mca pml ob1 -mca btl self,openib -mca btl_openib_receive_queues X,4096,1024:X,12288,512:X,65536,512 /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace-2/ompi_install1/thread_tests/thread-tests-1.1/latency_th 8
01:29:40 Size (bytes)    Time (us)
01:29:40 [jenkins01][[61884,1],0][btl_openib_endpoint.c:115:mca_btl_openib_endpoint_post_send] error posting send request error 12: Cannot allocate memory. size = 14
01:29:40 
01:29:40 [jenkins01][[61884,1],0][btl_openib_endpoint.c:115:mca_btl_openib_endpoint_post_send] error posting send request error 12: Cannot allocate memory. size = 14
01:29:40 
01:29:40 [jenkins01][[61884,1],0][btl_openib_endpoint.c:115:mca_btl_openib_endpoint_post_send] error posting send request error 12: Cannot allocate memory. size = 14
01:29:40 
01:29:40 *** glibc detected *** /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace-2/ompi_install1/thread_tests/thread-tests-1.1/latency_th: munmap_chunk(): invalid pointer: 0x00007fff9c0058a0 ***
01:29:40 [jenkins01][[61884,1],0][btl_openib_endpoint.c:115:mca_btl_openib_endpoint_post_send] error posting send request error 12: Cannot allocate memory. size = 14
01:29:40

[hjelmn]

@jladd-mlnx Is that a jenkins machine failure or legitimate?

[jladd-mlnx]

@hjelmn Something random. It ran once, and now it's failing repeatedly. The OSHMEM command line works, however. Jenkins is in fine health. This is a legitimate failure.

[jladd-mlnx]

@bosilca @hjelmn It doesn't reproduce the failure with CM PML but grinds very slowly. It seems to work beautifully with Yalla PML. Fast and safe. Only OB1 fails almost always (but not always) with the weird, random allocation error. Looks like something isn't thread safe.

[jladd-mlnx]

@bosilca @hjelmn It's an XRC failure.

[hjelmn]

@jladd-mlnx General XRC failure or openib btl XRC failure?

[jladd-mlnx]

@hjelmn OpenIB BTL XRC failure.

[hjelmn]

Well, since this is a different failure I will merge this. Will have to dissect the other failure tomorrow.

[jsquyres]

@hjelmn I'm afraid @thananon doesn't know anything about UCX. Who's available from the UCX side who can help here?

[bosilca]

I might be able to get to it either Friday afternoon, or early next week.

[thananon]

@jsquyres @bosilca I think I can get to it today. I just have a few things to make sure about UCX. I will PM you guys for advice.

[thananon]

#1719