Use manylinux2014 to get aarch64/ppc64le support

[hrw]

Manylinux2014 image got released some time ago. One of things it brings is support for !x86 architectures.

Scipy is using Travis CI so two things can be joined and used to generate wheel files for aarch64 and ppc64le architectures.

Amount of software installed to be able to run 'pip install scipy' would be cut. Also install times would be nicer.

[rgommers]

There's limited tooling support and a bunch of TODOs on the rollout plan (pypa/manylinux#338). So I'm wondering what the best plan is for producing wheels right now. It's too late for SciPy 1.4.0, which is ready to go. So we have 6 months till the next release.

Some questions:

• do we need to produce both manylinux2010 and manylinux2014 wheels for x86, or just one of them?
• do we want ppcle64 wheels before fixing the knownfails and persistent crash we have at the moment?
• we'll need aarch64 CI, do we use Drone or TravisCI or ...?

[tylerjereddy]

we'll need aarch64 CI, do we use Drone or TravisCI or ...?

I think Drone has been working well for OpenBLAS ARM testing (@martin-frbg @isuruf ?). I feel like Shippable was causing quite a few issues for NumPy, though maybe that has quieted down?

[isuruf]

Drone has been working well for conda-forge. Few advantages over Travis-CI.

• Faster to boot
• 60 minutes per job instead of 50 mins
• Large limit on log size
• Travis has timeout of 10 minutes with no output, but drone doesn't have it (AFAIK)
• arm32 (armhf) support

[martin-frbg]

Yes, Drone has been working quite nicely for OpenBLAS since isuruf set it up.

[hrw]

@rgommers which manylinux to use is a question of which distros you support. Manylinux2014 is CentOS 7+ while manylinux2010 is CentOS 6+ (afaik).

[matthew-brett]

My suggestion would be to make manylinux2010 wheels until CentOS 6 goes EOL, in November 30, 2020, and thereafter produce 2014 wheels. Unless there's some compelling gcc optimization we need before that.

…

On Thu, Dec 5, 2019 at 8:01 AM Marcin Juszkiewicz ***@***.***> wrote: @rgommers <https://github.com/rgommers> which manylinux to use is a question of which distros you support. Manylinux2014 is CentOS 7+ while manylinux2010 is CentOS 6+ (afaik). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11170?email_source=notifications&email_token=AAAQQHBVGGSQU6KIJZG7Y2TQXCYMNA5CNFSM4JVFHLSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF73CAI#issuecomment-562016513>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQQHH355TISVIHXCPLTUDQXCYMNANCNFSM4JVFHLSA> .

[odidev]

Hi @rgommers as per the discussion #11724, I have modified the Travis.yml file to run builds for aarch64 ( currently only for python v3.7 ), but since the tests are taking longer than 50 minutes to execute the build is getting timed-out. Please have a look:

https://travis-ci.com/github/odidev/scipy/builds/157507705

I have already requested the Travis community to increase the timeout here: https://travis-ci.community/t/request-regarding-timeout-extension-for-arm64-builds/7938

Also, since you suggested that using drone-ci will be a much preferable option, I have been working on creating the drone.yml for aarch64 support ( for python v3.7 ), The tests have been successfully triggered but here also the test suite is taking longer than 60 minutes to complete hence build is halted: https://cloud.drone.io/odidev/scipy/13/1/2

I will be more than happy to contribute to making the aarch64 ci support available for scipy, Please share your thoughts on this.

[hrw]

Can tests be run in parallel? AArch64 machines usually have more cores than x86-64 ones.

[rgommers]

TravisCI gets to 65%, Drone only to 25% of the test suite - not what I had expected.

Tests can be run in parallel. If I look at what's happening now, NPY_NUM_BUILD_JOBS=2 is set so that should speed up the build a little, however the tests aren't yet run in parallel. Installing pytest-xdist and running python runtests.py --parallel=2 should fix that. I believe a TravisCI job gets 2 cores, although it would be worth double checking that for ARM64.

[isuruf]

Drone machines have 96 cores each, but is shared across projects.

[odidev]

@rgommers I am running the Travis-ci builds and facing the below issue while building it, could you please suggest what am I missing: https://travis-ci.com/github/odidev/scipy